2024-03-26T02:40:55.533 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T02:40:55.533 DEBUG:teuthology.run:Teuthology command: teuthology --name adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi --archive /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398 --verbose --owner scheduled_adking@teuthology --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} -- /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/orig.config.yaml 2024-03-26T02:40:55.566 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T02:40:55.654 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398 branch: wip-adk-testing-2024-03-25-1628 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: adking@redhat.com first_in_suite: false job_id: '7622398' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-03-25-1628 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN - mons down, quorum - is down \(out of quorum\) - OSDMAP_FLAGS - PG_DEGRADED - OSD_DOWN - POOL_APP_NOT_ENABLED - CACHE_POOL_NO_HIT_SET - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-adk-testing-2024-03-25-1628 sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9192 sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-03-25-1628 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 targets: smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+HD1F+CsAoAuBiU0YXX6ZO8LDL6Gw8Wd1zlXvGNg99R52pLLesdbE07P2/AiRrsQhexeAGcyfhTVX78iBrlkE= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFDGsZgpfauHHw95gyfgQdV5c8hqgFowRMuIWnKgKgQvujCn7mndJNuDK3/TV/nLRqwgRrrDuliz13Baa87nbI= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_01:50:53 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T02:40:55.654 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa; will attempt to use it 2024-03-26T02:40:55.655 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa/tasks 2024-03-26T02:40:55.655 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T02:40:55.656 INFO:teuthology.task.internal:Checking packages... 2024-03-26T02:40:55.678 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003' 2024-03-26T02:40:55.678 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T02:40:55.678 INFO:teuthology.packaging:ref: None 2024-03-26T02:40:55.678 INFO:teuthology.packaging:tag: None 2024-03-26T02:40:55.678 INFO:teuthology.packaging:branch: wip-adk-testing-2024-03-25-1628 2024-03-26T02:40:55.678 INFO:teuthology.packaging:sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:40:55.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-03-25-1628 2024-03-26T02:40:55.899 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2475.ge42b8ea3 2024-03-26T02:40:55.900 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T02:40:55.910 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T02:40:55.910 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T02:40:55.949 INFO:teuthology.task.internal:Saving configuration 2024-03-26T02:40:55.961 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T02:40:56.030 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:40:56.055 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:33:57.769053', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+HD1F+CsAoAuBiU0YXX6ZO8LDL6Gw8Wd1zlXvGNg99R52pLLesdbE07P2/AiRrsQhexeAGcyfhTVX78iBrlkE='} 2024-03-26T02:40:56.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:33:57.770832', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFDGsZgpfauHHw95gyfgQdV5c8hqgFowRMuIWnKgKgQvujCn7mndJNuDK3/TV/nLRqwgRrrDuliz13Baa87nbI='} 2024-03-26T02:40:56.075 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T02:40:56.081 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T02:40:56.081 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T02:40:56.081 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T02:40:56.163 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f14db9df160>, signals=[15]) 2024-03-26T02:40:56.163 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T02:40:56.169 INFO:teuthology.task.internal:Opening connections... 2024-03-26T02:40:56.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-03-26T02:40:56.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:40:56.279 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-03-26T02:40:56.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:40:56.365 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T02:40:56.372 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-03-26T02:40:56.390 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-03-26T02:40:56.391 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-03-26T02:40:56.448 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T02:40:56.449 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T02:40:56.449 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T02:40:56.449 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-03-26T02:40:56.471 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-03-26T02:40:56.487 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-03-26T02:40:56.488 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-03-26T02:40:56.543 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T02:40:56.544 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-03-26T02:40:56.544 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T02:40:56.544 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T02:40:56.544 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T02:40:56.544 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-03-26T02:40:56.570 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T02:40:56.579 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T02:40:56.607 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T02:40:56.608 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-03-26T02:40:56.610 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-03-26T02:40:56.625 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T02:40:56.632 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T02:40:56.632 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-03-26T02:40:56.668 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-03-26T02:40:56.702 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T02:40:56.760 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T02:40:56.775 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T02:40:56.775 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T02:40:56.776 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T02:40:56.776 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T02:40:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:40:56.792 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-03-26T02:40:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:40:56.794 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-26T02:40:56.848 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-361.el9.x86_64 2024-03-26T02:40:56.849 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-361.el9.x86_64 2024-03-26T02:40:56.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-03-26T02:40:56.852 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-361.el9.x86_64 2024-03-26T02:40:56.852 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-361.el9.x86_64 2024-03-26T02:40:56.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-03-26T02:40:58.986 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T02:40:59.034 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-26T02:41:00.411 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-26T02:41:00.426 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-26T02:41:01.200 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-26T02:41:01.277 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T02:41:03.095 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-26T02:41:06.284 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-26T02:41:07.145 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-26T02:41:08.671 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-26T02:41:09.823 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-26T02:41:12.360 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-26T02:41:13.960 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-26T02:41:15.023 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-26T02:41:15.481 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T02:41:17.280 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T02:41:17.280 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T02:41:17.378 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T02:41:17.390 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T02:41:17.391 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:41:17.391 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:41:17.391 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:41:17.391 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-03-26T02:41:17.391 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:41:17.391 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 76 M 2024-03-26T02:41:17.392 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 123 M 2024-03-26T02:41:17.392 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:41:17.568 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:17.570 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:17.573 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:17.711 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:17.723 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:17.747 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:18.099 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 112 kB/s | 66 kB 00:00 2024-03-26T02:41:18.132 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:18.164 INFO:teuthology.orchestra.run.smithi066.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:41:20.520 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-26T02:41:22.662 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T02:41:22.662 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T02:41:22.760 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:41:22.773 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:41:22.774 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:41:22.774 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-03-26T02:41:22.774 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:41:22.774 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 76 M 2024-03-26T02:41:22.774 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 123 M 2024-03-26T02:41:22.774 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:41:22.930 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 663 kB/s | 66 kB 00:00 2024-03-26T02:41:23.364 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 28 MB/s | 15 MB 00:00 2024-03-26T02:41:23.722 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 38 MB/s | 33 MB 00:00 2024-03-26T02:41:24.039 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 25 MB/s | 27 MB 00:01 2024-03-26T02:41:24.039 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:41:24.039 INFO:teuthology.orchestra.run.smithi192.stdout:Total 60 MB/s | 76 MB 00:01 2024-03-26T02:41:24.706 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:41:25.048 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:41:25.048 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:41:26.052 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:41:26.052 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:41:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:41:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:41:29.316 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:41:32.058 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:41:32.184 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:41:33.265 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 970 kB/s | 15 MB 00:15 2024-03-26T02:41:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:41:36.943 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:41:41.705 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:41:50.388 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 872 kB/s | 27 MB 00:32 2024-03-26T02:41:56.788 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 873 kB/s | 33 MB 00:39 2024-03-26T02:41:56.791 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:41:56.791 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.9 MB/s | 76 MB 00:39 2024-03-26T02:41:57.485 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:41:57.892 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:41:57.892 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:41:58.935 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:41:58.935 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:42:01.560 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:42:02.196 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:42:02.280 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:42:05.178 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:42:05.314 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:42:10.395 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:42:10.509 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:11.044 INFO:teuthology.orchestra.run.smithi192.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T02:42:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:42:11.089 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:42:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:42:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:42:11.868 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:42:12.202 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:42:12.301 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:42:12.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:42:12.302 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T02:42:12.302 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T02:42:12.302 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T02:42:12.303 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T02:42:12.303 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T02:42:12.303 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:42:12.382 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:42:12.383 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:42:12.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-03-26T02:42:12.426 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-03-26T02:42:12.426 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-03-26T02:42:12.426 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 68.el9 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8264254 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-03-26T02:42:12.427 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:42:12.428 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T02:42:12.429 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T02:42:12.429 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-432.el9.x86_64 2024-03-26T02:42:12.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T02:42:12.973 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-03-26T02:42:14.368 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T02:42:14.388 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-03-26T02:42:14.390 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-03-26T02:42:14.405 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.JXS8oYOJdH 2024-03-26T02:42:14.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JXS8oYOJdH 2024-03-26T02:42:14.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.JXS8oYOJdH 2024-03-26T02:42:14.615 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.JXS8oYOJdH is 6KB 2024-03-26T02:42:14.663 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.JXS8oYOJdH 2024-03-26T02:42:14.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T02:42:14.742 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T02:42:14.742 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T02:42:14.742 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T02:42:14.742 INFO:teuthology.orchestra.run.smithi192.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T02:42:14.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T02:42:14.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-03-26T02:42:15.539 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:44.885 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T02:42:44.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-26T02:42:44.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:42:46.526 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:46.526 INFO:teuthology.orchestra.run.smithi066.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T02:42:46.526 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:42:46.571 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:46.920 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:46.920 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:42:46.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:42:46.921 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T02:42:47.295 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:42:47.296 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:42:47.621 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:42:47.722 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:42:47.722 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:42:47.723 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T02:42:47.723 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T02:42:47.723 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T02:42:47.723 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T02:42:47.723 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T02:42:47.723 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:42:47.805 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:42:47.805 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:42:47.805 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-03-26T02:42:47.852 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-03-26T02:42:47.852 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-03-26T02:42:47.852 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-03-26T02:42:47.852 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 68.el9 2024-03-26T02:42:47.852 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8264254 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-03-26T02:42:47.853 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:42:47.854 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T02:42:47.854 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T02:42:47.854 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T02:42:47.854 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-03-26T02:42:47.854 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:42:47.854 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T02:42:47.855 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T02:42:47.855 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-432.el9.x86_64 2024-03-26T02:42:47.855 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T02:42:48.444 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-03-26T02:42:49.867 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T02:42:49.895 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-03-26T02:42:49.898 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-26T02:42:49.913 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.HFirpomgpH 2024-03-26T02:42:49.913 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HFirpomgpH 2024-03-26T02:42:49.985 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.HFirpomgpH 2024-03-26T02:42:50.137 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.HFirpomgpH is 6KB 2024-03-26T02:42:50.186 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.HFirpomgpH 2024-03-26T02:42:50.201 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T02:42:50.277 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T02:42:50.277 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T02:42:50.277 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T02:42:50.277 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T02:42:50.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T02:42:50.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-03-26T02:43:19.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-03-26T02:43:20.435 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T02:43:20.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:43:20.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:43:28.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-26T02:43:28.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:43:46.851 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-03-26T02:43:55.039 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T02:43:58.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-26T02:43:58.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:43:59.285 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-26T02:43:59.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-26T02:43:59.673 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T02:43:59.673 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-03-26T02:43:59.688 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-432.el9.x86_64 2024-03-26T02:43:59.689 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T02:43:59.689 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T02:43:59.689 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T02:44:00.690 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T02:44:00.691 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T02:44:00.765 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-03-26T02:44:00.825 DEBUG:teuthology.parallel:result is None 2024-03-26T02:44:04.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:44:04.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:44:07.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T02:44:19.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:44:19.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:44:22.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T02:44:37.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:44:37.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:44:40.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-26T02:44:58.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:44:58.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:45:30.152 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-26T02:45:30.551 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:45:30.551 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T02:45:30.551 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-26T02:45:30.566 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-432.el9.x86_64 2024-03-26T02:45:30.567 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T02:45:30.567 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T02:45:30.567 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T02:45:31.568 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T02:45:31.568 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T02:45:31.662 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-03-26T02:45:31.700 DEBUG:teuthology.parallel:result is None 2024-03-26T02:45:31.700 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T02:45:31.708 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T02:45:31.708 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:45:31.710 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:45:31.729 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T02:45:31.746 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T02:45:31.780 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T02:45:31.780 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:45:31.783 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:45:31.826 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T02:45:31.834 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T02:45:31.834 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:45:31.850 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:45:31.886 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:45:31.904 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:45:31.906 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:45:31.920 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:45:31.921 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T02:45:31.931 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T02:45:31.932 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:45:31.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:45:31.980 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T02:45:31.990 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T02:45:31.990 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:45:32.024 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:45:32.040 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:45:32.122 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:45:32.173 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:45:32.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:45:32.303 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:45:32.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:45:32.376 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:45:32.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:45:32.469 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:45:32.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:45:32.599 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:45:32.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:45:32.670 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-03-26T02:45:32.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-03-26T02:45:32.719 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:45:32.764 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:45:33.147 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T02:45:33.156 INFO:teuthology.task.internal:Starting timer... 2024-03-26T02:45:33.156 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T02:45:33.191 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T02:45:33.222 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T02:45:33.223 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-03-26T02:45:33.270 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-03-26T02:45:33.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-03-26T02:45:33.370 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-03-26T02:45:33.372 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T02:45:33.372 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-03-26T02:45:33.402 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-03-26T02:45:33.403 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-03-26T02:45:33.430 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-03-26T02:45:33.430 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi066.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-03-26T02:45:33.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:45:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:45:33.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:45:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:45:33.508 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T02:45:33.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:45:33.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:45:33.599 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T02:45:33.608 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T02:45:33.632 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T02:45:33.634 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi066.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T02:50:07.397 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi066.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-03-26T02:50:07.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:50:07.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:50:07.503 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-26T02:50:07.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-26T02:50:07.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-26T02:50:07.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:50:07.670 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-26T02:50:07.749 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-26T02:50:07.749 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T02:50:07.760 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T02:50:07.760 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:50:07.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:50:07.763 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:50:07.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:50:07.797 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:50:07.818 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:50:07.821 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:50:07.838 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:50:07.854 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-03-26T02:50:07.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-03-26T02:50:07.871 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-03-26T02:50:07.882 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-03-26T02:50:07.892 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:50:07.898 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:50:07.911 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:50:07.914 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:50:07.965 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-26T02:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.968 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.970 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:50:07.974 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:50:07.974 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-26T02:50:07.974 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.974 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.974 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.974 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:50:07.975 INFO:teuthology.run_tasks:Running task install... 2024-03-26T02:50:07.983 DEBUG:teuthology.task.install:project ceph 2024-03-26T02:50:07.983 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}} 2024-03-26T02:50:07.983 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'} 2024-03-26T02:50:07.983 INFO:teuthology.task.install:Using flavor: default 2024-03-26T02:50:07.989 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T02:50:07.989 INFO:teuthology.task.install:extra packages: [] 2024-03-26T02:50:07.989 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'tag': None, 'wait_for_package': False} 2024-03-26T02:50:07.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:50:07.992 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'tag': None, 'wait_for_package': False} 2024-03-26T02:50:07.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:50:08.222 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/ 2024-03-26T02:50:08.222 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2475.ge42b8ea3 2024-03-26T02:50:08.227 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/ 2024-03-26T02:50:08.227 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2475.ge42b8ea3 2024-03-26T02:50:08.337 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T02:50:08.337 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:50:08.337 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T02:50:08.370 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T02:50:08.370 DEBUG:teuthology.orchestra.run.smithi066:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T02:50:08.392 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T02:50:08.392 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:50:08.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T02:50:08.421 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T02:50:08.421 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T02:50:08.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T02:50:08.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T02:50:08.529 DEBUG:teuthology.orchestra.run.smithi066:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T02:50:08.558 INFO:teuthology.orchestra.run.smithi066.stdout:check_obsoletes = 1 2024-03-26T02:50:08.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum clean all 2024-03-26T02:50:08.568 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T02:50:08.634 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-03-26T02:50:08.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-03-26T02:50:08.826 INFO:teuthology.orchestra.run.smithi066.stdout:74 files removed 2024-03-26T02:50:08.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw 2024-03-26T02:50:08.883 INFO:teuthology.orchestra.run.smithi192.stdout:74 files removed 2024-03-26T02:50:08.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-03-26T02:50:09.527 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 262 kB/s | 83 kB 00:00 2024-03-26T02:50:09.648 INFO:teuthology.orchestra.run.smithi066.stdout:ceph packages for x86_64 175 kB/s | 83 kB 00:00 2024-03-26T02:50:10.052 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-03-26T02:50:10.357 INFO:teuthology.orchestra.run.smithi066.stdout:ceph noarch packages 18 kB/s | 12 kB 00:00 2024-03-26T02:50:10.786 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-26T02:50:11.241 INFO:teuthology.orchestra.run.smithi066.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-26T02:50:11.347 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T02:50:11.818 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T02:50:12.723 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-26T02:50:13.360 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-26T02:50:15.044 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.0 MB 00:01 2024-03-26T02:50:17.871 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-26T02:50:19.839 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-26T02:50:20.152 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-26T02:50:23.941 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-26T02:50:25.375 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-03-26T02:50:26.680 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-26T02:50:27.713 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 21 MB 00:02 2024-03-26T02:50:28.187 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-26T02:50:32.677 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-26T02:50:35.146 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 9.8 M 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.4 M 2024-03-26T02:50:35.157 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.4 M 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 21 M 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 25 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 746 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T02:50:35.158 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 501 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.1 M 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 45 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 134 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 161 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 321 k 2024-03-26T02:50:35.159 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 298 k 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 99 k 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout:Install 30 Packages 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-03-26T02:50:35.160 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:35.161 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 59 M 2024-03-26T02:50:35.161 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:50:36.861 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-base-19.0.0-2475.ge42b8ea3.el9.x86 3.4 MB/s | 5.4 MB 00:01 2024-03-26T02:50:36.920 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-selinux-19.0.0-2475.ge42b8ea3.el9. 432 kB/s | 25 kB 00:00 2024-03-26T02:50:37.154 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): ceph-common-19.0.0-2475.ge42b8ea3.el9.x 11 MB/s | 21 MB 00:01 2024-03-26T02:50:37.187 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): libcephfs2-19.0.0-2475.ge42b8ea3.el9.x8 2.7 MB/s | 746 kB 00:00 2024-03-26T02:50:37.212 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): libradosstriper1-19.0.0-2475.ge42b8ea3. 8.4 MB/s | 501 kB 00:00 2024-03-26T02:50:37.329 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): ceph-radosgw-19.0.0-2475.ge42b8ea3.el9. 4.8 MB/s | 9.8 MB 00:02 2024-03-26T02:50:37.346 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): python3-ceph-argparse-19.0.0-2475.ge42b 337 kB/s | 45 kB 00:00 2024-03-26T02:50:37.396 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-cephfs-19.0.0-2475.ge42b8ea3.el 3.1 MB/s | 161 kB 00:00 2024-03-26T02:50:37.422 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-ceph-common-19.0.0-2475.ge42b8e 1.4 MB/s | 134 kB 00:00 2024-03-26T02:50:37.447 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-rados-19.0.0-2475.ge42b8ea3.el 6.2 MB/s | 321 kB 00:00 2024-03-26T02:50:37.489 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-rbd-19.0.0-2475.ge42b8ea3.el9. 4.3 MB/s | 298 kB 00:00 2024-03-26T02:50:37.514 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): python3-rgw-19.0.0-2475.ge42b8ea3.el9. 1.4 MB/s | 99 kB 00:00 2024-03-26T02:50:37.631 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 745 kB/s | 106 kB 00:00 2024-03-26T02:50:37.657 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-26T02:50:37.682 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-26T02:50:37.732 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T02:50:37.757 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-26T02:50:37.783 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-26T02:50:37.875 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): librgw2-19.0.0-2475.ge42b8ea3.el9.x86_ 7.5 MB/s | 5.1 MB 00:00 2024-03-26T02:50:37.919 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:50:37.922 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:50:37.951 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:50:38.069 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:50:38.091 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:50:38.142 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 100 kB/s | 36 kB 00:00 2024-03-26T02:50:38.142 INFO:teuthology.orchestra.run.smithi192.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:50:38.175 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 529 kB/s | 221 kB 00:00 2024-03-26T02:50:38.226 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-03-26T02:50:38.343 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-03-26T02:50:38.368 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-26T02:50:38.427 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 837 kB/s | 49 kB 00:00 2024-03-26T02:50:38.469 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-26T02:50:38.527 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-26T02:50:38.577 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-26T02:50:38.619 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-26T02:50:38.761 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-03-26T02:50:39.037 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): librados2-19.0.0-2475.ge42b8ea3.el9.x8 7.4 MB/s | 3.4 MB 00:00 2024-03-26T02:50:39.095 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): librbd1-19.0.0-2475.ge42b8ea3.el9.x86_ 6.6 MB/s | 3.1 MB 00:00 2024-03-26T02:50:39.099 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:50:39.099 INFO:teuthology.orchestra.run.smithi192.stdout:Total 15 MB/s | 59 MB 00:03 2024-03-26T02:50:39.130 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:50:39.165 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:50:39.165 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:50:39.660 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:50:39.660 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:50:41.118 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:50:41.205 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T02:50:41.242 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T02:50:41.267 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T02:50:41.290 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T02:50:41.307 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T02:50:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T02:50:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T02:50:41.884 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T02:50:41.936 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T02:50:42.028 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T02:50:42.114 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 8/34 2024-03-26T02:50:42.138 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T02:50:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T02:50:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T02:50:42.273 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 11/34 2024-03-26T02:50:42.341 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T02:50:42.365 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T02:50:42.429 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T02:50:42.463 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T02:50:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T02:50:42.607 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T02:50:42.649 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T02:50:42.680 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T02:50:42.709 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T02:50:42.741 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T02:50:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T02:50:43.088 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T02:50:43.162 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T02:50:43.193 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 23/34 2024-03-26T02:50:43.229 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T02:50:43.691 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T02:50:43.716 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T02:50:43.787 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T02:50:43.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 27/34 2024-03-26T02:50:43.885 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 28/34 2024-03-26T02:50:46.158 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T02:50:46.191 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T02:50:46.767 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T02:50:46.803 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T02:50:46.892 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T02:50:46.892 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T02:50:46.892 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T02:50:46.893 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:46.929 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T02:50:47.732 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-03-26T02:50:52.962 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-26T02:50:55.470 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 9.8 M 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.4 M 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options 2024-03-26T02:50:55.484 INFO:teuthology.orchestra.run.smithi066.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.4 M 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 21 M 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 25 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 746 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 501 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T02:50:55.485 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.1 M 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 45 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 134 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 161 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 321 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 298 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 99 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T02:50:55.486 INFO:teuthology.orchestra.run.smithi066.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout:Installing weak dependencies: 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout:Install 30 Packages 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 2 Packages 2024-03-26T02:50:55.487 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:50:55.488 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 59 M 2024-03-26T02:50:55.488 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:50:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T02:50:56.633 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-03-26T02:50:56.633 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-03-26T02:50:56.633 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-03-26T02:50:56.633 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-03-26T02:50:56.634 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-03-26T02:50:56.634 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-03-26T02:50:56.634 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-03-26T02:50:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:56.754 INFO:teuthology.orchestra.run.smithi066.stdout:(1/32): ceph-radosgw-19.0.0-2475.ge42b8ea3.el9. 9.1 MB/s | 9.8 MB 00:01 2024-03-26T02:50:56.797 INFO:teuthology.orchestra.run.smithi066.stdout:(2/32): ceph-selinux-19.0.0-2475.ge42b8ea3.el9. 602 kB/s | 25 kB 00:00 2024-03-26T02:50:56.905 INFO:teuthology.orchestra.run.smithi066.stdout:(3/32): libcephfs2-19.0.0-2475.ge42b8ea3.el9.x8 6.7 MB/s | 746 kB 00:00 2024-03-26T02:50:56.956 INFO:teuthology.orchestra.run.smithi066.stdout:(4/32): libradosstriper1-19.0.0-2475.ge42b8ea3. 9.8 MB/s | 501 kB 00:00 2024-03-26T02:50:57.098 INFO:teuthology.orchestra.run.smithi066.stdout:(5/32): ceph-base-19.0.0-2475.ge42b8ea3.el9.x86 3.8 MB/s | 5.4 MB 00:01 2024-03-26T02:50:57.181 INFO:teuthology.orchestra.run.smithi066.stdout:(6/32): python3-ceph-argparse-19.0.0-2475.ge42b 543 kB/s | 45 kB 00:00 2024-03-26T02:50:57.256 INFO:teuthology.orchestra.run.smithi066.stdout:(7/32): python3-ceph-common-19.0.0-2475.ge42b8e 1.8 MB/s | 134 kB 00:00 2024-03-26T02:50:57.332 INFO:teuthology.orchestra.run.smithi066.stdout:(8/32): librgw2-19.0.0-2475.ge42b8ea3.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-26T02:50:57.357 INFO:teuthology.orchestra.run.smithi066.stdout:(9/32): python3-cephfs-19.0.0-2475.ge42b8ea3.el 1.6 MB/s | 161 kB 00:00 2024-03-26T02:50:57.382 INFO:teuthology.orchestra.run.smithi066.stdout:(10/32): python3-rados-19.0.0-2475.ge42b8ea3.el 6.3 MB/s | 321 kB 00:00 2024-03-26T02:50:57.424 INFO:teuthology.orchestra.run.smithi066.stdout:(11/32): python3-rgw-19.0.0-2475.ge42b8ea3.el9. 2.3 MB/s | 99 kB 00:00 2024-03-26T02:50:57.449 INFO:teuthology.orchestra.run.smithi066.stdout:(12/32): python3-rbd-19.0.0-2475.ge42b8ea3.el9. 3.2 MB/s | 298 kB 00:00 2024-03-26T02:50:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T02:50:57.569 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T02:50:57.570 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:50:57.570 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T02:50:57.570 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T02:50:57.570 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T02:50:57.570 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:57.572 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T02:50:57.575 INFO:teuthology.orchestra.run.smithi066.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-26T02:50:57.600 INFO:teuthology.orchestra.run.smithi066.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-26T02:50:57.625 INFO:teuthology.orchestra.run.smithi066.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-26T02:50:57.638 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T02:50:57.638 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T02:50:57.675 INFO:teuthology.orchestra.run.smithi066.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T02:50:57.692 INFO:teuthology.orchestra.run.smithi066.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-26T02:50:57.718 INFO:teuthology.orchestra.run.smithi066.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 41 kB 00:00 2024-03-26T02:50:57.801 INFO:teuthology.orchestra.run.smithi066.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 433 kB/s | 36 kB 00:00 2024-03-26T02:50:57.860 INFO:teuthology.orchestra.run.smithi066.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 812 kB/s | 47 kB 00:00 2024-03-26T02:50:57.885 INFO:teuthology.orchestra.run.smithi066.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-03-26T02:50:58.052 INFO:teuthology.orchestra.run.smithi066.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-26T02:50:58.094 INFO:teuthology.orchestra.run.smithi066.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 628 kB/s | 26 kB 00:00 2024-03-26T02:50:58.136 INFO:teuthology.orchestra.run.smithi066.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-26T02:50:58.220 INFO:teuthology.orchestra.run.smithi066.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-26T02:50:58.245 INFO:teuthology.orchestra.run.smithi066.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-03-26T02:50:58.445 INFO:teuthology.orchestra.run.smithi066.stdout:(27/32): ceph-common-19.0.0-2475.ge42b8ea3.el9. 7.4 MB/s | 21 MB 00:02 2024-03-26T02:50:58.470 INFO:teuthology.orchestra.run.smithi066.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 851 kB/s | 192 kB 00:00 2024-03-26T02:50:58.495 INFO:teuthology.orchestra.run.smithi066.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-26T02:50:58.671 INFO:teuthology.orchestra.run.smithi066.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-03-26T02:50:58.764 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 6/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 8/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 9/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 10/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 11/34 2024-03-26T02:50:58.765 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 13/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T02:50:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T02:50:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T02:50:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T02:50:58.767 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T02:50:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T02:50:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T02:50:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T02:50:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T02:50:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T02:50:58.768 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/34 2024-03-26T02:50:58.788 INFO:teuthology.orchestra.run.smithi066.stdout:(31/32): librbd1-19.0.0-2475.ge42b8ea3.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-26T02:50:59.180 INFO:teuthology.orchestra.run.smithi066.stdout:(32/32): librados2-19.0.0-2475.ge42b8ea3.el9.x8 4.8 MB/s | 3.4 MB 00:00 2024-03-26T02:50:59.183 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:50:59.183 INFO:teuthology.orchestra.run.smithi066.stdout:Total 16 MB/s | 59 MB 00:03 2024-03-26T02:50:59.217 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:50:59.260 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:50:59.260 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:50:59.754 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T02:50:59.754 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T02:50:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T02:50:59.756 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:50:59.757 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:50:59.763 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:50:59.763 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:51:00.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-03-26T02:51:00.581 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:00.730 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 48 M 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T02:51:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 163 k 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-03-26T02:51:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:00.733 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 49 M 2024-03-26T02:51:00.733 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 206 M 2024-03-26T02:51:00.733 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:01.273 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 904 kB/s | 217 kB 00:00 2024-03-26T02:51:01.343 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:51:01.373 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-26T02:51:01.419 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T02:51:01.423 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-26T02:51:01.451 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T02:51:01.477 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T02:51:01.508 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T02:51:01.525 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T02:51:01.540 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): libcephsqlite-19.0.0-2475.ge42b8ea3.el9. 321 kB/s | 163 kB 00:00 2024-03-26T02:51:01.566 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-26T02:51:01.709 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T02:51:01.765 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T02:51:02.122 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T02:51:02.195 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T02:51:02.287 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T02:51:02.384 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 8/34 2024-03-26T02:51:02.413 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T02:51:02.476 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T02:51:02.504 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T02:51:02.548 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 11/34 2024-03-26T02:51:02.618 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T02:51:02.640 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T02:51:02.687 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T02:51:02.722 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T02:51:02.834 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T02:51:02.865 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T02:51:02.905 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T02:51:02.938 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T02:51:02.967 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T02:51:03.000 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T02:51:03.332 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T02:51:03.357 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-test-19.0.0-2475.ge42b8ea3.el9.x86_ 21 MB/s | 48 MB 00:02 2024-03-26T02:51:03.360 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:03.361 INFO:teuthology.orchestra.run.smithi192.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-26T02:51:03.362 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T02:51:03.375 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:03.396 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:03.396 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T02:51:03.478 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 23/34 2024-03-26T02:51:03.514 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T02:51:03.781 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:03.782 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:03.999 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T02:51:04.026 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T02:51:04.097 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T02:51:04.168 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 27/34 2024-03-26T02:51:04.194 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 28/34 2024-03-26T02:51:04.373 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:04.441 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T02:51:04.483 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T02:51:04.515 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T02:51:04.557 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T02:51:04.586 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T02:51:06.519 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T02:51:06.557 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T02:51:07.145 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T02:51:07.177 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T02:51:07.260 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T02:51:07.261 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T02:51:07.261 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T02:51:07.261 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:07.287 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T02:51:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T02:51:09.252 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T02:51:11.557 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T02:51:11.557 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/6 2024-03-26T02:51:11.557 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/6 2024-03-26T02:51:11.557 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T02:51:11.557 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T02:51:11.557 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T02:51:11.947 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T02:51:11.947 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-16.el9.x86_64 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:11.948 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:12.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-03-26T02:51:12.837 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:40 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:12.990 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================= 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================= 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 6.6 k 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 2.4 M 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 1.5 M 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 249 k 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 4.7 M 2024-03-26T02:51:12.993 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 17 M 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T02:51:12.994 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T02:51:12.995 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================= 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout:Install 36 Packages 2024-03-26T02:51:12.996 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:12.997 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 30 M 2024-03-26T02:51:12.997 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 108 M 2024-03-26T02:51:12.997 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:13.485 INFO:teuthology.orchestra.run.smithi192.stdout:(1/36): ceph-19.0.0-2475.ge42b8ea3.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-26T02:51:13.736 INFO:teuthology.orchestra.run.smithi192.stdout:(2/36): ceph-mgr-19.0.0-2475.ge42b8ea3.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-03-26T02:51:13.786 INFO:teuthology.orchestra.run.smithi192.stdout:(3/36): ceph-mds-19.0.0-2475.ge42b8ea3.el9.x86_ 5.0 MB/s | 2.4 MB 00:00 2024-03-26T02:51:13.878 INFO:teuthology.orchestra.run.smithi192.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2475.ge42b 2.7 MB/s | 249 kB 00:00 2024-03-26T02:51:14.011 INFO:teuthology.orchestra.run.smithi192.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-26T02:51:14.053 INFO:teuthology.orchestra.run.smithi192.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-26T02:51:14.120 INFO:teuthology.orchestra.run.smithi192.stdout:(7/36): ceph-mon-19.0.0-2475.ge42b8ea3.el9.x86_ 7.4 MB/s | 4.7 MB 00:00 2024-03-26T02:51:14.145 INFO:teuthology.orchestra.run.smithi192.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 343 kB/s | 32 kB 00:00 2024-03-26T02:51:14.171 INFO:teuthology.orchestra.run.smithi192.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-26T02:51:14.196 INFO:teuthology.orchestra.run.smithi192.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-26T02:51:14.221 INFO:teuthology.orchestra.run.smithi192.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T02:51:14.272 INFO:teuthology.orchestra.run.smithi192.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-26T02:51:14.296 INFO:teuthology.orchestra.run.smithi192.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-26T02:51:14.322 INFO:teuthology.orchestra.run.smithi192.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T02:51:14.347 INFO:teuthology.orchestra.run.smithi192.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T02:51:14.406 INFO:teuthology.orchestra.run.smithi192.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-03-26T02:51:14.514 INFO:teuthology.orchestra.run.smithi192.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-03-26T02:51:14.690 INFO:teuthology.orchestra.run.smithi192.stdout:(18/36): ceph-osd-19.0.0-2475.ge42b8ea3.el9.x86 18 MB/s | 17 MB 00:00 2024-03-26T02:51:14.898 INFO:teuthology.orchestra.run.smithi192.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2024-03-26T02:51:14.932 INFO:teuthology.orchestra.run.smithi192.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 287 kB/s | 151 kB 00:00 2024-03-26T02:51:14.957 INFO:teuthology.orchestra.run.smithi192.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2024-03-26T02:51:15.016 INFO:teuthology.orchestra.run.smithi192.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-26T02:51:15.041 INFO:teuthology.orchestra.run.smithi192.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-26T02:51:15.057 INFO:teuthology.orchestra.run.smithi192.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-26T02:51:15.083 INFO:teuthology.orchestra.run.smithi192.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-26T02:51:15.125 INFO:teuthology.orchestra.run.smithi192.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-26T02:51:15.150 INFO:teuthology.orchestra.run.smithi192.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-26T02:51:15.175 INFO:teuthology.orchestra.run.smithi192.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 501 kB/s | 46 kB 00:00 2024-03-26T02:51:15.201 INFO:teuthology.orchestra.run.smithi192.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-26T02:51:15.226 INFO:teuthology.orchestra.run.smithi192.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-26T02:51:15.243 INFO:teuthology.orchestra.run.smithi192.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 244 kB/s | 16 kB 00:00 2024-03-26T02:51:15.268 INFO:teuthology.orchestra.run.smithi192.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-26T02:51:15.285 INFO:teuthology.orchestra.run.smithi192.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 608 kB/s | 36 kB 00:00 2024-03-26T02:51:15.319 INFO:teuthology.orchestra.run.smithi192.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-26T02:51:15.353 INFO:teuthology.orchestra.run.smithi192.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-26T02:51:15.378 INFO:teuthology.orchestra.run.smithi192.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-26T02:51:15.383 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:15.383 INFO:teuthology.orchestra.run.smithi192.stdout:Total 13 MB/s | 30 MB 00:02 2024-03-26T02:51:15.407 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:15.448 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:15.448 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:15.856 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:15.856 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:17.179 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /sys 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /proc 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /mnt 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /var/tmp 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /home 2024-03-26T02:51:17.180 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /root 2024-03-26T02:51:17.181 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /tmp 2024-03-26T02:51:17.181 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:17.239 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T02:51:17.266 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T02:51:17.299 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T02:51:17.370 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T02:51:17.406 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T02:51:17.439 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T02:51:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T02:51:17.609 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T02:51:17.650 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T02:51:17.722 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T02:51:17.776 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T02:51:17.814 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T02:51:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T02:51:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T02:51:18.104 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T02:51:18.130 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T02:51:18.160 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T02:51:18.160 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:18.160 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T02:51:18.160 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T02:51:18.160 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T02:51:18.160 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T02:51:18.162 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T02:51:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T02:51:18.249 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T02:51:18.249 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T02:51:18.249 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T02:51:18.279 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T02:51:18.356 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T02:51:18.390 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T02:51:18.423 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T02:51:18.466 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T02:51:18.498 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T02:51:18.533 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T02:51:18.623 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T02:51:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T02:51:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T02:51:18.866 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T02:51:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T02:51:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 31/36 2024-03-26T02:51:19.177 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T02:51:19.202 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T02:51:19.202 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:19.202 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T02:51:19.202 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T02:51:19.202 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T02:51:19.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:19.341 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T02:51:19.341 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/34 2024-03-26T02:51:19.341 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/34 2024-03-26T02:51:19.341 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 6/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 8/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 9/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 10/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 11/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 13/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T02:51:19.342 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T02:51:19.343 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T02:51:19.344 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T02:51:19.344 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T02:51:19.344 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T02:51:19.345 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T02:51:19.345 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T02:51:19.345 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/34 2024-03-26T02:51:20.431 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T02:51:20.431 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T02:51:20.432 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T02:51:20.433 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T02:51:20.434 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T02:51:20.434 INFO:teuthology.orchestra.run.smithi066.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T02:51:20.434 INFO:teuthology.orchestra.run.smithi066.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T02:51:20.434 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:20.434 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T02:51:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T02:51:20.723 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:20.723 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T02:51:20.723 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T02:51:20.723 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T02:51:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:20.762 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-test 2024-03-26T02:51:21.171 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T02:51:21.194 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T02:51:21.195 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:21.195 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T02:51:21.195 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T02:51:21.195 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T02:51:21.195 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:21.347 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:29 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:21.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T02:51:21.444 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T02:51:21.444 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:21.444 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T02:51:21.444 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T02:51:21.444 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T02:51:21.445 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:21.510 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:21.510 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 48 M 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 163 k 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T02:51:21.511 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:21.512 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:51:21.512 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:21.512 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-03-26T02:51:21.512 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:21.512 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T02:51:21.513 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 49 M 2024-03-26T02:51:21.513 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 206 M 2024-03-26T02:51:21.513 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:51:21.951 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 979 kB/s | 217 kB 00:00 2024-03-26T02:51:21.976 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): libcephsqlite-19.0.0-2475.ge42b8ea3.el9. 661 kB/s | 163 kB 00:00 2024-03-26T02:51:22.009 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-26T02:51:22.026 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-26T02:51:22.268 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 724 kB/s | 187 kB 00:00 2024-03-26T02:51:23.229 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T02:51:23.229 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/36 2024-03-26T02:51:23.229 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 6/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T02:51:23.230 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T02:51:23.231 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T02:51:23.232 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T02:51:23.232 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T02:51:23.232 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T02:51:23.232 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T02:51:23.233 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T02:51:24.094 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): ceph-test-19.0.0-2475.ge42b8ea3.el9.x86_ 20 MB/s | 48 MB 00:02 2024-03-26T02:51:24.097 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:24.097 INFO:teuthology.orchestra.run.smithi066.stdout:Total 19 MB/s | 49 MB 00:02 2024-03-26T02:51:24.112 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:51:24.135 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:51:24.136 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:51:24.179 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T02:51:24.179 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T02:51:24.180 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T02:51:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:24.182 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:24.542 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:51:24.543 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:51:24.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-03-26T02:51:25.125 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:53 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:25.141 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:51:25.221 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T02:51:25.221 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:51:25.253 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T02:51:25.274 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:25.275 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:51:25.275 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:25.285 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T02:51:25.328 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T02:51:25.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-03-26T02:51:25.355 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T02:51:25.896 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:53 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:26.045 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:26.046 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:26.046 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:26.046 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 762 k 2024-03-26T02:51:26.046 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 762 k 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 769 k 2024-03-26T02:51:26.047 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:26.470 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-19.0.0-2475.ge42b8ea3.el9.noarch.rpm 1.8 MB/s | 762 kB 00:00 2024-03-26T02:51:26.470 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:26.471 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.8 MB/s | 762 kB 00:00 2024-03-26T02:51:26.471 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:26.476 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:26.476 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:26.481 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:26.482 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:26.553 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:26.983 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:27.066 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:27.470 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:27.761 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:27.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:27.761 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:27.761 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:27.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:27.761 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:27.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T02:51:28.522 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:28.672 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 145 k 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 145 k 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 435 k 2024-03-26T02:51:28.673 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:29.034 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-19.0.0-2475.ge42b8e 403 kB/s | 145 kB 00:00 2024-03-26T02:51:29.034 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:29.035 INFO:teuthology.orchestra.run.smithi192.stdout:Total 401 kB/s | 145 kB 00:00 2024-03-26T02:51:29.035 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:29.042 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:29.043 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:29.094 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:29.094 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:29.301 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T02:51:29.326 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T02:51:29.326 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:29.326 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T02:51:29.326 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:30.185 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T02:51:30.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T02:51:30.250 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:30.250 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:30.250 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:30.250 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:30.251 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:30.265 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T02:51:30.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-03-26T02:51:31.115 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:59 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:31.213 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:51:31.265 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:31.266 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:51:31.266 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:31.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T02:51:31.899 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:59 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:32.061 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:32.062 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-03-26T02:51:32.062 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:32.062 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 3.5 M 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 34 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 16 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T02:51:32.063 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 8.6 M 2024-03-26T02:51:32.064 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 104 M 2024-03-26T02:51:32.065 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:32.645 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T02:51:32.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/6 2024-03-26T02:51:32.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/6 2024-03-26T02:51:32.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T02:51:32.645 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T02:51:32.646 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T02:51:32.663 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2475.ge4 184 kB/s | 34 kB 00:00 2024-03-26T02:51:32.780 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2475.ge42 55 kB/s | 16 kB 00:00 2024-03-26T02:51:32.921 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): ceph-mgr-dashboard-19.0.0-2475.ge42b8ea 8.0 MB/s | 3.5 MB 00:00 2024-03-26T02:51:32.972 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-03-26T02:51:33.002 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: jq-1.6-16.el9.x86_64 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:33.003 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:33.005 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 238 kB 00:00 2024-03-26T02:51:33.081 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-03-26T02:51:33.273 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-26T02:51:33.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph 2024-03-26T02:51:33.606 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 865 kB/s | 548 kB 00:00 2024-03-26T02:51:33.698 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 340 kB/s | 144 kB 00:00 2024-03-26T02:51:33.882 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 112 kB/s | 31 kB 00:00 2024-03-26T02:51:33.924 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-26T02:51:33.933 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:41 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:34.104 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:34.106 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: ceph x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 6.6 k 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 2.4 M 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 1.5 M 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 249 k 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 4.7 M 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 17 M 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T02:51:34.107 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T02:51:34.108 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout:Install 36 Packages 2024-03-26T02:51:34.109 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:34.110 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 30 M 2024-03-26T02:51:34.110 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 108 M 2024-03-26T02:51:34.110 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:51:34.123 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 443 kB/s | 188 kB 00:00 2024-03-26T02:51:34.128 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:34.128 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.2 MB/s | 8.6 MB 00:02 2024-03-26T02:51:34.148 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:34.167 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:34.167 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:34.343 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:34.343 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:34.579 INFO:teuthology.orchestra.run.smithi066.stdout:(1/36): ceph-19.0.0-2475.ge42b8ea3.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-26T02:51:34.863 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:34.929 INFO:teuthology.orchestra.run.smithi066.stdout:(2/36): ceph-mds-19.0.0-2475.ge42b8ea3.el9.x86_ 4.2 MB/s | 2.4 MB 00:00 2024-03-26T02:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T02:51:34.967 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T02:51:34.971 INFO:teuthology.orchestra.run.smithi066.stdout:(3/36): ceph-mgr-19.0.0-2475.ge42b8ea3.el9.x86_ 2.5 MB/s | 1.5 MB 00:00 2024-03-26T02:51:35.014 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T02:51:35.053 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T02:51:35.080 INFO:teuthology.orchestra.run.smithi066.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2475.ge42b 2.2 MB/s | 249 kB 00:00 2024-03-26T02:51:35.163 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T02:51:35.205 INFO:teuthology.orchestra.run.smithi066.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-26T02:51:35.239 INFO:teuthology.orchestra.run.smithi066.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-26T02:51:35.286 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T02:51:35.306 INFO:teuthology.orchestra.run.smithi066.stdout:(7/36): ceph-mon-19.0.0-2475.ge42b8ea3.el9.x86_ 6.5 MB/s | 4.7 MB 00:00 2024-03-26T02:51:35.331 INFO:teuthology.orchestra.run.smithi066.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 343 kB/s | 32 kB 00:00 2024-03-26T02:51:35.356 INFO:teuthology.orchestra.run.smithi066.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-26T02:51:35.381 INFO:teuthology.orchestra.run.smithi066.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-26T02:51:35.388 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T02:51:35.407 INFO:teuthology.orchestra.run.smithi066.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T02:51:35.432 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T02:51:35.469 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T02:51:35.490 INFO:teuthology.orchestra.run.smithi066.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-26T02:51:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 10/12 2024-03-26T02:51:35.516 INFO:teuthology.orchestra.run.smithi066.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 948 kB/s | 103 kB 00:00 2024-03-26T02:51:35.541 INFO:teuthology.orchestra.run.smithi066.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T02:51:35.566 INFO:teuthology.orchestra.run.smithi066.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T02:51:35.700 INFO:teuthology.orchestra.run.smithi066.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-26T02:51:35.733 INFO:teuthology.orchestra.run.smithi066.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 650 kB/s | 125 kB 00:00 2024-03-26T02:51:36.034 INFO:teuthology.orchestra.run.smithi066.stdout:(18/36): ceph-osd-19.0.0-2475.ge42b8ea3.el9.x86 15 MB/s | 17 MB 00:01 2024-03-26T02:51:36.059 INFO:teuthology.orchestra.run.smithi066.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 133 kB/s | 43 kB 00:00 2024-03-26T02:51:36.126 INFO:teuthology.orchestra.run.smithi066.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 354 kB/s | 151 kB 00:00 2024-03-26T02:51:36.185 INFO:teuthology.orchestra.run.smithi066.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 187 kB/s | 11 kB 00:00 2024-03-26T02:51:36.227 INFO:teuthology.orchestra.run.smithi066.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-26T02:51:36.252 INFO:teuthology.orchestra.run.smithi066.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-26T02:51:36.286 INFO:teuthology.orchestra.run.smithi066.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-26T02:51:36.330 INFO:teuthology.orchestra.run.smithi066.stdout:(25/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 582 kB/s | 172 kB 00:00 2024-03-26T02:51:36.353 INFO:teuthology.orchestra.run.smithi066.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-26T02:51:36.378 INFO:teuthology.orchestra.run.smithi066.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-26T02:51:36.403 INFO:teuthology.orchestra.run.smithi066.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 632 kB/s | 46 kB 00:00 2024-03-26T02:51:36.438 INFO:teuthology.orchestra.run.smithi066.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-26T02:51:36.462 INFO:teuthology.orchestra.run.smithi066.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-26T02:51:36.479 INFO:teuthology.orchestra.run.smithi066.stdout:(31/36): python3-more-itertools-8.12.0-2.el9.no 782 kB/s | 79 kB 00:00 2024-03-26T02:51:36.504 INFO:teuthology.orchestra.run.smithi066.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-26T02:51:36.529 INFO:teuthology.orchestra.run.smithi066.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-03-26T02:51:36.553 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 11/12 2024-03-26T02:51:36.585 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T02:51:36.588 INFO:teuthology.orchestra.run.smithi066.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-26T02:51:36.605 INFO:teuthology.orchestra.run.smithi066.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-26T02:51:36.630 INFO:teuthology.orchestra.run.smithi066.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-26T02:51:36.635 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:36.635 INFO:teuthology.orchestra.run.smithi066.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-26T02:51:36.658 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:51:36.702 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:51:36.702 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:51:37.123 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:51:37.123 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:51:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T02:51:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 1/12 2024-03-26T02:51:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 2/12 2024-03-26T02:51:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 3/12 2024-03-26T02:51:37.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T02:51:37.933 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T02:51:38.273 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T02:51:38.273 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T02:51:38.274 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:38.275 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:38.467 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:51:38.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T02:51:38.533 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T02:51:38.576 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T02:51:38.610 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T02:51:38.684 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T02:51:38.726 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T02:51:38.759 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T02:51:38.855 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T02:51:38.940 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T02:51:38.987 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T02:51:39.062 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T02:51:39.070 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:39.113 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T02:51:39.151 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T02:51:39.209 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T02:51:39.223 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 7.4 M 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T02:51:39.224 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-03-26T02:51:39.225 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:39.226 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 37 M 2024-03-26T02:51:39.226 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 209 M 2024-03-26T02:51:39.226 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:39.272 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T02:51:39.469 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T02:51:39.513 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T02:51:39.552 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T02:51:39.572 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2024-03-26T02:51:39.586 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T02:51:39.606 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-03-26T02:51:39.616 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T02:51:39.681 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T02:51:39.695 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T02:51:39.698 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 377 kB/s | 34 kB 00:00 2024-03-26T02:51:39.743 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T02:51:39.776 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T02:51:39.816 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T02:51:39.860 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T02:51:39.894 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T02:51:39.988 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T02:51:40.024 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-26T02:51:40.025 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T02:51:40.100 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T02:51:40.124 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-24 11 MB/s | 7.4 MB 00:00 2024-03-26T02:51:40.183 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.9 MB/s | 4.8 MB 00:00 2024-03-26T02:51:40.207 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 217 kB 00:00 2024-03-26T02:51:40.243 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T02:51:40.396 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T02:51:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 31/36 2024-03-26T02:51:40.596 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T02:51:40.622 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T02:51:40.623 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:40.623 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T02:51:40.623 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T02:51:40.623 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T02:51:40.623 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:40.683 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-26T02:51:40.708 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 392 kB/s | 195 kB 00:00 2024-03-26T02:51:40.841 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-26T02:51:40.950 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 920 kB/s | 245 kB 00:00 2024-03-26T02:51:40.952 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:40.952 INFO:teuthology.orchestra.run.smithi192.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-26T02:51:40.993 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:41.016 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:41.017 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:41.375 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:41.375 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:42.093 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:42.138 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T02:51:42.166 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T02:51:42.166 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:42.166 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T02:51:42.166 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T02:51:42.166 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T02:51:42.166 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T02:51:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T02:51:42.310 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T02:51:42.349 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T02:51:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T02:51:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T02:51:42.630 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T02:51:42.632 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T02:51:42.655 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T02:51:42.655 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:42.655 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T02:51:42.655 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T02:51:42.655 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T02:51:42.655 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:42.889 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T02:51:42.914 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T02:51:42.914 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:42.914 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T02:51:42.914 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T02:51:42.914 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T02:51:42.914 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:42.980 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T02:51:43.288 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T02:51:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T02:51:44.638 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T02:51:44.657 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 6/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T02:51:44.658 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T02:51:44.659 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T02:51:44.660 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:45.485 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T02:51:45.486 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T02:51:45.487 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T02:51:45.488 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T02:51:45.488 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T02:51:45.488 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T02:51:45.488 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:45.488 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:45.659 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T02:51:45.681 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T02:51:45.855 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base 2024-03-26T02:51:46.445 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:46.545 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:51:46.600 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:46.601 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:51:46.602 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:46.673 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install cephadm 2024-03-26T02:51:47.255 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:47.410 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:47.411 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:47.411 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 762 k 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 762 k 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 769 k 2024-03-26T02:51:47.412 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:51:47.689 INFO:teuthology.orchestra.run.smithi066.stdout:cephadm-19.0.0-2475.ge42b8ea3.el9.noarch.rpm 2.7 MB/s | 762 kB 00:00 2024-03-26T02:51:47.689 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:47.690 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.7 MB/s | 762 kB 00:00 2024-03-26T02:51:47.690 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:51:47.695 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:51:47.695 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:51:47.702 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:51:47.702 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:51:47.778 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:51:48.103 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 1/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T02:51:48.104 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T02:51:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:48.279 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T02:51:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:48.509 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:48.687 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:48.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2024-03-26T02:51:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T02:51:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:48.950 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:51:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:48.951 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:49.171 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T02:51:49.460 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:17 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:49.618 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:49.619 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-26T02:51:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:49.619 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-26T02:51:49.619 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 49 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T02:51:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:49.621 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.0 M 2024-03-26T02:51:49.622 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2024-03-26T02:51:49.622 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:49.768 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:49.824 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 191 kB/s | 24 kB 00:00 2024-03-26T02:51:49.849 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-26T02:51:49.891 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-26T02:51:49.916 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-26T02:51:49.924 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:49.924 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:49.924 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-26T02:51:49.924 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:49.924 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 145 k 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:49.925 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 145 k 2024-03-26T02:51:49.926 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 435 k 2024-03-26T02:51:49.926 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:51:49.941 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T02:51:49.967 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-26T02:51:49.992 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): ceph-mgr-rook-19.0.0-2475.ge42b8ea3.el9 167 kB/s | 49 kB 00:00 2024-03-26T02:51:50.234 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-26T02:51:50.235 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-immutable-object-cache-19.0.0-2475.ge42b8e 467 kB/s | 145 kB 00:00 2024-03-26T02:51:50.236 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:50.236 INFO:teuthology.orchestra.run.smithi066.stdout:Total 465 kB/s | 145 kB 00:00 2024-03-26T02:51:50.237 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:51:50.245 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:51:50.245 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:51:50.293 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-03-26T02:51:50.300 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:51:50.301 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:51:50.410 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 486 kB/s | 203 kB 00:00 2024-03-26T02:51:50.446 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:51:50.503 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T02:51:50.529 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T02:51:50.529 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:51:50.529 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T02:51:50.529 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:50.627 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 2.6 MB/s | 1.0 MB 00:00 2024-03-26T02:51:50.651 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-03-26T02:51:50.677 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 336 kB/s | 90 kB 00:00 2024-03-26T02:51:50.682 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:50.682 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-26T02:51:50.682 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:50.694 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:50.694 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:50.784 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:50.785 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:51.295 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:51.464 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T02:51:51.519 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T02:51:51.546 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T02:51:51.547 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:51.547 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:51:51.547 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:51:51.547 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:51.547 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:51.575 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T02:51:51.612 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T02:51:51.654 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T02:51:51.746 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T02:51:51.781 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr 2024-03-26T02:51:51.856 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T02:51:51.905 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T02:51:52.367 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:52.467 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:51:52.522 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:52.523 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:51:52.523 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:52.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T02:51:52.627 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T02:51:52.677 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T02:51:52.711 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T02:51:52.758 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T02:51:52.780 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T02:51:53.178 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:51:53.334 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:51:53.335 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-26T02:51:53.335 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 3.5 M 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 34 k 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 16 k 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T02:51:53.336 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-03-26T02:51:53.337 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:53.338 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 8.6 M 2024-03-26T02:51:53.338 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 104 M 2024-03-26T02:51:53.338 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:51:53.542 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T02:51:53.543 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T02:51:53.907 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T02:51:53.908 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T02:51:53.909 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T02:51:53.909 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T02:51:53.909 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:53.909 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:53.922 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2475.ge42 89 kB/s | 16 kB 00:00 2024-03-26T02:51:53.981 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2475.ge4 140 kB/s | 34 kB 00:00 2024-03-26T02:51:54.064 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-26T02:51:54.098 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-26T02:51:54.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T02:51:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2475.ge42b8ea 7.4 MB/s | 3.5 MB 00:00 2024-03-26T02:51:54.290 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.8 MB/s | 863 kB 00:00 2024-03-26T02:51:54.366 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 130 kB/s | 19 kB 00:00 2024-03-26T02:51:54.458 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.6 MB/s | 144 kB 00:00 2024-03-26T02:51:54.491 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-26T02:51:54.516 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 524 kB/s | 31 kB 00:00 2024-03-26T02:51:54.567 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-26T02:51:54.701 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:22 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:54.849 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 142 k 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T02:51:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 6.8 M 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 33 M 2024-03-26T02:51:54.851 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:55.017 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.8 MB/s | 2.0 MB 00:00 2024-03-26T02:51:55.022 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:55.022 INFO:teuthology.orchestra.run.smithi066.stdout:Total 5.1 MB/s | 8.6 MB 00:01 2024-03-26T02:51:55.041 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:51:55.060 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:51:55.060 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:51:55.259 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:51:55.259 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:51:55.453 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-26T02:51:55.495 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2475.ge42b8ea3.e 469 kB/s | 142 kB 00:00 2024-03-26T02:51:55.570 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 658 kB/s | 248 kB 00:00 2024-03-26T02:51:55.806 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:51:55.885 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T02:51:55.928 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T02:51:55.970 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-03-26T02:51:55.975 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T02:51:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T02:51:56.037 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 121 kB/s | 51 kB 00:00 2024-03-26T02:51:56.105 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T02:51:56.171 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 763 kB/s | 548 kB 00:00 2024-03-26T02:51:56.174 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:56.174 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-03-26T02:51:56.188 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:56.200 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:56.200 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:56.236 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T02:51:56.289 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:56.289 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:51:56.328 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T02:51:56.377 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T02:51:56.404 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T02:51:56.437 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 10/12 2024-03-26T02:51:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:51:56.763 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T02:51:56.822 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T02:51:57.332 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T02:51:57.408 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T02:51:57.470 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T02:51:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T02:51:57.548 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 11/12 2024-03-26T02:51:57.578 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T02:51:58.312 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T02:51:58.312 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/6 2024-03-26T02:51:58.312 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T02:51:58.312 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T02:51:58.312 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T02:51:58.312 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T02:51:58.606 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T02:51:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T02:51:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T02:51:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:58.607 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:51:58.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 1/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 2/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 3/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T02:51:58.884 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T02:51:58.885 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T02:51:58.885 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T02:51:59.244 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T02:51:59.245 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T02:51:59.246 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:51:59.246 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:51:59.405 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:51:59.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T02:51:59.555 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 890 k 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-03-26T02:51:59.556 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:51:59.557 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 969 k 2024-03-26T02:51:59.557 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 M 2024-03-26T02:51:59.557 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:51:59.673 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 682 kB/s | 79 kB 00:00 2024-03-26T02:51:59.856 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-19.0.0-2475.ge42b8ea3.el9.x86_ 2.9 MB/s | 890 kB 00:00 2024-03-26T02:51:59.857 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:51:59.858 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.1 MB/s | 969 kB 00:00 2024-03-26T02:51:59.858 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:51:59.878 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:51:59.878 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:51:59.954 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:51:59.955 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:00.105 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:08 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:00.148 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:00.262 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 7.4 M 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T02:52:00.264 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-03-26T02:52:00.265 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:00.266 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 37 M 2024-03-26T02:52:00.266 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 209 M 2024-03-26T02:52:00.266 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:00.290 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T02:52:00.364 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T02:52:00.616 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2024-03-26T02:52:00.649 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-03-26T02:52:00.717 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T02:52:00.742 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 379 kB/s | 34 kB 00:00 2024-03-26T02:52:00.942 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-26T02:52:01.009 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-26T02:52:01.034 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T02:52:01.074 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T02:52:01.074 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/2 2024-03-26T02:52:01.143 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-24 11 MB/s | 7.4 MB 00:00 2024-03-26T02:52:01.388 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T02:52:01.388 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:01.388 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:01.388 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T02:52:01.388 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:01.388 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:01.418 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-26T02:52:01.502 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-26T02:52:01.527 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 512 kB/s | 195 kB 00:00 2024-03-26T02:52:01.577 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-26T02:52:01.579 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:01.579 INFO:teuthology.orchestra.run.smithi066.stdout:Total 28 MB/s | 37 MB 00:01 2024-03-26T02:52:01.620 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:01.644 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:01.644 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:01.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-volume 2024-03-26T02:52:02.028 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:02.028 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:02.236 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:02.391 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 264 k 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T02:52:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:02.393 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:52:02.393 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:02.393 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-03-26T02:52:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:02.393 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 332 k 2024-03-26T02:52:02.393 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.5 M 2024-03-26T02:52:02.394 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:52:02.527 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 517 kB/s | 69 kB 00:00 2024-03-26T02:52:02.750 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:02.769 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-volume-19.0.0-2475.ge42b8ea3.el9.no 704 kB/s | 264 kB 00:00 2024-03-26T02:52:02.770 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:02.770 INFO:teuthology.orchestra.run.smithi192.stdout:Total 880 kB/s | 332 kB 00:00 2024-03-26T02:52:02.771 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:52:02.779 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:52:02.779 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:52:02.817 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:52:02.817 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:02.834 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T02:52:02.865 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T02:52:02.963 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T02:52:02.974 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:03.001 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T02:52:03.141 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T02:52:03.171 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T02:52:03.186 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T02:52:03.187 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:52:03.187 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T02:52:03.187 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:03.189 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T02:52:03.225 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T02:52:03.334 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T02:52:03.974 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/2 2024-03-26T02:52:04.017 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T02:52:04.125 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:04.287 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:04.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-03-26T02:52:05.173 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:05.323 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:05.324 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:05.324 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-26T02:52:05.324 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:05.324 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:52:05.324 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 127 k 2024-03-26T02:52:05.324 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 127 k 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 456 k 2024-03-26T02:52:05.325 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:52:05.463 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T02:52:05.633 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-19.0.0-2475.ge42b8ea3.el9.x86_64 413 kB/s | 127 kB 00:00 2024-03-26T02:52:05.633 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:05.633 INFO:teuthology.orchestra.run.smithi192.stdout:Total 411 kB/s | 127 kB 00:00 2024-03-26T02:52:05.633 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:52:05.639 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:52:05.639 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:52:05.701 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:52:05.702 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:05.847 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:06.407 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:06.548 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T02:52:06.574 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T02:52:06.688 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:06.688 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:06.688 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:06.689 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:06.689 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:06.689 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:06.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-03-26T02:52:07.474 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:07.575 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:07.630 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:07.631 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:07.631 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:07.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-03-26T02:52:08.254 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:08.404 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:08.405 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:08.405 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-26T02:52:08.405 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:08.405 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:52:08.405 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 32 k 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 32 k 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 142 k 2024-03-26T02:52:08.406 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:52:08.644 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-19.0.0-2475.ge42b8ea3.el9.x86_6 133 kB/s | 32 kB 00:00 2024-03-26T02:52:08.645 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:08.645 INFO:teuthology.orchestra.run.smithi192.stdout:Total 132 kB/s | 32 kB 00:00 2024-03-26T02:52:08.645 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:52:08.649 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:52:08.649 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:52:08.676 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:52:08.677 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:08.808 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:08.893 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:09.019 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 1/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T02:52:09.020 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T02:52:09.021 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T02:52:09.021 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T02:52:09.293 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:09.446 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T02:52:09.446 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:09.446 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:09.447 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:09.552 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:09.552 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:09.552 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:09.552 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:09.552 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:09.552 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:09.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-03-26T02:52:09.821 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-rook 2024-03-26T02:52:10.279 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:10.376 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:10.409 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:10.428 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:10.429 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:10.429 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:10.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-03-26T02:52:10.566 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:10.567 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-26T02:52:10.567 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:52:10.567 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-26T02:52:10.567 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:10.567 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 49 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T02:52:10.568 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout:Install 13 Packages 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.0 M 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 26 M 2024-03-26T02:52:10.569 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:10.741 INFO:teuthology.orchestra.run.smithi066.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 191 kB/s | 24 kB 00:00 2024-03-26T02:52:10.767 INFO:teuthology.orchestra.run.smithi066.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-26T02:52:10.817 INFO:teuthology.orchestra.run.smithi066.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-26T02:52:10.842 INFO:teuthology.orchestra.run.smithi066.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-26T02:52:10.868 INFO:teuthology.orchestra.run.smithi066.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T02:52:10.893 INFO:teuthology.orchestra.run.smithi066.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-26T02:52:10.918 INFO:teuthology.orchestra.run.smithi066.stdout:(7/13): ceph-mgr-rook-19.0.0-2475.ge42b8ea3.el9 162 kB/s | 49 kB 00:00 2024-03-26T02:52:10.977 INFO:teuthology.orchestra.run.smithi066.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-03-26T02:52:11.027 INFO:teuthology.orchestra.run.smithi066.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-26T02:52:11.031 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:11.060 INFO:teuthology.orchestra.run.smithi066.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-26T02:52:11.111 INFO:teuthology.orchestra.run.smithi066.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-26T02:52:11.129 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:11.136 INFO:teuthology.orchestra.run.smithi066.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-03-26T02:52:11.181 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:11.182 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:11.182 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:11.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-03-26T02:52:11.244 INFO:teuthology.orchestra.run.smithi066.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-26T02:52:11.246 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:11.246 INFO:teuthology.orchestra.run.smithi066.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-03-26T02:52:11.247 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:11.261 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:11.261 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:11.364 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:11.365 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:11.801 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:11.892 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:11.898 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:11.951 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:11.952 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:11.952 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:12.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-03-26T02:52:12.065 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T02:52:12.131 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T02:52:12.187 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T02:52:12.251 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T02:52:12.297 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T02:52:12.395 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T02:52:12.505 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T02:52:12.558 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T02:52:12.560 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:12.657 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:12.708 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:12.709 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:12.709 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:12.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-03-26T02:52:13.321 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:13.325 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T02:52:13.370 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T02:52:13.404 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T02:52:13.419 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:13.452 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T02:52:13.470 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:13.474 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:13.474 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:13.481 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T02:52:13.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-03-26T02:52:14.085 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:14.180 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:14.232 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:14.233 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-03-26T02:52:14.233 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:14.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T02:52:14.291 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T02:52:14.653 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:14.654 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:14.845 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:14.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T02:52:14.995 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 85 k 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:14.996 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:52:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:14.997 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2024-03-26T02:52:14.997 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 231 k 2024-03-26T02:52:14.997 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:52:15.205 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 410 kB/s | 85 kB 00:00 2024-03-26T02:52:15.206 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:15.206 INFO:teuthology.orchestra.run.smithi192.stdout:Total 407 kB/s | 85 kB 00:00 2024-03-26T02:52:15.206 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:52:15.213 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:52:15.213 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:52:15.268 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:52:15.269 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:15.408 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:15.511 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:15.516 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:23 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:15.672 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:15.673 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-26T02:52:15.673 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:52:15.673 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-26T02:52:15.673 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 142 k 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-03-26T02:52:15.674 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:15.675 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 6.8 M 2024-03-26T02:52:15.675 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 33 M 2024-03-26T02:52:15.675 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:16.162 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-26T02:52:16.211 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:16.211 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:16.211 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:16.211 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:16.211 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:16.211 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:16.287 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2475.ge42b8ea3.e 333 kB/s | 142 kB 00:00 2024-03-26T02:52:16.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-03-26T02:52:16.420 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2024-03-26T02:52:16.446 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-03-26T02:52:16.471 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-26T02:52:16.496 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 390 kB/s | 248 kB 00:00 2024-03-26T02:52:16.499 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:16.499 INFO:teuthology.orchestra.run.smithi066.stdout:Total 8.2 MB/s | 6.8 MB 00:00 2024-03-26T02:52:16.512 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:16.523 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:16.523 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:16.619 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:16.619 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:16.914 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:16.988 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:17.085 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T02:52:17.133 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T02:52:17.143 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:17.143 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:17.143 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-03-26T02:52:17.143 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:17.143 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:17.144 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.1 M 2024-03-26T02:52:17.145 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 13 M 2024-03-26T02:52:17.145 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:52:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T02:52:17.683 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 5.7 MB/s | 3.1 MB 00:00 2024-03-26T02:52:17.684 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:17.684 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-26T02:52:17.684 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:52:17.693 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:52:17.693 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:52:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T02:52:17.766 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:52:17.767 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:17.831 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T02:52:17.860 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T02:52:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:18.267 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:18.291 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:18.291 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:52:18.291 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T02:52:18.291 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T02:52:18.291 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T02:52:18.292 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:18.670 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T02:52:18.670 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/6 2024-03-26T02:52:18.670 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T02:52:18.670 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T02:52:18.670 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T02:52:18.670 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T02:52:18.966 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:18.967 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:19.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-fuse 2024-03-26T02:52:19.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:19.410 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:19.410 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:19.410 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:19.410 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:19.410 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:19.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-03-26T02:52:19.787 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:19.943 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:19.943 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:52:19.943 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:19.943 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 890 k 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-26T02:52:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:19.945 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 969 k 2024-03-26T02:52:19.945 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.8 M 2024-03-26T02:52:19.945 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:20.073 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 615 kB/s | 79 kB 00:00 2024-03-26T02:52:20.231 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 02:50:32 AM UTC. 2024-03-26T02:52:20.273 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-fuse-19.0.0-2475.ge42b8ea3.el9.x86_ 2.6 MB/s | 890 kB 00:00 2024-03-26T02:52:20.274 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:20.274 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.9 MB/s | 969 kB 00:00 2024-03-26T02:52:20.275 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:20.297 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:20.297 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:20.375 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:20.375 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:20.385 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-26T02:52:20.385 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:20.385 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-03-26T02:52:20.385 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:20.385 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 171 k 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:20.386 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 171 k 2024-03-26T02:52:20.387 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 494 k 2024-03-26T02:52:20.387 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-26T02:52:20.583 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:20.611 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 760 kB/s | 171 kB 00:00 2024-03-26T02:52:20.612 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:20.613 INFO:teuthology.orchestra.run.smithi192.stdout:Total 755 kB/s | 171 kB 00:00 2024-03-26T02:52:20.613 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-26T02:52:20.621 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-26T02:52:20.621 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-26T02:52:20.679 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-26T02:52:20.680 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-26T02:52:20.729 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T02:52:20.809 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T02:52:20.827 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-26T02:52:20.919 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:21.365 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:21.567 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T02:52:21.567 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/2 2024-03-26T02:52:21.635 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:21.636 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:21.636 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-26T02:52:21.636 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:21.636 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:52:21.636 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-26T02:52:21.859 DEBUG:teuthology.parallel:result is None 2024-03-26T02:52:21.876 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T02:52:21.876 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:21.876 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:21.876 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T02:52:21.876 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:21.876 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:22.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-volume 2024-03-26T02:52:22.787 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:22.942 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:22.943 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:22.943 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-26T02:52:22.943 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:22.943 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 264 k 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 332 k 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.5 M 2024-03-26T02:52:22.944 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:23.069 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 553 kB/s | 69 kB 00:00 2024-03-26T02:52:23.311 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-volume-19.0.0-2475.ge42b8ea3.el9.no 720 kB/s | 264 kB 00:00 2024-03-26T02:52:23.313 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:23.313 INFO:teuthology.orchestra.run.smithi066.stdout:Total 901 kB/s | 332 kB 00:00 2024-03-26T02:52:23.313 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:23.323 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:23.323 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:23.359 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:23.359 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:23.516 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:23.694 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T02:52:23.730 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T02:52:23.767 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T02:52:23.767 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:52:23.768 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T02:52:23.768 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:24.590 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/2 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:24.942 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:25.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados-devel 2024-03-26T02:52:25.884 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:26.044 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 127 k 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:52:26.045 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:26.046 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 127 k 2024-03-26T02:52:26.046 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 456 k 2024-03-26T02:52:26.046 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:26.374 INFO:teuthology.orchestra.run.smithi066.stdout:librados-devel-19.0.0-2475.ge42b8ea3.el9.x86_64 386 kB/s | 127 kB 00:00 2024-03-26T02:52:26.375 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:26.375 INFO:teuthology.orchestra.run.smithi066.stdout:Total 385 kB/s | 127 kB 00:00 2024-03-26T02:52:26.375 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:26.381 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:26.382 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:26.446 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:26.447 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:26.603 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:26.710 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:27.164 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:27.449 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:27.449 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:27.450 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:27.450 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:27.450 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:27.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs2 2024-03-26T02:52:28.296 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:36 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:28.396 INFO:teuthology.orchestra.run.smithi066.stdout:Package libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:28.452 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:28.454 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:28.454 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:28.532 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs-devel 2024-03-26T02:52:29.120 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:29.277 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:29.277 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:29.277 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 32 k 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 32 k 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 142 k 2024-03-26T02:52:29.278 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:29.437 INFO:teuthology.orchestra.run.smithi066.stdout:libcephfs-devel-19.0.0-2475.ge42b8ea3.el9.x86_6 201 kB/s | 32 kB 00:00 2024-03-26T02:52:29.437 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:29.437 INFO:teuthology.orchestra.run.smithi066.stdout:Total 200 kB/s | 32 kB 00:00 2024-03-26T02:52:29.437 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:29.440 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:29.440 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:29.466 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:29.467 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:29.602 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:29.695 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:30.082 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:30.349 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:30.349 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:30.349 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:30.349 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:30.349 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:30.349 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:30.535 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados2 2024-03-26T02:52:31.124 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:31.223 INFO:teuthology.orchestra.run.smithi066.stdout:Package librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:31.278 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:31.279 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:31.279 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:31.335 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librbd1 2024-03-26T02:52:31.923 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:32.024 INFO:teuthology.orchestra.run.smithi066.stdout:Package librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:32.079 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:32.080 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:32.080 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:32.135 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rados 2024-03-26T02:52:32.722 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:32.822 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:32.876 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:32.877 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:32.878 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:32.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rgw 2024-03-26T02:52:33.529 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:33.628 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:33.685 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:33.686 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:33.686 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:33.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-cephfs 2024-03-26T02:52:34.331 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:34.431 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:34.488 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:34.489 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:34.489 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:34.539 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rbd 2024-03-26T02:52:35.126 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:35.225 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T02:52:35.280 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:35.281 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-26T02:52:35.281 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:35.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-fuse 2024-03-26T02:52:36.003 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:36.159 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:36.159 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:36.159 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-26T02:52:36.159 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:36.159 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:36.159 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 85 k 2024-03-26T02:52:36.160 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:36.160 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:36.160 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:36.160 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:52:36.160 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 85 k 2024-03-26T02:52:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 231 k 2024-03-26T02:52:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:36.382 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-fuse-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 388 kB/s | 85 kB 00:00 2024-03-26T02:52:36.382 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:36.383 INFO:teuthology.orchestra.run.smithi066.stdout:Total 385 kB/s | 85 kB 00:00 2024-03-26T02:52:36.383 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:36.389 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:36.390 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:36.449 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:36.449 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:36.607 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:36.719 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:37.099 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:37.422 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:37.423 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:37.423 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:37.423 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:37.423 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:37.423 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:37.649 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror 2024-03-26T02:52:38.261 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:38.422 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:52:38.423 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:38.424 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 3.1 M 2024-03-26T02:52:38.424 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 13 M 2024-03-26T02:52:38.424 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:39.059 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-mirror-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 4.9 MB/s | 3.1 MB 00:00 2024-03-26T02:52:39.059 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:39.060 INFO:teuthology.orchestra.run.smithi066.stdout:Total 4.9 MB/s | 3.1 MB 00:00 2024-03-26T02:52:39.060 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:39.070 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:39.071 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:39.148 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:39.148 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:39.599 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:39.660 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:39.686 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:39.686 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T02:52:39.686 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T02:52:39.687 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T02:52:39.687 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T02:52:39.687 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:40.888 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:40.888 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:40.888 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:40.888 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:40.888 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:40.888 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:41.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-nbd 2024-03-26T02:52:41.730 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 02:50:52 AM UTC. 2024-03-26T02:52:41.885 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 171 k 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-26T02:52:41.886 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:41.887 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 171 k 2024-03-26T02:52:41.887 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 494 k 2024-03-26T02:52:41.887 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-26T02:52:42.115 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-nbd-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 760 kB/s | 171 kB 00:00 2024-03-26T02:52:42.115 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:52:42.115 INFO:teuthology.orchestra.run.smithi066.stdout:Total 754 kB/s | 171 kB 00:00 2024-03-26T02:52:42.115 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-26T02:52:42.121 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-26T02:52:42.121 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-26T02:52:42.182 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-26T02:52:42.182 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-26T02:52:42.340 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-26T02:52:42.457 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:42.876 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:43.181 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T02:52:43.182 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:43.182 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-26T02:52:43.182 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T02:52:43.182 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:52:43.182 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-26T02:52:43.411 DEBUG:teuthology.parallel:result is None 2024-03-26T02:52:43.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:52:43.558 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T02:52:43.580 INFO:teuthology.orchestra.run.smithi066.stdout:19.0.0-2475.ge42b8ea3.el9 2024-03-26T02:52:43.581 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2475.ge42b8ea3.el9 2024-03-26T02:52:43.581 INFO:teuthology.task.install:The correct ceph version 19.0.0-2475.ge42b8ea3 is installed. 2024-03-26T02:52:43.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:52:43.723 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T02:52:43.750 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-2475.ge42b8ea3.el9 2024-03-26T02:52:43.750 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2475.ge42b8ea3.el9 2024-03-26T02:52:43.750 INFO:teuthology.task.install:The correct ceph version 19.0.0-2475.ge42b8ea3 is installed. 2024-03-26T02:52:43.752 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T02:52:43.752 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:52:43.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T02:52:43.785 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:52:43.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T02:52:43.823 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T02:52:43.823 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:52:43.823 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T02:52:43.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T02:52:43.928 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:52:43.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T02:52:43.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T02:52:44.028 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T02:52:44.028 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:52:44.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T02:52:44.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T02:52:44.132 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:52:44.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T02:52:44.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T02:52:44.232 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T02:52:44.232 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:52:44.232 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T02:52:44.263 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T02:52:44.334 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:52:44.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T02:52:44.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T02:52:44.432 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T02:52:44.533 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'OSDMAP_FLAGS', 'PG_DEGRADED', 'OSD_DOWN', 'POOL_APP_NOT_ENABLED', 'CACHE_POOL_NO_HIT_SET', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'cephadm_mode': 'root'} 2024-03-26T02:52:44.533 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:52:44.534 INFO:tasks.cephadm:Cluster fsid is eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:52:44.534 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T02:52:44.534 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.66:6789]', 'mon.c': '[v1:172.21.15.66:6790]', 'mon.b': '[v1:172.21.15.192:6789]'} 2024-03-26T02:52:44.534 INFO:tasks.cephadm:First mon is mon.a on smithi066 2024-03-26T02:52:44.534 INFO:tasks.cephadm:First mgr is y 2024-03-26T02:52:44.534 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T02:52:44.534 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname $(hostname -s) 2024-03-26T02:52:44.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-03-26T02:52:44.595 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T02:52:44.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:52:44.734 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77960/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2475-ge42b8ea3', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2475.ge42b8ea3'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-26 00:31:23.988041', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-03-25-1628', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T02:52:44.873 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-adk-testing-2024-03-25-1628, sha1 e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:52:44.874 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/x86_64/flavors/default/cephadm 2024-03-26T02:52:44.874 DEBUG:teuthology.orchestra.run.smithi066:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:52:45.586 INFO:teuthology.orchestra.run.smithi066.stdout:-rw-r--r--. 1 ubuntu ubuntu 781509 Mar 26 02:52 /home/ubuntu/cephtest/cephadm 2024-03-26T02:52:45.586 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:52:46.093 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 781509 Mar 26 02:52 /home/ubuntu/cephtest/cephadm 2024-03-26T02:52:46.094 DEBUG:teuthology.orchestra.run.smithi066:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:52:46.113 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:52:46.172 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 on all hosts... 2024-03-26T02:52:46.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 pull 2024-03-26T02:52:46.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 pull 2024-03-26T02:52:47.019 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T02:52:47.115 INFO:teuthology.orchestra.run.smithi066.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T02:53:15.449 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T02:53:15.450 INFO:teuthology.orchestra.run.smithi066.stdout: "ceph_version": "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)", 2024-03-26T02:53:15.450 INFO:teuthology.orchestra.run.smithi066.stdout: "image_id": "f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354", 2024-03-26T02:53:15.450 INFO:teuthology.orchestra.run.smithi066.stdout: "repo_digests": [ 2024-03-26T02:53:15.450 INFO:teuthology.orchestra.run.smithi066.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717" 2024-03-26T02:53:15.450 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T02:53:15.450 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T02:53:19.760 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T02:53:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)", 2024-03-26T02:53:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354", 2024-03-26T02:53:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-03-26T02:53:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717" 2024-03-26T02:53:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T02:53:19.761 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T02:53:19.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph 2024-03-26T02:53:19.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-03-26T02:53:19.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /etc/ceph 2024-03-26T02:53:19.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-03-26T02:53:19.930 INFO:tasks.cephadm:Writing seed config... 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T02:53:19.931 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-26T02:53:19.932 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-26T02:53:19.933 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T02:53:19.933 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-26T02:53:19.933 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:53:19.934 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T02:53:19.961 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eba7a548-eb1b-11ee-95d0-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-26T02:53:19.962 DEBUG:teuthology.orchestra.run.smithi066:mon.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T02:53:20.004 DEBUG:teuthology.orchestra.run.smithi066:mgr.y> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T02:53:20.046 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T02:53:20.046 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -v bootstrap --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.66:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:53:20.254 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:53:20.254 INFO:teuthology.orchestra.run.smithi066.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', '-v', 'bootstrap', '--fsid', 'eba7a548-eb1b-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.66:6789]', '--skip-admin-label'] 2024-03-26T02:53:20.255 INFO:teuthology.orchestra.run.smithi066.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T02:53:20.255 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying podman|docker is present... 2024-03-26T02:53:20.280 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T02:53:20.280 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying lvm2 is present... 2024-03-26T02:53:20.281 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying time synchronization is in place... 2024-03-26T02:53:20.288 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T02:53:20.289 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T02:53:20.295 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T02:53:20.295 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout inactive 2024-03-26T02:53:20.301 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout enabled 2024-03-26T02:53:20.307 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout active 2024-03-26T02:53:20.307 INFO:teuthology.orchestra.run.smithi066.stdout:Unit chronyd.service is enabled and running 2024-03-26T02:53:20.307 INFO:teuthology.orchestra.run.smithi066.stdout:Repeating the final host check... 2024-03-26T02:53:20.331 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T02:53:20.331 INFO:teuthology.orchestra.run.smithi066.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T02:53:20.331 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl is present 2024-03-26T02:53:20.331 INFO:teuthology.orchestra.run.smithi066.stdout:lvcreate is present 2024-03-26T02:53:20.338 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T02:53:20.338 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T02:53:20.344 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T02:53:20.344 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout inactive 2024-03-26T02:53:20.350 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout enabled 2024-03-26T02:53:20.357 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout active 2024-03-26T02:53:20.357 INFO:teuthology.orchestra.run.smithi066.stdout:Unit chronyd.service is enabled and running 2024-03-26T02:53:20.357 INFO:teuthology.orchestra.run.smithi066.stdout:Host looks OK 2024-03-26T02:53:20.357 INFO:teuthology.orchestra.run.smithi066.stdout:Cluster fsid: eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:20.357 INFO:teuthology.orchestra.run.smithi066.stdout:Acquiring lock 139953053921968 on /run/cephadm/eba7a548-eb1b-11ee-95d0-87774f69a715.lock 2024-03-26T02:53:20.358 INFO:teuthology.orchestra.run.smithi066.stdout:Lock 139953053921968 acquired on /run/cephadm/eba7a548-eb1b-11ee-95d0-87774f69a715.lock 2024-03-26T02:53:20.358 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying IP 172.21.15.66 port 6789 ... 2024-03-26T02:53:20.358 INFO:teuthology.orchestra.run.smithi066.stdout:Base mon IP(s) is [172.21.15.66:6789], mon addrv is [v1:172.21.15.66:6789] 2024-03-26T02:53:20.361 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.66 metric 100 2024-03-26T02:53:20.361 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.66 metric 100 2024-03-26T02:53:20.363 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T02:53:20.363 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T02:53:20.363 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1773sec hoplimit 64 pref medium 2024-03-26T02:53:20.366 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T02:53:20.366 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T02:53:20.366 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T02:53:20.366 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T02:53:20.366 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8bd/64 scope link 2024-03-26T02:53:20.366 INFO:teuthology.orchestra.run.smithi066.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T02:53:20.367 INFO:teuthology.orchestra.run.smithi066.stdout:Mon IP `172.21.15.66` is in CIDR network `172.21.0.0/20` 2024-03-26T02:53:20.367 INFO:teuthology.orchestra.run.smithi066.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-26T02:53:20.367 INFO:teuthology.orchestra.run.smithi066.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T02:53:20.368 INFO:teuthology.orchestra.run.smithi066.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stdout f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stderr Copying blob sha256:225af1ab3ea34577a62e5c6f6ac1e3381fc0d9eac77fbb4e31c233b967549b54 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stderr Copying config sha256:f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 2024-03-26T02:53:20.876 INFO:teuthology.orchestra.run.smithi066.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T02:53:21.815 INFO:teuthology.orchestra.run.smithi066.stdout:ceph: stdout ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev) 2024-03-26T02:53:21.815 INFO:teuthology.orchestra.run.smithi066.stdout:Ceph version: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev) 2024-03-26T02:53:21.815 INFO:teuthology.orchestra.run.smithi066.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T02:53:22.821 INFO:teuthology.orchestra.run.smithi066.stdout:stat: stdout 167 167 2024-03-26T02:53:22.821 INFO:teuthology.orchestra.run.smithi066.stdout:Creating initial keys... 2024-03-26T02:53:23.742 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph-authtool: stdout AQCjOAJmfHFPCRAAPKlE/NiC4vVPZ4oPLduLVg== 2024-03-26T02:53:24.609 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph-authtool: stdout AQCkOAJmxBinBBAAdlrdef/BSYHIdr3+AuA6DQ== 2024-03-26T02:53:25.646 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph-authtool: stdout AQCkOAJmXYMeOBAA2j+ukmTSQQMhXzgukSebbw== 2024-03-26T02:53:25.646 INFO:teuthology.orchestra.run.smithi066.stdout:Creating initial monmap... 2024-03-26T02:53:26.510 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool for a [v1:172.21.15.66:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:setting min_mon_release = quincy 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/monmaptool: set fsid to eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:53:26.511 INFO:teuthology.orchestra.run.smithi066.stdout:Creating mon... 2024-03-26T02:53:27.358 INFO:teuthology.orchestra.run.smithi066.stdout:create mon.a on 2024-03-26T02:53:27.578 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T02:53:27.742 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T02:53:27.938 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715.target. 2024-03-26T02:53:27.938 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715.target. 2024-03-26T02:53:28.187 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a 2024-03-26T02:53:28.187 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T02:53:28.368 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715.target.wants/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@.service. 2024-03-26T02:53:28.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 podman[29276]: 2024-03-26 02:53:28.51732213 +0000 UTC m=+0.018446644 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:53:28.907 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:53:28.907 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout disabled 2024-03-26T02:53:28.915 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:53:28.915 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout inactive 2024-03-26T02:53:28.915 INFO:teuthology.orchestra.run.smithi066.stdout:firewalld.service is not enabled 2024-03-26T02:53:28.915 INFO:teuthology.orchestra.run.smithi066.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T02:53:28.915 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for mon to start... 2024-03-26T02:53:28.915 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for mon... 2024-03-26T02:53:28.997 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 podman[29276]: 2024-03-26 02:53:28.817286758 +0000 UTC m=+0.318411252 container init 13051836c284f0985403590bdc09c712d29476e53fee75bd58283b27005d5293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T02:53:28.997 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 podman[29276]: 2024-03-26 02:53:28.820393956 +0000 UTC m=+0.321518448 container start 13051836c284f0985403590bdc09c712d29476e53fee75bd58283b27005d5293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:53:28.997 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 ceph-mon[29295]: mkfs eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:28.997 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 ceph-mon[29295]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:53:28.997 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 bash[29276]: 13051836c284f0985403590bdc09c712d29476e53fee75bd58283b27005d5293 2024-03-26T02:53:28.997 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:28 smithi066 systemd[1]: Started Ceph mon.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:53:29.985 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout id: eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout services: 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.556692s) 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T02:53:29.987 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout data: 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:mon is available 2024-03-26T02:53:29.988 INFO:teuthology.orchestra.run.smithi066.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: monmap epoch 1 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: last_changed 2024-03-26T02:53:25.962264+0000 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: min_mon_release 19 (squid) 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: election_strategy: 1 2024-03-26T02:53:30.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:53:30.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: fsmap 2024-03-26T02:53:30.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T02:53:30.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: mgrmap e1: no daemons active 2024-03-26T02:53:30.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:29 smithi066 ceph-mon[29295]: from='client.? v1:172.21.15.66:0/842931709' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:53:31.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:30 smithi066 ceph-mon[29295]: from='client.? v1:172.21.15.66:0/1307124863' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T02:53:31.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:30 smithi066 ceph-mon[29295]: from='client.? v1:172.21.15.66:0/1307124863' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout fsid = eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.66:6789] 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T02:53:31.084 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T02:53:31.085 INFO:teuthology.orchestra.run.smithi066.stdout:Generating new minimal ceph.conf... 2024-03-26T02:53:32.130 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:31 smithi066 ceph-mon[29295]: from='client.? v1:172.21.15.66:0/1327682510' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:32.130 INFO:teuthology.orchestra.run.smithi066.stdout:Restarting the monitor... 2024-03-26T02:53:32.382 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:32 smithi066 systemd[1]: Stopping Ceph mon.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T02:53:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29291]: 2024-03-26T02:53:32.380+0000 7f169f0d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:53:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29291]: 2024-03-26T02:53:32.380+0000 7f169f0d7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T02:53:32.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:32 smithi066 podman[29541]: 2024-03-26 02:53:32.626930192 +0000 UTC m=+0.334389570 container died 13051836c284f0985403590bdc09c712d29476e53fee75bd58283b27005d5293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T02:53:33.157 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:32 smithi066 podman[29541]: 2024-03-26 02:53:32.963928659 +0000 UTC m=+0.671388034 container cleanup 13051836c284f0985403590bdc09c712d29476e53fee75bd58283b27005d5293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:53:33.157 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:32 smithi066 bash[29541]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:53:33.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 podman[29555]: 2024-03-26 02:53:33.15636794 +0000 UTC m=+0.530978286 container remove 13051836c284f0985403590bdc09c712d29476e53fee75bd58283b27005d5293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T02:53:33.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T02:53:33.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 systemd[1]: Stopped Ceph mon.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:53:33.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 systemd[1]: Starting Ceph mon.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T02:53:33.815 INFO:teuthology.orchestra.run.smithi066.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 podman[29647]: 2024-03-26 02:53:33.576037479 +0000 UTC m=+0.104405564 container create 858570b43b8513babaeb250bfa0924b4121f20ba00b9b7794b6ae7bac41c2ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD) 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 podman[29647]: 2024-03-26 02:53:33.490561594 +0000 UTC m=+0.018929647 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 podman[29647]: 2024-03-26 02:53:33.75309161 +0000 UTC m=+0.281459654 container init 858570b43b8513babaeb250bfa0924b4121f20ba00b9b7794b6ae7bac41c2ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 podman[29647]: 2024-03-26 02:53:33.755988992 +0000 UTC m=+0.284357039 container start 858570b43b8513babaeb250bfa0924b4121f20ba00b9b7794b6ae7bac41c2ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T02:53:33.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: pidfile_write: ignore empty --pid-file 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: load: jerasure load: lrc 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Git sha 0 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: DB SUMMARY 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: DB Session ID: 8J8OTD42PC0APJJ3OKHP 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T02:53:33.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 93181 ; 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:53:33.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.env: 0x561bdda28c60 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.info_log: 0x561bdf4f3b20 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.statistics: (nil) 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.use_fsync: 0 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:53:33.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.db_log_dir: 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.wal_dir: 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:53:33.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.write_buffer_manager: 0x561bdf4c41e0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.unordered_write: 0 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:53:33.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.row_cache: None 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.wal_filter: None 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.wal_compression: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:53:33.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:53:33.865 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_open_files: -1 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Compression algorithms supported: 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kZSTD supported: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:53:33.866 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.merge_operator: 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_filter: None 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:53:33.867 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561bdf4f3d00) 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: index_type: 0 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: data_block_index_type: 0 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: index_shortening: 1 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: checksum: 4 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: no_block_cache: 0 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_cache: 0x561bdf4f7090 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_cache_options: 2024-03-26T02:53:33.868 INFO:journalctl@ceph.mon.a.smithi066.stdout: capacity : 536870912 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: num_shard_bits : 4 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: strict_capacity_limit : 0 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_cache_compressed: (nil) 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: persistent_cache: (nil) 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_size: 4096 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_size_deviation: 10 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_restart_interval: 16 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: index_block_restart_interval: 1 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: metadata_block_size: 4096 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: partition_filters: 0 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: use_delta_encoding: 1 2024-03-26T02:53:33.869 INFO:journalctl@ceph.mon.a.smithi066.stdout: filter_policy: bloomfilter 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: whole_key_filtering: 1 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: verify_compression: 0 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: format_version: 5 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: enable_index_compression: 1 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: block_align: 0 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: max_auto_readahead_size: 262144 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: prepopulate_block_cache: 0 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression: NoCompression 2024-03-26T02:53:33.870 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.num_levels: 7 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:53:33.871 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:53:33.872 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:53:33.873 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:53:33.874 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:53:33.875 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.ttl: 2592000 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:53:33.876 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a90f410d-3e06-4050-8379-310ae93d89e5 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421613785650, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421613786624, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 89093, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 286, "table_properties": {"data_size": 87155, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11865, "raw_average_key_size": 48, "raw_value_size": 80634, "raw_average_value_size": 329, "num_data_blocks": 11, "num_entries": 245, "num_filter_entries": 245, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711421613, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a90f410d-3e06-4050-8379-310ae93d89e5", "db_session_id": "8J8OTD42PC0APJJ3OKHP", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421613786713, "job": 1, "event": "recovery_finished"} 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T02:53:33.877 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561bdf5e6000 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: DB pointer 0x561bdf5d0000 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: ** DB Stats ** 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:53:33.878 INFO:journalctl@ceph.mon.a.smithi066.stdout: 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: L0 2/0 88.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: Sum 2/0 88.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: 2024-03-26T02:53:33.879 INFO:journalctl@ceph.mon.a.smithi066.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Cumulative compaction: 0.00 GB write, 28.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Interval compaction: 0.00 GB write, 28.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Block cache BinnedLRUCache@0x561bdf4f7090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T02:53:33.880 INFO:journalctl@ceph.mon.a.smithi066.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: starting mon.a rank 0 at public addrs v1:172.21.15.66:6789/0 at bind addrs v1:172.21.15.66:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???) e1 preinit fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).mds e1 new map 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).mds e1 print_map 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: e1 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: legacy client fscid: -1 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout: No filesystems configured 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:53:33.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 bash[29647]: 858570b43b8513babaeb250bfa0924b4121f20ba00b9b7794b6ae7bac41c2ccf 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 systemd[1]: Started Ceph mon.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: monmap epoch 1 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: last_changed 2024-03-26T02:53:25.962264+0000 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: min_mon_release 19 (squid) 2024-03-26T02:53:33.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: election_strategy: 1 2024-03-26T02:53:33.883 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:53:33.883 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: fsmap 2024-03-26T02:53:33.883 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T02:53:33.883 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:33 smithi066 ceph-mon[29666]: mgrmap e1: no daemons active 2024-03-26T02:53:34.988 INFO:teuthology.orchestra.run.smithi066.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T02:53:35.010 INFO:teuthology.orchestra.run.smithi066.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:53:35.010 INFO:teuthology.orchestra.run.smithi066.stdout:Creating mgr... 2024-03-26T02:53:35.010 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T02:53:35.011 INFO:teuthology.orchestra.run.smithi066.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T02:53:35.264 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y 2024-03-26T02:53:35.265 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T02:53:35.419 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715.target.wants/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@.service. 2024-03-26T02:53:35.757 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:35 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4032438926' entity='client.admin' 2024-03-26T02:53:35.953 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:53:35.953 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout disabled 2024-03-26T02:53:35.960 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:53:35.960 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout inactive 2024-03-26T02:53:35.960 INFO:teuthology.orchestra.run.smithi066.stdout:firewalld.service is not enabled 2024-03-26T02:53:35.960 INFO:teuthology.orchestra.run.smithi066.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T02:53:35.969 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:53:35.969 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout disabled 2024-03-26T02:53:35.976 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:53:35.976 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout inactive 2024-03-26T02:53:35.976 INFO:teuthology.orchestra.run.smithi066.stdout:firewalld.service is not enabled 2024-03-26T02:53:35.976 INFO:teuthology.orchestra.run.smithi066.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T02:53:35.976 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for mgr to start... 2024-03-26T02:53:35.977 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for mgr... 2024-03-26T02:53:36.532 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:36 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:36.438+0000 7fdff34cc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:53:36.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:36 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2702784757' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:53:36.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:36 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:36.551+0000 7fdff34cc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:53:37.573 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:37.231+0000 7fdff34cc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:53:37.573 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:37.327+0000 7fdff34cc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:53:37.573 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:37.473+0000 7fdff34cc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:53:37.573 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:37.572+0000 7fdff34cc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:53:37.763 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:53:37.764 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:53:37.768 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:53:28.887484+0000", 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:53:37.769 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:53:37.770 INFO:teuthology.orchestra.run.smithi066.stdout:mgr not available, waiting (1/15)... 2024-03-26T02:53:37.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:37.658+0000 7fdff34cc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:53:38.238 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:37.934+0000 7fdff34cc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:53:38.239 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:38.049+0000 7fdff34cc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:53:38.239 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:38.141+0000 7fdff34cc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:53:38.239 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:38.237+0000 7fdff34cc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:53:38.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:38.419+0000 7fdff34cc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:53:39.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:39 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:39.448+0000 7fdff34cc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:53:39.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:39 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:39.583+0000 7fdff34cc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:53:40.228 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:39 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:39.932+0000 7fdff34cc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:53:40.228 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:40 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:40.027+0000 7fdff34cc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:53:40.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:40 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4271461243' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:53:40.609 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:40 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:40.236+0000 7fdff34cc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:53:40.609 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:40 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:40.435+0000 7fdff34cc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:53:40.990 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:40 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:40.732+0000 7fdff34cc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:53:40.995 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:53:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:40.997 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:53:40.997 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:53:40.997 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:53:40.997 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:53:40.997 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:53:40.997 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:53:41.000 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:41.000 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:53:41.000 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:53:41.000 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:53:41.001 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:53:28.887484+0000", 2024-03-26T02:53:41.002 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:53:41.003 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:41.003 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:53:41.003 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:53:41.003 INFO:teuthology.orchestra.run.smithi066.stdout:mgr not available, waiting (2/15)... 2024-03-26T02:53:41.251 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:41 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:41.003+0000 7fdff34cc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:53:41.251 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:41 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:41.249+0000 7fdff34cc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:53:42.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:42 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:42.084+0000 7fdff34cc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:53:43.001 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:42 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:42.675+0000 7fdff34cc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:53:43.001 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:42 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:42.776+0000 7fdff34cc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:53:43.001 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:43.000+0000 7fdff34cc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:53:43.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: Activating manager daemon y 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: mgrmap e2: y(active, starting, since 0.00341025s) 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: Manager daemon y is now available 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' 2024-03-26T02:53:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' 2024-03-26T02:53:43.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:43 smithi066 ceph-mon[29666]: from='mgr.14100 v1:172.21.15.66:0/2029315996' entity='mgr.y' 2024-03-26T02:53:45.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:45 smithi066 ceph-mon[29666]: mgrmap e3: y(active, since 1.00767s) 2024-03-26T02:53:46.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1205987449' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:53:47.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:47 smithi066 ceph-mon[29666]: mgrmap e4: y(active, since 3s) 2024-03-26T02:53:51.321 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:51.327 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:53:51.327 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T02:53:51.327 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T02:53:51.327 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T02:53:51.327 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T02:53:51.327 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 0 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T02:53:51.328 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T02:53:51.329 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T02:53:51.330 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ], 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T02:53:28.887484+0000", 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout }, 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:53:51.331 INFO:teuthology.orchestra.run.smithi066.stdout:mgr is available 2024-03-26T02:53:53.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3853110913' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout fsid = eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T02:53:54.007 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 2024-03-26T02:53:54.008 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T02:53:54.008 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T02:53:54.008 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T02:53:54.008 INFO:teuthology.orchestra.run.smithi066.stdout:Enabling cephadm module... 2024-03-26T02:53:55.026 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1074554281' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T02:53:55.026 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:54 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-03-26T02:53:55.026 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:54 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-03-26T02:53:55.538 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:55 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:55.280+0000 7fdadf373200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:53:55.538 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:55 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:55.382+0000 7fdadf373200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:53:55.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1074554281' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T02:53:55.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:55 smithi066 ceph-mon[29666]: mgrmap e5: y(active, since 11s) 2024-03-26T02:53:56.315 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.076+0000 7fdadf373200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:53:56.315 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.174+0000 7fdadf373200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:53:56.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.326+0000 7fdadf373200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:53:56.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.426+0000 7fdadf373200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:53:56.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.513+0000 7fdadf373200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:53:56.653 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:53:56.653 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T02:53:56.653 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T02:53:56.653 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T02:53:56.653 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T02:53:56.654 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:53:56.654 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for the mgr to restart... 2024-03-26T02:53:56.654 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for mgr epoch 5... 2024-03-26T02:53:56.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1351109101' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T02:53:56.899 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.801+0000 7fdadf373200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:53:57.294 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:56 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:56.918+0000 7fdadf373200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:53:57.294 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:57 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:57.008+0000 7fdadf373200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:53:57.294 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:57 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:57.108+0000 7fdadf373200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:53:57.294 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:57 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:57.293+0000 7fdadf373200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:53:58.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:58 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:58.314+0000 7fdadf373200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:53:58.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:58 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:58.450+0000 7fdadf373200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:53:59.097 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:58 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:58.797+0000 7fdadf373200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:53:59.097 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:58 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:58.890+0000 7fdadf373200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:53:59.097 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:59 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:59.096+0000 7fdadf373200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:53:59.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:59 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:59.289+0000 7fdadf373200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:53:59.855 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:59 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:59.586+0000 7fdadf373200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:53:59.855 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:53:59 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:53:59.854+0000 7fdadf373200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:54:00.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:00.102+0000 7fdadf373200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:54:01.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:00.944+0000 7fdadf373200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:54:01.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:01.538+0000 7fdadf373200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:54:01.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:01.640+0000 7fdadf373200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:54:02.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:01.865+0000 7fdadf373200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: Active manager daemon y restarted 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: Activating manager daemon y 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: mgrmap e6: y(active, starting, since 0.0031973s) 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: Manager daemon y is now available 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:01 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: Found migration_current of "None". Setting to last migration. 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:03.189 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:02 smithi066 ceph-mon[29666]: mgrmap e7: y(active, since 1.00491s) 2024-03-26T02:54:03.459 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:54:03.459 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T02:54:03.459 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T02:54:03.459 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:54:03.459 INFO:teuthology.orchestra.run.smithi066.stdout:mgr epoch 5 is available 2024-03-26T02:54:03.459 INFO:teuthology.orchestra.run.smithi066.stdout:Setting orchestrator backend to cephadm... 2024-03-26T02:54:04.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:03 smithi066 ceph-mon[29666]: from='client.14122 v1:172.21.15.66:0/2158046361' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T02:54:04.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:03 smithi066 ceph-mon[29666]: from='client.14122 v1:172.21.15.66:0/2158046361' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: mgrmap e8: y(active, since 2s) 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: from='client.14130 v1:172.21.15.66:0/186282710' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:04] ENGINE Bus STARTING 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:04] ENGINE Serving on http://172.21.15.66:8765 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:04] ENGINE Serving on https://172.21.15.66:7150 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:04] ENGINE Bus STARTED 2024-03-26T02:54:05.032 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:04 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:05.930 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T02:54:05.931 INFO:teuthology.orchestra.run.smithi066.stdout:Generating ssh key... 2024-03-26T02:54:06.246 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:05 smithi066 ceph-mon[29666]: from='client.14132 v1:172.21.15.66:0/3733789804' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:06 smithi066 ceph-mon[29666]: from='client.14134 v1:172.21.15.66:0/1572520586' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:06 smithi066 ceph-mon[29666]: Generating ssh key... 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: Generating public/private rsa key pair. 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: Your identification has been saved in /tmp/tmphggjrmxc/key. 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: Your public key has been saved in /tmp/tmphggjrmxc/key.pub. 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: The key fingerprint is: 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: SHA256:3sfB7AGw1eGbxcm4S+iOKTbgHRTNIZzhg5HsZ2lCjIw ceph-eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: The key's randomart image is: 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: +---[RSA 3072]----+ 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | o +.oo=o..... | 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: |E o =++.o+ ..+ . | 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | o. oo. . o = | 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | o *. = = | 2024-03-26T02:54:07.359 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | * S . X | 2024-03-26T02:54:07.360 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | . .. o + + | 2024-03-26T02:54:07.360 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | . o .. o = | 2024-03-26T02:54:07.360 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | . = + . | 2024-03-26T02:54:07.360 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: | . oo . | 2024-03-26T02:54:07.360 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:07 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: +----[SHA256]-----+ 2024-03-26T02:54:08.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:08 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:08.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:08 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:08.809 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYvyLYefIyb5xftgwgSP6ta2iWofhVQYxDlT4kuO8BpdZIIiISi6wRFLuOz35IT7jic2fYEIh7Tg1DXe9O26CfnJPbimM17ezhXDieTbsCsOaY/oX8Ckd0L1fogEraT5MfZDXGzif/83tJsGLfkFWjgMVe7XcBJqr3K/1J6lkRvJ7r/aDLLRgxRwsis3xB2Wco6TCztWj70ShMh8IMBOaeQTTo+W+lBnnP19fLjJuReVVNobRGseZAZqk4RC8h9L6AqcK1v1/qtor1+p6bv/Mbz1+UjFWjy8/YrRAhDeaqlhwoaOIIaRHvI+/bwFnZwKM1dhx7vGVw+e6tKfMtaQn/gwnGZIw3g8n/cIiQAgTHsAhWsKnkJeT/QEvP4HMn99hAzJ8TDlqlvGw8bzgof+itQ/2j70dkKUmWxeqiKXEZcxI/yVlzGMCoxKFd8nYp7zuNcloj7ddw4xw+NKeKVy+Xz4qPI4wB2ViiH+8ciUKcVmUs8yz+l0geSraV1wzwgXc= ceph-eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:54:08.809 INFO:teuthology.orchestra.run.smithi066.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:54:08.809 INFO:teuthology.orchestra.run.smithi066.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T02:54:08.810 INFO:teuthology.orchestra.run.smithi066.stdout:Adding host smithi066... 2024-03-26T02:54:09.217 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:09 smithi066 ceph-mon[29666]: from='client.14136 v1:172.21.15.66:0/4244618269' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:10.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:10 smithi066 ceph-mon[29666]: from='client.14138 v1:172.21.15.66:0/2822832754' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi066", "addr": "172.21.15.66", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:11.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:11 smithi066 ceph-mon[29666]: Deploying cephadm binary to smithi066 2024-03-26T02:54:14.809 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout Added host 'smithi066' with addr '172.21.15.66' 2024-03-26T02:54:14.809 INFO:teuthology.orchestra.run.smithi066.stdout:Deploying unmanaged mon service... 2024-03-26T02:54:15.406 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:15 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:15.406 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:15 smithi066 ceph-mon[29666]: Added host smithi066 2024-03-26T02:54:15.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:15 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:16.050 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T02:54:16.050 INFO:teuthology.orchestra.run.smithi066.stdout:Deploying unmanaged mgr service... 2024-03-26T02:54:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:16 smithi066 ceph-mon[29666]: from='client.14140 v1:172.21.15.66:0/1125422194' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:16 smithi066 ceph-mon[29666]: Saving service mon spec with placement count:5 2024-03-26T02:54:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:16 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:17.578 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T02:54:17.952 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:17 smithi066 ceph-mon[29666]: from='client.14142 v1:172.21.15.66:0/3931798775' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:17.952 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:17 smithi066 ceph-mon[29666]: Saving service mgr spec with placement count:2 2024-03-26T02:54:17.952 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:17 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:19.266 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:18 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:19.266 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:18 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:19.266 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:18 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/650619871' entity='client.admin' 2024-03-26T02:54:21.009 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:20 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3870944075' entity='client.admin' 2024-03-26T02:54:21.009 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:20 smithi066 ceph-mon[29666]: from='mgr.14118 v1:172.21.15.66:0/2379000903' entity='mgr.y' 2024-03-26T02:54:21.011 INFO:teuthology.orchestra.run.smithi066.stdout:Enabling the dashboard module... 2024-03-26T02:54:22.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:21 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1692448527' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T02:54:22.225 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:21 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-03-26T02:54:22.225 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:21 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-03-26T02:54:22.755 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:22.487+0000 7fd93a5ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:54:22.755 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:22.593+0000 7fd93a5ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:54:23.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:22 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1692448527' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T02:54:23.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:22 smithi066 ceph-mon[29666]: mgrmap e9: y(active, since 20s) 2024-03-26T02:54:23.522 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:23.274+0000 7fd93a5ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:54:23.522 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:23.370+0000 7fd93a5ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:54:23.522 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:23.521+0000 7fd93a5ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:54:23.780 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:23.621+0000 7fd93a5ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:54:23.780 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:23.709+0000 7fd93a5ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for the mgr to restart... 2024-03-26T02:54:23.784 INFO:teuthology.orchestra.run.smithi066.stdout:Waiting for mgr epoch 9... 2024-03-26T02:54:24.083 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2478051278' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T02:54:24.083 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:24 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:24.000+0000 7fd93a5ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:54:24.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:24 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:24.116+0000 7fd93a5ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:54:24.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:24 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:24.210+0000 7fd93a5ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:54:24.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:24 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:24.307+0000 7fd93a5ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:54:24.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:24 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:24.490+0000 7fd93a5ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:54:25.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:25 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:25.507+0000 7fd93a5ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:54:25.859 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:25 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:25.643+0000 7fd93a5ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:54:26.291 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:25 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:25.990+0000 7fd93a5ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:54:26.291 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:26 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:26.083+0000 7fd93a5ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:54:26.291 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:26 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:26.289+0000 7fd93a5ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:54:26.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:26 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:26.482+0000 7fd93a5ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:54:27.051 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:26 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:26.780+0000 7fd93a5ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:54:27.051 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:27.049+0000 7fd93a5ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:54:27.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:27.297+0000 7fd93a5ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:54:28.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:28 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:28.139+0000 7fd93a5ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:54:29.065 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:28 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:28.733+0000 7fd93a5ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:54:29.065 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:28 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:28.836+0000 7fd93a5ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:54:29.066 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:54:29.063+0000 7fd93a5ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:54:29.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: Active manager daemon y restarted 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: Activating manager daemon y 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: mgrmap e10: y(active, starting, since 0.00333242s) 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: Manager daemon y is now available 2024-03-26T02:54:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:30.313 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:54:30.313 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:54:30.313 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:30 smithi066 ceph-mon[29666]: mgrmap e11: y(active, since 1.00596s) 2024-03-26T02:54:30.867 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout { 2024-03-26T02:54:30.867 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T02:54:30.867 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T02:54:30.867 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout } 2024-03-26T02:54:30.867 INFO:teuthology.orchestra.run.smithi066.stdout:mgr epoch 9 is available 2024-03-26T02:54:30.867 INFO:teuthology.orchestra.run.smithi066.stdout:Generating a dashboard self-signed certificate... 2024-03-26T02:54:32.213 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:31 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:30] ENGINE Bus STARTING 2024-03-26T02:54:32.213 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:31 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:30] ENGINE Serving on https://172.21.15.66:7150 2024-03-26T02:54:32.213 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:32.213 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:32.213 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:32.215 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T02:54:32.215 INFO:teuthology.orchestra.run.smithi066.stdout:Creating initial admin user... 2024-03-26T02:54:33.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:32 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:30] ENGINE Serving on http://172.21.15.66:8765 2024-03-26T02:54:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:32 smithi066 ceph-mon[29666]: [26/Mar/2024:02:54:30] ENGINE Bus STARTED 2024-03-26T02:54:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:32 smithi066 ceph-mon[29666]: from='client.14162 v1:172.21.15.66:0/1552634803' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:32 smithi066 ceph-mon[29666]: mgrmap e12: y(active, since 2s) 2024-03-26T02:54:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:33.696 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$aKVY3Y.fFdMSrXeJoxaCjeenNJfZ4sNG3G7Q8Sa20s7z5XKoPIa2y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711421673, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T02:54:33.697 INFO:teuthology.orchestra.run.smithi066.stdout:Fetching dashboard port number... 2024-03-26T02:54:34.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:33 smithi066 ceph-mon[29666]: from='client.14164 v1:172.21.15.66:0/3187408199' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:34.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:35.279 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4203303492' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T02:54:35.651 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T02:54:35.662 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T02:54:35.663 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout disabled 2024-03-26T02:54:35.671 INFO:teuthology.orchestra.run.smithi066.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T02:54:35.671 INFO:teuthology.orchestra.run.smithi066.stdout:systemctl: stdout inactive 2024-03-26T02:54:35.671 INFO:teuthology.orchestra.run.smithi066.stdout:firewalld.service is not enabled 2024-03-26T02:54:35.671 INFO:teuthology.orchestra.run.smithi066.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T02:54:35.673 INFO:teuthology.orchestra.run.smithi066.stdout:Ceph Dashboard is now available at: 2024-03-26T02:54:35.673 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:35.673 INFO:teuthology.orchestra.run.smithi066.stdout: URL: https://smithi066.front.sepia.ceph.com:8443/ 2024-03-26T02:54:35.673 INFO:teuthology.orchestra.run.smithi066.stdout: User: admin 2024-03-26T02:54:35.674 INFO:teuthology.orchestra.run.smithi066.stdout: Password: pn11sp5ppi 2024-03-26T02:54:35.674 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:35.674 INFO:teuthology.orchestra.run.smithi066.stdout:Saving cluster configuration to /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config directory 2024-03-26T02:54:35.674 INFO:teuthology.orchestra.run.smithi066.stdout:Enabling autotune for osd_memory_target 2024-03-26T02:54:37.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:36 smithi066 ceph-mon[29666]: mgrmap e13: y(active, since 6s) 2024-03-26T02:54:38.088 INFO:teuthology.orchestra.run.smithi066.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T02:54:38.088 INFO:teuthology.orchestra.run.smithi066.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: ceph telemetry on 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout:For more information see: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T02:54:38.089 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:38.090 INFO:teuthology.orchestra.run.smithi066.stdout:Bootstrap complete. 2024-03-26T02:54:38.136 INFO:tasks.cephadm:Fetching config... 2024-03-26T02:54:38.136 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:54:38.137 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T02:54:38.153 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T02:54:38.153 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:54:38.153 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T02:54:38.208 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T02:54:38.208 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:54:38.208 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T02:54:38.279 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T02:54:38.279 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:54:38.280 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T02:54:38.335 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T02:54:38.335 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYvyLYefIyb5xftgwgSP6ta2iWofhVQYxDlT4kuO8BpdZIIiISi6wRFLuOz35IT7jic2fYEIh7Tg1DXe9O26CfnJPbimM17ezhXDieTbsCsOaY/oX8Ckd0L1fogEraT5MfZDXGzif/83tJsGLfkFWjgMVe7XcBJqr3K/1J6lkRvJ7r/aDLLRgxRwsis3xB2Wco6TCztWj70ShMh8IMBOaeQTTo+W+lBnnP19fLjJuReVVNobRGseZAZqk4RC8h9L6AqcK1v1/qtor1+p6bv/Mbz1+UjFWjy8/YrRAhDeaqlhwoaOIIaRHvI+/bwFnZwKM1dhx7vGVw+e6tKfMtaQn/gwnGZIw3g8n/cIiQAgTHsAhWsKnkJeT/QEvP4HMn99hAzJ8TDlqlvGw8bzgof+itQ/2j70dkKUmWxeqiKXEZcxI/yVlzGMCoxKFd8nYp7zuNcloj7ddw4xw+NKeKVy+Xz4qPI4wB2ViiH+8ciUKcVmUs8yz+l0geSraV1wzwgXc= ceph-eba7a548-eb1b-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:54:38.424 INFO:teuthology.orchestra.run.smithi066.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYvyLYefIyb5xftgwgSP6ta2iWofhVQYxDlT4kuO8BpdZIIiISi6wRFLuOz35IT7jic2fYEIh7Tg1DXe9O26CfnJPbimM17ezhXDieTbsCsOaY/oX8Ckd0L1fogEraT5MfZDXGzif/83tJsGLfkFWjgMVe7XcBJqr3K/1J6lkRvJ7r/aDLLRgxRwsis3xB2Wco6TCztWj70ShMh8IMBOaeQTTo+W+lBnnP19fLjJuReVVNobRGseZAZqk4RC8h9L6AqcK1v1/qtor1+p6bv/Mbz1+UjFWjy8/YrRAhDeaqlhwoaOIIaRHvI+/bwFnZwKM1dhx7vGVw+e6tKfMtaQn/gwnGZIw3g8n/cIiQAgTHsAhWsKnkJeT/QEvP4HMn99hAzJ8TDlqlvGw8bzgof+itQ/2j70dkKUmWxeqiKXEZcxI/yVlzGMCoxKFd8nYp7zuNcloj7ddw4xw+NKeKVy+Xz4qPI4wB2ViiH+8ciUKcVmUs8yz+l0geSraV1wzwgXc= ceph-eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:54:38.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYvyLYefIyb5xftgwgSP6ta2iWofhVQYxDlT4kuO8BpdZIIiISi6wRFLuOz35IT7jic2fYEIh7Tg1DXe9O26CfnJPbimM17ezhXDieTbsCsOaY/oX8Ckd0L1fogEraT5MfZDXGzif/83tJsGLfkFWjgMVe7XcBJqr3K/1J6lkRvJ7r/aDLLRgxRwsis3xB2Wco6TCztWj70ShMh8IMBOaeQTTo+W+lBnnP19fLjJuReVVNobRGseZAZqk4RC8h9L6AqcK1v1/qtor1+p6bv/Mbz1+UjFWjy8/YrRAhDeaqlhwoaOIIaRHvI+/bwFnZwKM1dhx7vGVw+e6tKfMtaQn/gwnGZIw3g8n/cIiQAgTHsAhWsKnkJeT/QEvP4HMn99hAzJ8TDlqlvGw8bzgof+itQ/2j70dkKUmWxeqiKXEZcxI/yVlzGMCoxKFd8nYp7zuNcloj7ddw4xw+NKeKVy+Xz4qPI4wB2ViiH+8ciUKcVmUs8yz+l0geSraV1wzwgXc= ceph-eba7a548-eb1b-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:54:38.487 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYvyLYefIyb5xftgwgSP6ta2iWofhVQYxDlT4kuO8BpdZIIiISi6wRFLuOz35IT7jic2fYEIh7Tg1DXe9O26CfnJPbimM17ezhXDieTbsCsOaY/oX8Ckd0L1fogEraT5MfZDXGzif/83tJsGLfkFWjgMVe7XcBJqr3K/1J6lkRvJ7r/aDLLRgxRwsis3xB2Wco6TCztWj70ShMh8IMBOaeQTTo+W+lBnnP19fLjJuReVVNobRGseZAZqk4RC8h9L6AqcK1v1/qtor1+p6bv/Mbz1+UjFWjy8/YrRAhDeaqlhwoaOIIaRHvI+/bwFnZwKM1dhx7vGVw+e6tKfMtaQn/gwnGZIw3g8n/cIiQAgTHsAhWsKnkJeT/QEvP4HMn99hAzJ8TDlqlvGw8bzgof+itQ/2j70dkKUmWxeqiKXEZcxI/yVlzGMCoxKFd8nYp7zuNcloj7ddw4xw+NKeKVy+Xz4qPI4wB2ViiH+8ciUKcVmUs8yz+l0geSraV1wzwgXc= ceph-eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:54:38.502 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T02:54:38.751 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:54:38.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3515446249' entity='client.admin' 2024-03-26T02:54:41.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:41.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:41.242 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi066", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:54:41.242 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:41.242 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:41.242 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:41.732 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T02:54:41.733 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T02:54:41.979 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:54:42.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1271118542' entity='client.admin' 2024-03-26T02:54:42.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:42 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:42 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:44.338 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-03-26T02:54:44.338 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:54:44.338 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-03-26T02:54:44.358 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:54:44.358 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:54:44.416 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-03-26T02:54:44.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch host add smithi192 2024-03-26T02:54:44.666 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:54:44.912 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='client.14174 v1:172.21.15.66:0/2465630891' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:44.912 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:44.912 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:44.913 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:46.008 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:45 smithi066 ceph-mon[29666]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:54:46.008 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:45 smithi066 ceph-mon[29666]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:54:47.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:47 smithi066 ceph-mon[29666]: from='client.14176 v1:172.21.15.66:0/3694405961' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:48 smithi066 ceph-mon[29666]: Deploying cephadm binary to smithi192 2024-03-26T02:54:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:50 smithi066 ceph-mon[29666]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:54:50.796 INFO:teuthology.orchestra.run.smithi066.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-03-26T02:54:51.421 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T02:54:51.667 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:54:52.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:51 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:52.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:51 smithi066 ceph-mon[29666]: Added host smithi192 2024-03-26T02:54:52.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:51 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:52.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:51 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:53.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:52 smithi066 ceph-mon[29666]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:54:53.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:53.346 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:54:53.346 INFO:teuthology.orchestra.run.smithi066.stdout:[{"addr": "172.21.15.66", "hostname": "smithi066", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-03-26T02:54:53.928 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T02:54:53.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T02:54:54.176 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:54:54.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:54 smithi066 ceph-mon[29666]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:54:54.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:54 smithi066 ceph-mon[29666]: from='client.14178 v1:172.21.15.66:0/3511570842' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:54:54.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:54 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:56.424 INFO:teuthology.orchestra.run.smithi066.stderr:adjusted tunables profile to default 2024-03-26T02:54:56.751 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:56 smithi066 ceph-mon[29666]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:54:56.752 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1344720598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:54:57.097 INFO:tasks.cephadm:Adding mon.a on smithi066 2024-03-26T02:54:57.097 INFO:tasks.cephadm:Adding mon.c on smithi066 2024-03-26T02:54:57.097 INFO:tasks.cephadm:Adding mon.b on smithi192 2024-03-26T02:54:57.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b' 2024-03-26T02:54:57.331 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1344720598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:54:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:54:57.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:57.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:57.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:54:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:58 smithi066 ceph-mon[29666]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:54:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:58 smithi066 ceph-mon[29666]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:54:58.913 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-03-26T02:54:59.494 DEBUG:teuthology.orchestra.run.smithi066:mon.c> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T02:54:59.497 DEBUG:teuthology.orchestra.run.smithi192:mon.b> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T02:54:59.500 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:54:59.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:55:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='client.14182 v1:172.21.15.192:0/3231518521' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: Saving service mon spec with placement smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b;count:3 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:54:59 smithi066 ceph-mon[29666]: Deploying daemon mon.b on smithi192 2024-03-26T02:55:01.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:00 smithi066 ceph-mon[29666]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:01.693 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:55:03.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:02 smithi066 ceph-mon[29666]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(???) e0 preinit fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout: e1 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout: legacy client fscid: -1 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout: No filesystems configured 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T02:55:03.569 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='client.14178 v1:172.21.15.66:0/3511570842' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:55:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1344720598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:55:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1344720598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:55:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='client.14182 v1:172.21.15.192:0/3231518521' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: Saving service mon spec with placement smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b;count:3 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: Deploying daemon mon.b on smithi192 2024-03-26T02:55:03.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:03.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 ceph-mon[33110]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T02:55:03.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 bash[33063]: 61c0eb3397e33e3e2373581017cf854eb5a6343cf348be0737d9f8a864809110 2024-03-26T02:55:03.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:03 smithi192 systemd[1]: Started Ceph mon.b for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:55:08.620 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:55:08.620 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","modified":"2024-03-26T02:55:03.572947Z","created":"2024-03-26T02:53:25.962264Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:55:08.623 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: mon.a calling monitor election 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/3971079614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: mon.b calling monitor election 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: monmap epoch 2 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: last_changed 2024-03-26T02:55:03.572947+0000 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: min_mon_release 19 (squid) 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: election_strategy: 1 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: 1: v1:172.21.15.192:6789/0 mon.b 2024-03-26T02:55:08.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: fsmap 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: mgrmap e13: y(active, since 39s) 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: overall HEALTH_OK 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:08.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: mon.a calling monitor election 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/3971079614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: mon.b calling monitor election 2024-03-26T02:55:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: monmap epoch 2 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: last_changed 2024-03-26T02:55:03.572947+0000 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: min_mon_release 19 (squid) 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: election_strategy: 1 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:55:09.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: 1: v1:172.21.15.192:6789/0 mon.b 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: fsmap 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: mgrmap e13: y(active, since 39s) 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: overall HEALTH_OK 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:09.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:09 smithi066 ceph-mon[29666]: Deploying daemon mon.c on smithi066 2024-03-26T02:55:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:09 smithi192 ceph-mon[33110]: Deploying daemon mon.c on smithi066 2024-03-26T02:55:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:10.125 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:55:10.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:55:10.376 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:55:10.839 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:10 smithi192 ceph-mon[33110]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:10.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:10 smithi066 ceph-mon[29666]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:11.627 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:11 smithi066 systemd[1]: Starting Ceph mon.c for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T02:55:11.934 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:11 smithi066 podman[37891]: 2024-03-26 02:55:11.683841182 +0000 UTC m=+0.020012797 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:55:11.934 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:11 smithi066 podman[37891]: 2024-03-26 02:55:11.80088538 +0000 UTC m=+0.137056968 container create 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:55:12.096 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:55:12.097 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","modified":"2024-03-26T02:55:03.572947Z","created":"2024-03-26T02:53:25.962264Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:55:12.099 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 podman[37891]: 2024-03-26 02:55:12.061580784 +0000 UTC m=+0.397752376 container init 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 podman[37891]: 2024-03-26 02:55:12.066242566 +0000 UTC m=+0.402414156 container start 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pidfile_write: ignore empty --pid-file 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: load: jerasure load: lrc 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Git sha 0 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T02:55:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: DB SUMMARY 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: DB Session ID: MWKJ16J2WQ27CSKW4VLM 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.env: 0x55b371f7ec60 2024-03-26T02:55:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:55:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.info_log: 0x55b37399bb00 2024-03-26T02:55:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.statistics: (nil) 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.use_fsync: 0 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:55:12.363 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.db_log_dir: 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.wal_dir: 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.write_buffer_manager: 0x55b37396c1e0 2024-03-26T02:55:12.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.unordered_write: 0 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:55:12.365 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.row_cache: None 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.wal_filter: None 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.wal_compression: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:55:12.366 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:55:12.367 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_open_files: -1 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Compression algorithms supported: 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kZSTD supported: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:55:12.368 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.merge_operator: 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_filter: None 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:55:12.369 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b37399bce0) 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: index_type: 0 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: data_block_index_type: 0 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: index_shortening: 1 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: checksum: 4 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: no_block_cache: 0 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache: 0x55b37399f090 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache_options: 2024-03-26T02:55:12.370 INFO:journalctl@ceph.mon.c.smithi066.stdout: capacity : 536870912 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: num_shard_bits : 4 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: strict_capacity_limit : 0 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_cache_compressed: (nil) 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: persistent_cache: (nil) 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_size: 4096 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_size_deviation: 10 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_restart_interval: 16 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: index_block_restart_interval: 1 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: metadata_block_size: 4096 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: partition_filters: 0 2024-03-26T02:55:12.371 INFO:journalctl@ceph.mon.c.smithi066.stdout: use_delta_encoding: 1 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: filter_policy: bloomfilter 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: whole_key_filtering: 1 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: verify_compression: 0 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: format_version: 5 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: enable_index_compression: 1 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: block_align: 0 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: max_auto_readahead_size: 262144 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: prepopulate_block_cache: 0 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:55:12.372 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression: NoCompression 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.num_levels: 7 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:55:12.373 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:55:12.374 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:55:12.375 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:55:12.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:55:12.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.ttl: 2592000 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:55:12.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 236a99bd-db81-432d-b530-b17b6f8cea84 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421712096163, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421712096605, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711421712, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "236a99bd-db81-432d-b530-b17b6f8cea84", "db_session_id": "MWKJ16J2WQ27CSKW4VLM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:55:12.379 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421712096689, "job": 1, "event": "recovery_finished"} 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b373a8e000 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: DB pointer 0x55b373a78000 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** DB Stats ** 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:55:12.380 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** Compaction Stats [default] ** 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T02:55:12.381 INFO:journalctl@ceph.mon.c.smithi066.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Block cache BinnedLRUCache@0x55b37399f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T02:55:12.382 INFO:journalctl@ceph.mon.c.smithi066.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: using public_addrv v1:172.21.15.66:6790/0 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: starting mon.c rank -1 at public addrs v1:172.21.15.66:6790/0 at bind addrs v1:172.21.15.66:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(???) e0 preinit fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout: e1 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout: legacy client fscid: -1 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout: 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout: No filesystems configured 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T02:55:12.383 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='client.14178 v1:172.21.15.66:0/3511570842' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1344720598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:55:12.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1344720598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='client.14182 v1:172.21.15.192:0/3231518521' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Saving service mon spec with placement smithi066:[v1:172.21.15.66:6789]=a;smithi066:[v1:172.21.15.66:6790]=c;smithi192:[v1:172.21.15.192:6789]=b;count:3 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Deploying daemon mon.b on smithi192 2024-03-26T02:55:12.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.a calling monitor election 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/3971079614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.b calling monitor election 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.387 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: monmap epoch 2 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: last_changed 2024-03-26T02:55:03.572947+0000 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: min_mon_release 19 (squid) 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: election_strategy: 1 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: 1: v1:172.21.15.192:6789/0 mon.b 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: fsmap 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mgrmap e13: y(active, since 39s) 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: overall HEALTH_OK 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: Deploying daemon mon.c on smithi066 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 ceph-mon[37920]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 bash[37891]: 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e 2024-03-26T02:55:12.389 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:12 smithi066 systemd[1]: Started Ceph mon.c for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:55:13.550 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:55:13.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:55:13.794 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:55:17.465 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-26T02:55:17.466 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","modified":"2024-03-26T02:55:12.118480Z","created":"2024-03-26T02:53:25.962264Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6789","nonce":0}]},"addr":"172.21.15.66:6789/0","public_addr":"172.21.15.66:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6790","nonce":0}]},"addr":"172.21.15.66:6790/0","public_addr":"172.21.15.66:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T02:55:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:55:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: mon.a calling monitor election 2024-03-26T02:55:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: mon.b calling monitor election 2024-03-26T02:55:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: mon.c calling monitor election 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: monmap epoch 3 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: last_changed 2024-03-26T02:55:12.118480+0000 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: min_mon_release 19 (squid) 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: election_strategy: 1 2024-03-26T02:55:17.467 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: 1: v1:172.21.15.192:6789/0 mon.b 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: 2: v1:172.21.15.66:6790/0 mon.c 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: fsmap 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: mgrmap e13: y(active, since 48s) 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: overall HEALTH_OK 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.468 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.468 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-03-26T02:55:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:55:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: mon.a calling monitor election 2024-03-26T02:55:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: mon.b calling monitor election 2024-03-26T02:55:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: mon.c calling monitor election 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: monmap epoch 3 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: last_changed 2024-03-26T02:55:12.118480+0000 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: min_mon_release 19 (squid) 2024-03-26T02:55:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: election_strategy: 1 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: 1: v1:172.21.15.192:6789/0 mon.b 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: 2: v1:172.21.15.66:6790/0 mon.c 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: fsmap 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: mgrmap e13: y(active, since 48s) 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: overall HEALTH_OK 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:55:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: mon.a calling monitor election 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: mon.b calling monitor election 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: mon.c calling monitor election 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: monmap epoch 3 2024-03-26T02:55:17.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: last_changed 2024-03-26T02:55:12.118480+0000 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: created 2024-03-26T02:53:25.962264+0000 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: min_mon_release 19 (squid) 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: election_strategy: 1 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: 0: v1:172.21.15.66:6789/0 mon.a 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: 1: v1:172.21.15.192:6789/0 mon.b 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: 2: v1:172.21.15.66:6790/0 mon.c 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: fsmap 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: mgrmap e13: y(active, since 48s) 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: overall HEALTH_OK 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:17.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:18.029 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T02:55:18.029 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T02:55:18.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[37920]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:18.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:18.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/3528303309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:55:18.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:18.331 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[29666]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:18.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:18.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/3528303309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:55:18.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:18 smithi192 ceph-mon[33110]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:18 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/3528303309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:55:18.406 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:55:18.559 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:55:19.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:55:19.117+0000 7fd8f0770700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-26T02:55:19.953 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.954 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.955 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:19.956 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:19.957 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:20.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:20.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:20.403 INFO:teuthology.orchestra.run.smithi066.stdout:# minimal ceph.conf for eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:20.403 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2024-03-26T02:55:20.403 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T02:55:20.403 INFO:teuthology.orchestra.run.smithi066.stdout: mon_host = 172.21.15.66:6789/0 172.21.15.192:6789/0 v1:172.21.15.66:6790/0 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[29666]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[29666]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[29666]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[29666]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[29666]: Reconfiguring daemon mon.a on smithi066 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/365209208' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[37920]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:21.098 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[37920]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:21.099 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[37920]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:21.099 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[37920]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T02:55:21.099 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[37920]: Reconfiguring daemon mon.a on smithi066 2024-03-26T02:55:21.099 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:20 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/365209208' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:21.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:20 smithi192 ceph-mon[33110]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:21.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:20 smithi192 ceph-mon[33110]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:21.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:20 smithi192 ceph-mon[33110]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:55:21.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:20 smithi192 ceph-mon[33110]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T02:55:21.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:20 smithi192 ceph-mon[33110]: Reconfiguring daemon mon.a on smithi066 2024-03-26T02:55:21.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:20 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/365209208' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:21.356 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T02:55:21.356 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:55:21.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:55:21.391 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:55:21.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:55:21.464 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:55:21.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:55:21.498 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:55:21.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:55:21.568 INFO:tasks.cephadm:Adding mgr.y on smithi066 2024-03-26T02:55:21.569 INFO:tasks.cephadm:Adding mgr.x on smithi192 2024-03-26T02:55:21.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi066=y;smithi192=x' 2024-03-26T02:55:21.844 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:55:23.063 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:23.063 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:23.063 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:23.063 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:55:23.063 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:23.063 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:23.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:23.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:22 smithi192 ceph-mon[33110]: Reconfiguring daemon mon.c on smithi066 2024-03-26T02:55:23.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:23.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[29666]: Reconfiguring daemon mon.c on smithi066 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:55:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:23.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:23.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:23.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:22 smithi066 ceph-mon[37920]: Reconfiguring daemon mon.c on smithi066 2024-03-26T02:55:23.502 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-03-26T02:55:24.115 DEBUG:teuthology.orchestra.run.smithi192:mgr.x> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T02:55:24.117 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T02:55:24.117 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:55:24.117 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:55:24.137 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:55:24.137 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 02:55:11.489594505 +0000 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 02:49:32.011251064 +0000 2024-03-26T02:55:24.195 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 02:49:32.011251064 +0000 2024-03-26T02:55:24.196 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 02:49:32.011251064 +0000 2024-03-26T02:55:24.196 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:55:24.270 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T02:55:24.271 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T02:55:24.271 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000267389 s, 1.9 MB/s 2024-03-26T02:55:24.272 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:55:24.332 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-03-26T02:55:24.389 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:55:24.389 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:24.389 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-26T02:55:24.389 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:24.389 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:24.390 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 02:55:11.490594496 +0000 2024-03-26T02:55:24.390 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 02:49:32.355248158 +0000 2024-03-26T02:55:24.390 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 02:49:32.355248158 +0000 2024-03-26T02:55:24.390 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 02:49:32.355248158 +0000 2024-03-26T02:55:24.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:55:24.462 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T02:55:24.462 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T02:55:24.462 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000324129 s, 1.6 MB/s 2024-03-26T02:55:24.463 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:55:24.521 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-03-26T02:55:24.578 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:55:24.578 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:24.578 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T02:55:24.578 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:24.578 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:24.579 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 02:55:11.490594496 +0000 2024-03-26T02:55:24.579 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 02:49:32.705245201 +0000 2024-03-26T02:55:24.579 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 02:49:32.705245201 +0000 2024-03-26T02:55:24.579 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 02:49:32.705245201 +0000 2024-03-26T02:55:24.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:55:24.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:24.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: Reconfiguring daemon mon.b on smithi192 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='client.24103 v1:172.21.15.192:0/132184769' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi066=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: Saving service mgr spec with placement smithi066=y;smithi192=x;count:2 2024-03-26T02:55:24.584 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:24.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: Reconfiguring daemon mon.b on smithi192 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='client.24103 v1:172.21.15.192:0/132184769' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi066=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: Saving service mgr spec with placement smithi066=y;smithi192=x;count:2 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: Reconfiguring daemon mon.b on smithi192 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='client.24103 v1:172.21.15.192:0/132184769' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi066=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: Saving service mgr spec with placement smithi066=y;smithi192=x;count:2 2024-03-26T02:55:24.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:24.641 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T02:55:24.641 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T02:55:24.641 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000212259 s, 2.4 MB/s 2024-03-26T02:55:24.642 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:55:24.700 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-26 02:55:11.490594496 +0000 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-26 02:49:33.069242126 +0000 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-26 02:49:33.069242126 +0000 2024-03-26T02:55:24.756 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-26 02:49:33.069242126 +0000 2024-03-26T02:55:24.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:55:24.825 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-26T02:55:24.825 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-26T02:55:24.825 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000186318 s, 2.7 MB/s 2024-03-26T02:55:24.827 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:55:24.883 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:55:24.883 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:55:24.899 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:55:24.899 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-26 02:55:02.618188963 +0000 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-26 02:49:23.168715962 +0000 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-26 02:49:23.168715962 +0000 2024-03-26T02:55:24.954 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-26 02:49:23.168715962 +0000 2024-03-26T02:55:24.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:55:25.020 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-26T02:55:25.020 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-26T02:55:25.020 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000216538 s, 2.4 MB/s 2024-03-26T02:55:25.021 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:55:25.077 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-26 02:55:02.618188963 +0000 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-26 02:49:23.505709082 +0000 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-26 02:49:23.505709082 +0000 2024-03-26T02:55:25.132 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-26 02:49:23.505709082 +0000 2024-03-26T02:55:25.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:55:25.197 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-26T02:55:25.197 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-26T02:55:25.197 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000219747 s, 2.3 MB/s 2024-03-26T02:55:25.198 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:55:25.254 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-03-26T02:55:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:55:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:25.309 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T02:55:25.309 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:25.310 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:25.310 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-26 02:55:02.618188963 +0000 2024-03-26T02:55:25.310 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-26 02:49:23.840702242 +0000 2024-03-26T02:55:25.310 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-26 02:49:23.840702242 +0000 2024-03-26T02:55:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-26 02:49:23.840702242 +0000 2024-03-26T02:55:25.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:55:25.376 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-26T02:55:25.376 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-26T02:55:25.376 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000170621 s, 3.0 MB/s 2024-03-26T02:55:25.377 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:55:25.433 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-03-26T02:55:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-26 02:55:02.618188963 +0000 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-26 02:49:24.156695790 +0000 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-26 02:49:24.156695790 +0000 2024-03-26T02:55:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-26 02:49:24.156695790 +0000 2024-03-26T02:55:25.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:55:25.555 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-26T02:55:25.555 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-26T02:55:25.555 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000168471 s, 3.0 MB/s 2024-03-26T02:55:25.557 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:55:25.612 INFO:tasks.cephadm:Deploying osd.0 on smithi066 with /dev/vg_nvme/lv_4... 2024-03-26T02:55:25.612 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T02:55:25.868 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:55:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:26 smithi192 ceph-mon[33110]: Deploying daemon mgr.x on smithi192 2024-03-26T02:55:26.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[37920]: Deploying daemon mgr.x on smithi192 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:26 smithi066 ceph-mon[29666]: Deploying daemon mgr.x on smithi192 2024-03-26T02:55:27.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 systemd[1]: Starting Ceph mgr.x for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T02:55:27.662 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T02:55:27.662 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T02:55:27.662 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-26T02:55:27.662 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-26T02:55:27.662 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112812 s, 929 MB/s 2024-03-26T02:55:27.662 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-26T02:55:27.808 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 podman[34972]: 2024-03-26 02:55:27.405502067 +0000 UTC m=+0.017197549 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:55:27.808 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 podman[34972]: 2024-03-26 02:55:27.504856281 +0000 UTC m=+0.116551756 container create 43fd2080f308581c77d9aadc559189df18b207b512fb9d61f4b2667a11db4a4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T02:55:27.809 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 podman[34972]: 2024-03-26 02:55:27.732750429 +0000 UTC m=+0.344445904 container init 43fd2080f308581c77d9aadc559189df18b207b512fb9d61f4b2667a11db4a4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T02:55:27.809 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 podman[34972]: 2024-03-26 02:55:27.735875224 +0000 UTC m=+0.347570708 container start 43fd2080f308581c77d9aadc559189df18b207b512fb9d61f4b2667a11db4a4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T02:55:27.809 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 bash[34972]: 43fd2080f308581c77d9aadc559189df18b207b512fb9d61f4b2667a11db4a4d 2024-03-26T02:55:28.238 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:27 smithi192 systemd[1]: Started Ceph mgr.x for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:55:28.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_4 2024-03-26T02:55:28.561 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:55:28.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:28.296+0000 7f747146b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:55:28.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:28.393+0000 7f747146b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:55:29.009 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[37920]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[29666]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:29.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-mon[33110]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:29.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.089 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:29.090 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:29.354 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.088+0000 7f747146b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:55:29.354 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.210+0000 7f747146b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:55:29.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.353+0000 7f747146b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:55:29.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.447+0000 7f747146b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:55:29.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.530+0000 7f747146b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:55:30.076 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.791+0000 7f747146b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:55:30.077 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.899+0000 7f747146b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:55:30.077 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:29.983+0000 7f747146b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:55:30.077 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:30.075+0000 7f747146b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:30.252 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[37920]: Reconfiguring daemon mgr.y on smithi066 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.253 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:55:30.254 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:55:30.254 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:55:30.254 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:30.254 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:30 smithi066 ceph-mon[29666]: Reconfiguring daemon mgr.y on smithi066 2024-03-26T02:55:30.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:30.250+0000 7f747146b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:30 smithi192 ceph-mon[33110]: Reconfiguring daemon mgr.y on smithi066 2024-03-26T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[37920]: from='client.14214 v1:172.21.15.66:0/1067956038' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:31.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[29666]: from='client.14214 v1:172.21.15.66:0/1067956038' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-mon[33110]: from='client.14214 v1:172.21.15.66:0/1067956038' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:31.214+0000 7f747146b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:55:31.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:31.346+0000 7f747146b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:55:31.957 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:31.676+0000 7f747146b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:55:31.957 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:31.762+0000 7f747146b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:55:31.957 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:31.955+0000 7f747146b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:55:32.237 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:32.135+0000 7f747146b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:55:32.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-mon[33110]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:32.445 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[29666]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:32.445 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[37920]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:32.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:32 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:32.679 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:32.415+0000 7f747146b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:55:32.679 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:32.678+0000 7f747146b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:55:32.987 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:32.915+0000 7f747146b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:55:33.987 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:33.725+0000 7f747146b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:55:34.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:34.301+0000 7f747146b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:55:34.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:34.399+0000 7f747146b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:55:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-mon[33110]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2390817183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9"}]: dispatch 2024-03-26T02:55:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2390817183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9"}]': finished 2024-03-26T02:55:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-mon[33110]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:55:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[29666]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2390817183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9"}]: dispatch 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2390817183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9"}]': finished 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[29666]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[37920]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2390817183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9"}]: dispatch 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2390817183' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9"}]': finished 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[37920]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:55:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:34 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:34.987 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:55:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:55:34.616+0000 7f747146b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:55:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:35 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2831608308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:55:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:35 smithi192 ceph-mon[33110]: Standby manager daemon x started 2024-03-26T02:55:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:35 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:55:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:35 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:55:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:35 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:55:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:35 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:55:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2831608308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[29666]: Standby manager daemon x started 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2831608308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:55:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[37920]: Standby manager daemon x started 2024-03-26T02:55:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:55:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:55:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:55:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:35 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/3877133881' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:55:36.454 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:36 smithi066 ceph-mon[37920]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:36.454 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:36 smithi066 ceph-mon[37920]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-26T02:55:36.454 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:36 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:55:36.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:36 smithi066 ceph-mon[29666]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:36.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:36 smithi066 ceph-mon[29666]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-26T02:55:36.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:36 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:55:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:36 smithi192 ceph-mon[33110]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:36 smithi192 ceph-mon[33110]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-26T02:55:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:36 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:55:38.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:38 smithi066 ceph-mon[29666]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:38 smithi066 ceph-mon[37920]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:38 smithi192 ceph-mon[33110]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:40.468 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[29666]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:40.468 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:55:40.468 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:40.468 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[29666]: Deploying daemon osd.0 on smithi066 2024-03-26T02:55:40.469 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[37920]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:40.469 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:55:40.469 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:40.469 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:40 smithi066 ceph-mon[37920]: Deploying daemon osd.0 on smithi066 2024-03-26T02:55:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:40 smithi192 ceph-mon[33110]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:55:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:40 smithi192 ceph-mon[33110]: Deploying daemon osd.0 on smithi066 2024-03-26T02:55:42.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:42 smithi066 ceph-mon[29666]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:42.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:42 smithi066 ceph-mon[37920]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:42 smithi192 ceph-mon[33110]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:45.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:44 smithi066 ceph-mon[29666]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:45.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:44 smithi066 ceph-mon[37920]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:44 smithi192 ceph-mon[33110]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[29666]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[37920]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:46.515 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:46 smithi192 ceph-mon[33110]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:48.804 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 0 on host 'smithi066' 2024-03-26T02:55:49.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:49.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='osd.0 v1:172.21.15.66:6801/1028658447' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:49.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='osd.0 v1:172.21.15.66:6801/1028658447' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:49.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:49.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='osd.0 v1:172.21.15.66:6801/1028658447' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:49.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:49.480 DEBUG:teuthology.orchestra.run.smithi066:osd.0> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T02:55:49.483 INFO:tasks.cephadm:Deploying osd.1 on smithi066 with /dev/vg_nvme/lv_3... 2024-03-26T02:55:49.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T02:55:50.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[29666]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[29666]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[29666]: from='osd.0 v1:172.21.15.66:6801/1028658447' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[29666]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[37920]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[37920]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[37920]: from='osd.0 v1:172.21.15.66:6801/1028658447' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:55:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-mon[37920]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:55:50.110 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0[41834]: 2024-03-26T02:55:49.910+0000 7fa0c7472700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T02:55:50.110 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:55:49 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0[41834]: 2024-03-26T02:55:49.912+0000 7fa0c3265700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:55:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:49 smithi192 ceph-mon[33110]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:55:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:49 smithi192 ceph-mon[33110]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:55:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:49 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:49 smithi192 ceph-mon[33110]: from='osd.0 v1:172.21.15.66:6801/1028658447' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:55:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:49 smithi192 ceph-mon[33110]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:55:50.949 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[37920]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:50.949 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[37920]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:55:50.949 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[37920]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:55:50.949 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.949 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.949 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.950 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[29666]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:50.950 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[29666]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:55:50.950 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[29666]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:55:50.950 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.950 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:50.950 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:50 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:51.138 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:55:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:50 smithi192 ceph-mon[33110]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:50 smithi192 ceph-mon[33110]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:55:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:50 smithi192 ceph-mon[33110]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:55:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:50 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:50 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:50 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:52.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:51 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:55:52.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:51 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:55:52.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:51 smithi192 ceph-mon[33110]: osd.0 v1:172.21.15.66:6801/1028658447 boot 2024-03-26T02:55:52.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:51 smithi192 ceph-mon[33110]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:55:52.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:51 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:52.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:55:52.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:55:52.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[29666]: osd.0 v1:172.21.15.66:6801/1028658447 boot 2024-03-26T02:55:52.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[29666]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:55:52.347 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:52.347 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:55:52.347 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:55:52.347 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[37920]: osd.0 v1:172.21.15.66:6801/1028658447 boot 2024-03-26T02:55:52.347 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[37920]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:55:52.347 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:51 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:52.986 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:52.987 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:53.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:55:53.541 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T02:55:53.541 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T02:55:53.541 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-26T02:55:53.542 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-26T02:55:53.542 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114558 s, 915 MB/s 2024-03-26T02:55:53.542 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-26T02:55:54.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_3 2024-03-26T02:55:54.158 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:53 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi066 2024-03-26T02:55:54.158 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:53 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi066 to 16064M 2024-03-26T02:55:54.158 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:53 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi066 2024-03-26T02:55:54.158 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:53 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi066 to 16064M 2024-03-26T02:55:54.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:53 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi066 2024-03-26T02:55:54.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:53 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi066 to 16064M 2024-03-26T02:55:54.377 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:55:55.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:54 smithi066 ceph-mon[29666]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:55.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:54 smithi066 ceph-mon[37920]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:55.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:54 smithi192 ceph-mon[33110]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:57.010 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[29666]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[37920]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:55:57.011 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:56 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:57.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:56 smithi192 ceph-mon[33110]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:57.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:56 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:55:57.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:56 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:55:57.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:56 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:58.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:57 smithi066 ceph-mon[29666]: from='client.34136 v1:172.21.15.66:0/569719847' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:58.232 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:57 smithi066 ceph-mon[37920]: from='client.34136 v1:172.21.15.66:0/569719847' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:58.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:57 smithi192 ceph-mon[33110]: from='client.34136 v1:172.21.15.66:0/569719847' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:59.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:58 smithi192 ceph-mon[33110]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3366250155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9"}]: dispatch 2024-03-26T02:55:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3366250155' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9"}]': finished 2024-03-26T02:55:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:58 smithi192 ceph-mon[33110]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:55:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:58 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:55:59.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[29666]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:59.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3366250155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9"}]: dispatch 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3366250155' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9"}]': finished 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[29666]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[37920]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3366250155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9"}]: dispatch 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3366250155' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9"}]': finished 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[37920]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:55:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:58 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:00.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:55:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/854412777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:55:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/854412777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:00.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:55:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/854412777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:01.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:00 smithi192 ceph-mon[33110]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:01.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:00 smithi066 ceph-mon[29666]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:01.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:00 smithi066 ceph-mon[37920]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:03.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:02 smithi192 ceph-mon[33110]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:03.242 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:02 smithi066 ceph-mon[29666]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:03.242 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:02 smithi066 ceph-mon[37920]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:05.052 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:04 smithi066 ceph-mon[29666]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:05.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:04 smithi066 ceph-mon[37920]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:05.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:04 smithi192 ceph-mon[33110]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:06.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:05 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:56:06.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:05 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:06.239 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:05 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:56:06.239 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:05 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:06.239 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:05 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:56:06.239 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:05 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:07.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:06 smithi192 ceph-mon[33110]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:07.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:06 smithi192 ceph-mon[33110]: Deploying daemon osd.1 on smithi066 2024-03-26T02:56:07.286 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:06 smithi066 ceph-mon[37920]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:07.286 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:06 smithi066 ceph-mon[37920]: Deploying daemon osd.1 on smithi066 2024-03-26T02:56:07.286 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:06 smithi066 ceph-mon[29666]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:07.286 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:06 smithi066 ceph-mon[29666]: Deploying daemon osd.1 on smithi066 2024-03-26T02:56:09.096 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:08 smithi066 ceph-mon[29666]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:09.096 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:08 smithi066 ceph-mon[37920]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:09.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:08 smithi192 ceph-mon[33110]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:10 smithi192 ceph-mon[33110]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:11.243 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[29666]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[37920]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:11.244 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:13.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:12 smithi066 ceph-mon[29666]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:13.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:12 smithi066 ceph-mon[37920]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:13.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:12 smithi192 ceph-mon[33110]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:14.712 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 1 on host 'smithi066' 2024-03-26T02:56:14.898 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:14.898 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.898 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.898 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:14.898 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:14.898 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='osd.1 v1:172.21.15.66:6805/3170415936' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='osd.1 v1:172.21.15.66:6805/3170415936' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:56:14.900 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:56:14.900 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.900 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:14.900 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:15.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:15.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='osd.1 v1:172.21.15.66:6805/3170415936' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:15.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:15.327 DEBUG:teuthology.orchestra.run.smithi066:osd.1> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T02:56:15.369 INFO:tasks.cephadm:Deploying osd.2 on smithi066 with /dev/vg_nvme/lv_2... 2024-03-26T02:56:15.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T02:56:16.098 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:56:16.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:15 smithi192 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:56:16.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:15 smithi192 ceph-mon[33110]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:56:16.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:15 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:16.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:15 smithi192 ceph-mon[33110]: from='osd.1 v1:172.21.15.66:6805/3170415936' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:16.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:15 smithi192 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:16.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[29666]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:56:16.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[29666]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:56:16.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:16.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[29666]: from='osd.1 v1:172.21.15.66:6805/3170415936' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[29666]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[37920]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:56:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[37920]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:56:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[37920]: from='osd.1 v1:172.21.15.66:6805/3170415936' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:15 smithi066 ceph-mon[37920]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:16.701 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1[47622]: 2024-03-26T02:56:16.424+0000 7fd869e47700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T02:56:16.701 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1[47622]: 2024-03-26T02:56:16.426+0000 7fd861c32700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:56:17.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:17.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:16 smithi192 ceph-mon[33110]: from='osd.1 ' entity='osd.1' 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:17.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[29666]: from='osd.1 ' entity='osd.1' 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:16 smithi066 ceph-mon[37920]: from='osd.1 ' entity='osd.1' 2024-03-26T02:56:18.177 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:18.177 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[29666]: osd.1 v1:172.21.15.66:6805/3170415936 boot 2024-03-26T02:56:18.177 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[29666]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:56:18.177 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:18.177 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:18.178 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[37920]: osd.1 v1:172.21.15.66:6805/3170415936 boot 2024-03-26T02:56:18.178 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[37920]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:56:18.178 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:18.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:18.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:17 smithi192 ceph-mon[33110]: osd.1 v1:172.21.15.66:6805/3170415936 boot 2024-03-26T02:56:18.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:17 smithi192 ceph-mon[33110]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:56:18.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:19.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:19.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.486 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T02:56:19.486 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T02:56:19.486 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-26T02:56:19.486 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-26T02:56:19.487 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011186 s, 937 MB/s 2024-03-26T02:56:19.487 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-26T02:56:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:56:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:20.103 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:19 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi066 2024-03-26T02:56:20.104 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:19 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi066 to 8032M 2024-03-26T02:56:20.104 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:19 smithi066 ceph-mon[29666]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:56:20.104 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:20 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi066 2024-03-26T02:56:20.104 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:20 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi066 to 8032M 2024-03-26T02:56:20.104 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:20 smithi066 ceph-mon[37920]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:56:20.132 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_2 2024-03-26T02:56:20.386 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:56:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:19 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi066 2024-03-26T02:56:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:19 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi066 to 8032M 2024-03-26T02:56:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:19 smithi192 ceph-mon[33110]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:56:21.059 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:20 smithi066 ceph-mon[29666]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:21.059 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:20 smithi066 ceph-mon[37920]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:20 smithi192 ceph-mon[33110]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:22.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[29666]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:22.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[29666]: from='client.34163 v1:172.21.15.66:0/1189163713' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[37920]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[37920]: from='client.34163 v1:172.21.15.66:0/1189163713' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:56:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:22 smithi192 ceph-mon[33110]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:22 smithi192 ceph-mon[33110]: from='client.34163 v1:172.21.15.66:0/1189163713' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:56:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:56:22.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:24.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:24 smithi066 ceph-mon[29666]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:24.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:24 smithi066 ceph-mon[37920]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:24 smithi192 ceph-mon[33110]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1996750142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b"}]: dispatch 2024-03-26T02:56:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1996750142' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b"}]': finished 2024-03-26T02:56:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:25 smithi192 ceph-mon[33110]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:56:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:25 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1996750142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b"}]: dispatch 2024-03-26T02:56:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1996750142' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b"}]': finished 2024-03-26T02:56:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[29666]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:56:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1996750142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b"}]: dispatch 2024-03-26T02:56:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1996750142' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b"}]': finished 2024-03-26T02:56:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[37920]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:56:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:25 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:26 smithi192 ceph-mon[33110]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:26 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2927072032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:26 smithi066 ceph-mon[29666]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:26 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2927072032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:26 smithi066 ceph-mon[37920]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:26 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2927072032' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:28.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:28 smithi192 ceph-mon[33110]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:28 smithi066 ceph-mon[29666]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:28.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:28 smithi066 ceph-mon[37920]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:31.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:30 smithi066 ceph-mon[29666]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:31.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:30 smithi066 ceph-mon[37920]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:31.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:30 smithi192 ceph-mon[33110]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:32.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:56:32.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:32.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:31 smithi066 ceph-mon[37920]: Deploying daemon osd.2 on smithi066 2024-03-26T02:56:32.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:56:32.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:32.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:31 smithi066 ceph-mon[29666]: Deploying daemon osd.2 on smithi066 2024-03-26T02:56:32.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:56:32.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:32.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:31 smithi192 ceph-mon[33110]: Deploying daemon osd.2 on smithi066 2024-03-26T02:56:32.933 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:32 smithi066 ceph-mon[29666]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:32.933 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:32 smithi066 ceph-mon[37920]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:33.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:32 smithi192 ceph-mon[33110]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:35.092 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:34 smithi066 ceph-mon[29666]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:35.092 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:34 smithi066 ceph-mon[37920]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:35.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:34 smithi192 ceph-mon[33110]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:36.881 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[29666]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:36.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:36.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:36.882 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:36.882 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[37920]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:36.882 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:36.883 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:36.883 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:36 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:37.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:36 smithi192 ceph-mon[33110]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:37.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:36 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:37.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:36 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:37.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:36 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:39.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:38 smithi066 ceph-mon[29666]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:38 smithi066 ceph-mon[37920]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:39.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:38 smithi192 ceph-mon[33110]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:39.773 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 2 on host 'smithi066' 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:56:40.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:40 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:40.451 DEBUG:teuthology.orchestra.run.smithi066:osd.2> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T02:56:40.453 INFO:tasks.cephadm:Deploying osd.3 on smithi066 with /dev/vg_nvme/lv_1... 2024-03-26T02:56:40.453 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T02:56:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:40 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[29666]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[29666]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[29666]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[37920]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[37920]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[37920]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:41.407 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:41.408 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2[53675]: 2024-03-26T02:56:41.152+0000 7f3d73b5d700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T02:56:41.408 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:41 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2[53675]: 2024-03-26T02:56:41.156+0000 7f3d6f950700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:56:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:41 smithi192 ceph-mon[33110]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:56:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:41 smithi192 ceph-mon[33110]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:56:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:41 smithi192 ceph-mon[33110]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:56:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:41 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:41.904 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:56:42.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[29666]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:42.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[29666]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:56:42.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[29666]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:56:42.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:42.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:42.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[37920]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:42.362 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[37920]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:56:42.362 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[37920]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:56:42.362 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:42.362 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:42 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:42 smithi192 ceph-mon[33110]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:42 smithi192 ceph-mon[33110]: from='osd.2 v1:172.21.15.66:6809/2520642381' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:56:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:42 smithi192 ceph-mon[33110]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:56:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:42 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:42 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:43.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:56:43.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:56:43.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[29666]: osd.2 v1:172.21.15.66:6809/2520642381 boot 2024-03-26T02:56:43.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[29666]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:56:43.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:43.322 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:56:43.322 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:56:43.322 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[37920]: osd.2 v1:172.21.15.66:6809/2520642381 boot 2024-03-26T02:56:43.322 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[37920]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:56:43.322 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:43 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:43 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:56:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:43 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:56:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:43 smithi192 ceph-mon[33110]: osd.2 v1:172.21.15.66:6809/2520642381 boot 2024-03-26T02:56:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:43 smithi192 ceph-mon[33110]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:56:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:43 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi066 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi066 to 5354M 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi066 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.445 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi066 to 5354M 2024-03-26T02:56:44.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:44.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:44.446 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.483 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T02:56:44.483 INFO:teuthology.orchestra.run.smithi066.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T02:56:44.483 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 10+0 records in 2024-03-26T02:56:44.484 INFO:teuthology.orchestra.run.smithi066.stderr:10+0 records out 2024-03-26T02:56:44.484 INFO:teuthology.orchestra.run.smithi066.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113619 s, 923 MB/s 2024-03-26T02:56:44.484 INFO:teuthology.orchestra.run.smithi066.stderr: stderr: 2024-03-26T02:56:44.484 INFO:teuthology.orchestra.run.smithi066.stderr:--> Zapping successful for: 2024-03-26T02:56:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:56:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:56:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:56:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi066 2024-03-26T02:56:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi066 to 5354M 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:45.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi066:vg_nvme/lv_1 2024-03-26T02:56:45.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T02:56:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 ceph-mon[29666]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:56:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:56:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T02:56:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 ceph-mon[37920]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:56:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:56:45.359 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57343]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:56:45.359 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.359 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57343]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.360 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57343]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.360 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57359]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:56:45.360 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57359]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.360 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57359]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.360 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57359]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.406 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:56:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T02:56:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 ceph-mon[33110]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:56:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:56:45.670 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:56:45.670 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.670 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.670 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57415]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.670 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57419]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:56:45.671 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57419]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.671 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57419]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.671 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57419]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57398]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57398]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57398]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57398]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57412]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57412]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57412]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.671 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57412]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57422]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57422]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57422]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57422]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57425]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57425]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57425]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.672 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57425]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.672 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57376]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:56:45.672 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57376]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.672 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57376]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.672 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57376]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.673 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57391]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:56:45.673 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57391]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.673 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57391]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.673 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 02:56:45 smithi066 sudo[57391]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35689]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:56:45.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35689]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35689]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:45.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35692]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:56:45.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35692]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T02:56:45.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35692]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T02:56:45.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:45 smithi192 sudo[35692]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:46 smithi192 ceph-mon[33110]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:46.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[29666]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:56:46.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:56:46.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:56:46.540 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:56:46.540 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:56:46.540 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:56:46.540 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:46 smithi066 ceph-mon[37920]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:56:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:47 smithi192 ceph-mon[33110]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T02:56:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:47 smithi192 ceph-mon[33110]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T02:56:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:47 smithi066 ceph-mon[29666]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T02:56:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:47 smithi066 ceph-mon[29666]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T02:56:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:47 smithi066 ceph-mon[37920]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T02:56:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:47 smithi066 ceph-mon[37920]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T02:56:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: from='client.34199 v1:172.21.15.66:0/2454404308' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:56:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:56:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:56:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:48 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T02:56:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: from='client.34199 v1:172.21.15.66:0/2454404308' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: from='client.34199 v1:172.21.15.66:0/2454404308' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi066:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:56:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:48 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3438676620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]: dispatch 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: from='client.34205 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]: dispatch 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]': finished 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3817420744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:51.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:51.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3438676620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]: dispatch 2024-03-26T02:56:51.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: from='client.34205 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]: dispatch 2024-03-26T02:56:51.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]': finished 2024-03-26T02:56:51.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T02:56:51.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:56:51.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:50 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3817420744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3438676620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]: dispatch 2024-03-26T02:56:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: from='client.34205 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]: dispatch 2024-03-26T02:56:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: from='client.34205 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2"}]': finished 2024-03-26T02:56:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T02:56:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:56:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:50 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3817420744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:56:53.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:52 smithi066 ceph-mon[37920]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:53.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:52 smithi066 ceph-mon[29666]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:53.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:52 smithi192 ceph-mon[33110]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:55.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:54 smithi066 ceph-mon[29666]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:55.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:54 smithi066 ceph-mon[37920]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:55.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:54 smithi192 ceph-mon[33110]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:56 smithi066 ceph-mon[29666]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:56 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:56:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:56 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:57.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:56 smithi066 ceph-mon[37920]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:57.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:56 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:56:57.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:56 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:57.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:56 smithi192 ceph-mon[33110]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:57.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:56 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:56:57.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:56 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:58.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:57 smithi066 ceph-mon[29666]: Deploying daemon osd.3 on smithi066 2024-03-26T02:56:58.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:57 smithi066 ceph-mon[37920]: Deploying daemon osd.3 on smithi066 2024-03-26T02:56:58.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:57 smithi192 ceph-mon[33110]: Deploying daemon osd.3 on smithi066 2024-03-26T02:56:59.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:56:58 smithi066 ceph-mon[29666]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:59.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:56:58 smithi066 ceph-mon[37920]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:56:59.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:56:58 smithi192 ceph-mon[33110]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:01.090 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:00 smithi066 ceph-mon[29666]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:01.090 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:00 smithi066 ceph-mon[37920]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:01.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:00 smithi192 ceph-mon[33110]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:02.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[37920]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:02.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:02.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:02.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:02.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[29666]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:02.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:02.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:02.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:02 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:02 smithi192 ceph-mon[33110]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:02 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:02 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:02 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:04.961 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:04 smithi066 ceph-mon[29666]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:04.961 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:04.961 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:04.961 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:04 smithi066 ceph-mon[37920]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:04.961 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:04.961 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:04 smithi192 ceph-mon[33110]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:05.297 INFO:teuthology.orchestra.run.smithi066.stdout:Created osd(s) 3 on host 'smithi066' 2024-03-26T02:57:05.948 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:05 smithi066 ceph-mon[37920]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:57:05.948 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:05 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:05.948 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:05 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:05.948 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:05 smithi066 ceph-mon[29666]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:57:05.949 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:05 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:05.949 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:05 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:05.973 DEBUG:teuthology.orchestra.run.smithi066:osd.3> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T02:57:05.975 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_4... 2024-03-26T02:57:05.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T02:57:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:05 smithi192 ceph-mon[33110]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:57:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:05 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:05 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:06.220 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:57:06.946 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[29666]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:06.946 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[29666]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:57:06.946 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[29666]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T02:57:06.946 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[29666]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:57:06.946 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:06.947 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[37920]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:06.947 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[37920]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:57:06.947 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[37920]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T02:57:06.947 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[37920]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:57:06.947 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:06.947 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[59389]: 2024-03-26T02:57:06.648+0000 7f5bee949700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T02:57:06.947 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 02:57:06 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[59389]: 2024-03-26T02:57:06.653+0000 7f5bea73c700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:57:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:06 smithi192 ceph-mon[33110]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:06 smithi192 ceph-mon[33110]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:57:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:06 smithi192 ceph-mon[33110]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T02:57:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:06 smithi192 ceph-mon[33110]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T02:57:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:06 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:07.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:07 smithi192 ceph-mon[33110]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:57:07.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:07 smithi192 ceph-mon[33110]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T02:57:07.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:07 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:07.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:07 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:08.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[29666]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:57:08.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[29666]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T02:57:08.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:08.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:08.080 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[37920]: from='osd.3 v1:172.21.15.66:6813/2366374823' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]': finished 2024-03-26T02:57:08.081 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[37920]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T02:57:08.081 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:08.081 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:07 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:08.084 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T02:57:08.085 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T02:57:08.085 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-26T02:57:08.085 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-26T02:57:08.085 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122374 s, 857 MB/s 2024-03-26T02:57:08.085 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-26T02:57:08.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-03-26T02:57:08.979 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:57:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: osd.3 v1:172.21.15.66:6813/2366374823 boot 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T02:57:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:09.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:57:09.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:57:09.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:09.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:57:09.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: osd.3 v1:172.21.15.66:6813/2366374823 boot 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: osd.3 v1:172.21.15.66:6813/2366374823 boot 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T02:57:09.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:57:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi066 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi066 to 4016M 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:09.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:09.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:09.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:09.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:09.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:09 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi066 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi066 to 4016M 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T02:57:10.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi066 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi066 to 4016M 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:10.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:09 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:11.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[29666]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:11.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:11.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[37920]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:10 smithi192 ceph-mon[33110]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:11.959 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:11 smithi192 ceph-mon[33110]: from='client.24232 v1:172.21.15.192:0/2875666908' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:12.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:11 smithi066 ceph-mon[29666]: from='client.24232 v1:172.21.15.192:0/2875666908' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:12.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:11 smithi066 ceph-mon[37920]: from='client.24232 v1:172.21.15.192:0/2875666908' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:13.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:12 smithi192 ceph-mon[33110]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:13.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:12 smithi066 ceph-mon[29666]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:13.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:12 smithi066 ceph-mon[37920]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:14.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:13 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/56058447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]: dispatch 2024-03-26T02:57:14.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:13 smithi192 ceph-mon[33110]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]: dispatch 2024-03-26T02:57:14.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:13 smithi192 ceph-mon[33110]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]': finished 2024-03-26T02:57:14.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:13 smithi192 ceph-mon[33110]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T02:57:14.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:13 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:14.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/56058447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]: dispatch 2024-03-26T02:57:14.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[29666]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]: dispatch 2024-03-26T02:57:14.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[29666]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]': finished 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[29666]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/56058447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]: dispatch 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[37920]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]: dispatch 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[37920]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883"}]': finished 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[37920]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T02:57:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:13 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:15.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:14 smithi192 ceph-mon[33110]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:15.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:14 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/1619967416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:57:15.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:14 smithi066 ceph-mon[29666]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:15.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:14 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/1619967416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:57:15.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:14 smithi066 ceph-mon[37920]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:15.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:14 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/1619967416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:57:17.160 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:16 smithi066 ceph-mon[29666]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:17.160 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:16 smithi066 ceph-mon[37920]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:17.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:16 smithi192 ceph-mon[33110]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:19.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:18 smithi192 ceph-mon[33110]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:19.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:18 smithi066 ceph-mon[29666]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:19.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:18 smithi066 ceph-mon[37920]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:20.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:57:20.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:20.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:57:20.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:20.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:57:20.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:21.181 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:20 smithi192 ceph-mon[33110]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:21.181 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:20 smithi192 ceph-mon[33110]: Deploying daemon osd.4 on smithi192 2024-03-26T02:57:21.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:20 smithi066 ceph-mon[29666]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:21.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:20 smithi066 ceph-mon[29666]: Deploying daemon osd.4 on smithi192 2024-03-26T02:57:21.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:20 smithi066 ceph-mon[37920]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:21.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:20 smithi066 ceph-mon[37920]: Deploying daemon osd.4 on smithi192 2024-03-26T02:57:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:22 smithi192 ceph-mon[33110]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:23.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:22 smithi066 ceph-mon[29666]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:23.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:22 smithi066 ceph-mon[37920]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:24 smithi192 ceph-mon[33110]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:24 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[29666]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[37920]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:24 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:27.155 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:26 smithi192 ceph-mon[33110]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:27.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:26 smithi066 ceph-mon[29666]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:27.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:26 smithi066 ceph-mon[37920]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:28.262 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='osd.4 v1:172.21.15.192:6800/1766416869' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:28 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:28.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='osd.4 v1:172.21.15.192:6800/1766416869' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='osd.4 v1:172.21.15.192:6800/1766416869' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:28.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:28 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:28.869 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T02:57:28.871 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-03-26T02:57:28.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T02:57:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:57:29.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-mon[33110]: from='osd.4 v1:172.21.15.192:6800/1766416869' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:29.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-mon[33110]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T02:57:29.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:29.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:29.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[37920]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:57:29.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[37920]: from='osd.4 v1:172.21.15.192:6800/1766416869' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:29.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[37920]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[37920]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[29666]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[29666]: from='osd.4 v1:172.21.15.192:6800/1766416869' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[29666]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:29 smithi066 ceph-mon[29666]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:29.939 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:57:30.024 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T02:57:29.741+0000 7fcd934b9700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T02:57:30.024 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 02:57:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T02:57:29.744+0000 7fcd8b2a4700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:57:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[29666]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[29666]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:57:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[29666]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T02:57:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[29666]: from='osd.4 ' entity='osd.4' 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[37920]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[37920]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[37920]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:30 smithi066 ceph-mon[37920]: from='osd.4 ' entity='osd.4' 2024-03-26T02:57:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:30 smithi192 ceph-mon[33110]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-26T02:57:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:30 smithi192 ceph-mon[33110]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:57:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:30 smithi192 ceph-mon[33110]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T02:57:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:30 smithi192 ceph-mon[33110]: from='osd.4 ' entity='osd.4' 2024-03-26T02:57:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:31 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:57:31.408 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:31 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:57:31.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:31 smithi192 ceph-mon[33110]: osd.4 v1:172.21.15.192:6800/1766416869 boot 2024-03-26T02:57:31.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:31 smithi192 ceph-mon[33110]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T02:57:31.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:31 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:31.409 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:31 smithi192 ceph-mon[33110]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T02:57:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:57:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[29666]: osd.4 v1:172.21.15.192:6800/1766416869 boot 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[29666]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[29666]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[37920]: osd.4 v1:172.21.15.192:6800/1766416869 boot 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[37920]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:57:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:31 smithi066 ceph-mon[37920]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T02:57:32.094 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T02:57:32.094 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T02:57:32.094 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-26T02:57:32.094 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-26T02:57:32.094 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121755 s, 861 MB/s 2024-03-26T02:57:32.095 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-26T02:57:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:32 smithi192 ceph-mon[33110]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:32.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:32 smithi066 ceph-mon[29666]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:32.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:32 smithi066 ceph-mon[37920]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:32.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-03-26T02:57:33.196 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:33.404 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:33 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T02:57:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:33.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:33 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:34.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[37920]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T02:57:34.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi192 2024-03-26T02:57:34.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi192 to 17088M 2024-03-26T02:57:34.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[37920]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T02:57:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[29666]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T02:57:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi192 2024-03-26T02:57:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi192 to 17088M 2024-03-26T02:57:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:34 smithi066 ceph-mon[29666]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T02:57:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:34 smithi192 ceph-mon[33110]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T02:57:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:34 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi192 2024-03-26T02:57:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:34 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi192 to 17088M 2024-03-26T02:57:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:34 smithi192 ceph-mon[33110]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T02:57:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[29666]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[29666]: from='client.24259 v1:172.21.15.192:0/2356175624' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[37920]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[37920]: from='client.24259 v1:172.21.15.192:0/2356175624' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:35 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:35 smithi192 ceph-mon[33110]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T02:57:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:35 smithi192 ceph-mon[33110]: from='client.24259 v1:172.21.15.192:0/2356175624' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:35 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:35 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:35.659 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:35 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:36.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:36 smithi192 ceph-mon[33110]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T02:57:36.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:36 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:57:36.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:36 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:57:36.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:36 smithi192 ceph-mon[33110]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T02:57:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[29666]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T02:57:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:57:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:57:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[29666]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T02:57:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[37920]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T02:57:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:57:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:57:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:36 smithi066 ceph-mon[37920]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T02:57:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/1124717578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]: dispatch 2024-03-26T02:57:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:37 smithi192 ceph-mon[33110]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]: dispatch 2024-03-26T02:57:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:37 smithi192 ceph-mon[33110]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]': finished 2024-03-26T02:57:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:37 smithi192 ceph-mon[33110]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:57:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:37 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:37.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/1124717578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]: dispatch 2024-03-26T02:57:37.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[29666]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]: dispatch 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[29666]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]': finished 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[29666]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/1124717578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]: dispatch 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[37920]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]: dispatch 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[37920]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0"}]': finished 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[37920]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:57:37.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:37 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:38 smithi192 ceph-mon[33110]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T02:57:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/2128352236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:57:38.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:38 smithi066 ceph-mon[29666]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T02:57:38.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/2128352236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:57:38.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:38 smithi066 ceph-mon[37920]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T02:57:38.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/2128352236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:57:40.720 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:40 smithi192 ceph-mon[33110]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:40 smithi066 ceph-mon[29666]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:40 smithi066 ceph-mon[37920]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:42.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:42 smithi192 ceph-mon[33110]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:42.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:42 smithi066 ceph-mon[29666]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:42.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:42 smithi066 ceph-mon[37920]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:44 smithi192 ceph-mon[33110]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:57:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:44 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:44 smithi192 ceph-mon[33110]: Deploying daemon osd.5 on smithi192 2024-03-26T02:57:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[29666]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[29666]: Deploying daemon osd.5 on smithi192 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[37920]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:44 smithi066 ceph-mon[37920]: Deploying daemon osd.5 on smithi192 2024-03-26T02:57:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:46 smithi192 ceph-mon[33110]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:46.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:46 smithi066 ceph-mon[29666]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:46.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:46 smithi066 ceph-mon[37920]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:48 smithi192 ceph-mon[33110]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:48.660 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:48 smithi066 ceph-mon[29666]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:48.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:48 smithi066 ceph-mon[37920]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:49 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:49 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:49 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:50.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:49 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:49 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:49 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:49 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:49 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:49 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:51.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:50 smithi066 ceph-mon[29666]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:51.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:50 smithi066 ceph-mon[37920]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:51.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:50 smithi192 ceph-mon[33110]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:52.043 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-03-26T02:57:52.661 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T02:57:52.663 INFO:tasks.cephadm:Deploying osd.6 on smithi192 with /dev/vg_nvme/lv_2... 2024-03-26T02:57:52.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T02:57:52.986 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='osd.5 v1:172.21.15.192:6804/2546913262' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:53.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='osd.5 v1:172.21.15.192:6804/2546913262' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:53.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='osd.5 v1:172.21.15.192:6804/2546913262' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:53.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:53.977 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:53 smithi192 ceph-mon[33110]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:57:53.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:53 smithi192 ceph-mon[33110]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T02:57:53.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:53 smithi192 ceph-mon[33110]: from='osd.5 v1:172.21.15.192:6804/2546913262' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:53.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:53 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:53.978 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:53 smithi192 ceph-mon[33110]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:54.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[37920]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:57:54.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[37920]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T02:57:54.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[37920]: from='osd.5 v1:172.21.15.192:6804/2546913262' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[37920]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[29666]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[29666]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[29666]: from='osd.5 v1:172.21.15.192:6804/2546913262' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:54.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:53 smithi066 ceph-mon[29666]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:57:54.155 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:57:54.532 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T02:57:54.271+0000 7f2ddeb9f700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T02:57:54.532 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T02:57:54.274+0000 7f2dd698a700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:57:55.102 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-mon[33110]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:55.102 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-mon[33110]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:57:55.102 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-mon[33110]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T02:57:55.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.103 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:54 smithi192 ceph-mon[33110]: from='osd.5 ' entity='osd.5' 2024-03-26T02:57:55.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[37920]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:55.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[37920]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:57:55.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[37920]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T02:57:55.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[37920]: from='osd.5 ' entity='osd.5' 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[29666]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[29666]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[29666]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:54 smithi066 ceph-mon[29666]: from='osd.5 ' entity='osd.5' 2024-03-26T02:57:55.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: osd.5 v1:172.21.15.192:6804/2546913262 boot 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:55.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:55.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:55.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:55 smithi192 ceph-mon[33110]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:57:56.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:57:56.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:57:56.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: osd.5 v1:172.21.15.192:6804/2546913262 boot 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[37920]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: osd.5 v1:172.21.15.192:6804/2546913262 boot 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:57:56.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:56.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:56.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:57:56.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:55 smithi066 ceph-mon[29666]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:57:56.526 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T02:57:56.526 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T02:57:56.526 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-26T02:57:56.526 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-26T02:57:56.526 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122807 s, 854 MB/s 2024-03-26T02:57:56.526 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-26T02:57:57.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[37920]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:57:57.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi192 2024-03-26T02:57:57.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi192 to 8544M 2024-03-26T02:57:57.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[37920]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:57:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[29666]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:57:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi192 2024-03-26T02:57:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi192 to 8544M 2024-03-26T02:57:57.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:56 smithi066 ceph-mon[29666]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:57:57.112 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:56 smithi192 ceph-mon[33110]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:57:57.112 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:56 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi192 2024-03-26T02:57:57.112 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:56 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi192 to 8544M 2024-03-26T02:57:57.112 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:56 smithi192 ceph-mon[33110]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:57:57.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-03-26T02:57:57.374 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:57:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:58 smithi192 ceph-mon[33110]: pgmap v136: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:57:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:58 smithi192 ceph-mon[33110]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T02:57:59.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:58 smithi066 ceph-mon[29666]: pgmap v136: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:57:59.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:58 smithi066 ceph-mon[29666]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T02:57:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:58 smithi066 ceph-mon[37920]: pgmap v136: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:57:59.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:58 smithi066 ceph-mon[37920]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T02:57:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:59 smithi192 ceph-mon[33110]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T02:57:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:59 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:57:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:59 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:57:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:57:59 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[29666]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T02:58:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:58:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:58:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[37920]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T02:58:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:58:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:58:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:57:59 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:01.023 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:00 smithi192 ceph-mon[33110]: from='client.24286 v1:172.21.15.192:0/2027168660' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:01.023 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:00 smithi192 ceph-mon[33110]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:01.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:00 smithi066 ceph-mon[29666]: from='client.24286 v1:172.21.15.192:0/2027168660' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:01.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:00 smithi066 ceph-mon[29666]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:00 smithi066 ceph-mon[37920]: from='client.24286 v1:172.21.15.192:0/2027168660' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:00 smithi066 ceph-mon[37920]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:02.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/2823088949' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]: dispatch 2024-03-26T02:58:02.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]: dispatch 2024-03-26T02:58:02.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]': finished 2024-03-26T02:58:02.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:58:02.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:02.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T02:58:02.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:01 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T02:58:02.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/2823088949' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]: dispatch 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]: dispatch 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]': finished 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/2823088949' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]: dispatch 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]: dispatch 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb"}]': finished 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T02:58:02.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:01 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T02:58:03.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:02 smithi192 ceph-mon[33110]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:03.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:02 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/542069183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:58:03.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:02 smithi066 ceph-mon[29666]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:03.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:02 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/542069183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:58:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:02 smithi066 ceph-mon[37920]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:02 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/542069183' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:58:05.111 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:04 smithi192 ceph-mon[33110]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:05.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:04 smithi066 ceph-mon[29666]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:05.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:04 smithi066 ceph-mon[37920]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:06 smithi192 ceph-mon[33110]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:06 smithi066 ceph-mon[29666]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:06 smithi066 ceph-mon[37920]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:08.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:08 smithi192 ceph-mon[33110]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:08.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:58:08.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:08 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:09.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:08 smithi066 ceph-mon[29666]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:58:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:08 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:08 smithi066 ceph-mon[37920]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:08 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:58:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:08 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:10.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:09 smithi192 ceph-mon[33110]: Deploying daemon osd.6 on smithi192 2024-03-26T02:58:10.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:09 smithi066 ceph-mon[29666]: Deploying daemon osd.6 on smithi192 2024-03-26T02:58:10.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:09 smithi066 ceph-mon[37920]: Deploying daemon osd.6 on smithi192 2024-03-26T02:58:11.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:10 smithi192 ceph-mon[33110]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:11.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:10 smithi066 ceph-mon[29666]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:10 smithi066 ceph-mon[37920]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:13.181 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:12 smithi192 ceph-mon[33110]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:13.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:12 smithi066 ceph-mon[29666]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:13.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:12 smithi066 ceph-mon[37920]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:14 smithi192 ceph-mon[33110]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[29666]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[37920]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:16.733 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:16 smithi192 ceph-mon[33110]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:16.783 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 6 on host 'smithi192' 2024-03-26T02:58:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:16 smithi066 ceph-mon[29666]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:16 smithi066 ceph-mon[37920]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:17.355 DEBUG:teuthology.orchestra.run.smithi192:osd.6> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T02:58:17.357 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_1... 2024-03-26T02:58:17.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T02:58:17.652 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:58:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:17 smithi192 ceph-mon[33110]: from='osd.6 v1:172.21.15.192:6808/4039268292' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:58:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:17 smithi192 ceph-mon[33110]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:58:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:17.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[29666]: from='osd.6 v1:172.21.15.192:6808/4039268292' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[29666]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[37920]: from='osd.6 v1:172.21.15.192:6808/4039268292' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[37920]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: from='osd.6 v1:172.21.15.192:6808/4039268292' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:18.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:18.677 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T02:58:18.446+0000 7f2f362cf700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T02:58:18.677 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 02:58:18 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T02:58:18.453+0000 7f2f320c2700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: from='osd.6 v1:172.21.15.192:6808/4039268292' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:18.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: from='osd.6 v1:172.21.15.192:6808/4039268292' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:18 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:19.618 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:58:19.618 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T02:58:19.618 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.618 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: osd.6 v1:172.21.15.192:6808/4039268292 boot 2024-03-26T02:58:19.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:58:19.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:19 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:58:19.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T02:58:19.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: osd.6 v1:172.21.15.192:6808/4039268292 boot 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: osd.6 v1:172.21.15.192:6808/4039268292 boot 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:58:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:19 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:58:20.081 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T02:58:20.082 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T02:58:20.082 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-26T02:58:20.082 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-26T02:58:20.082 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123079 s, 852 MB/s 2024-03-26T02:58:20.082 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-26T02:58:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:20 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:58:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:20 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:58:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:20 smithi192 ceph-mon[33110]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:20 smithi192 ceph-mon[33110]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:58:20.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-03-26T02:58:20.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[29666]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[29666]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[37920]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:58:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:20 smithi066 ceph-mon[37920]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:58:21.246 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:58:21.867 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:21 smithi192 ceph-mon[33110]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:58:22.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:21 smithi066 ceph-mon[29666]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:58:22.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:21 smithi066 ceph-mon[37920]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:58:23.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:23.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:23.111 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:22 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:23.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T02:58:23.165 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:23.166 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:22 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:24.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:23 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi192 2024-03-26T02:58:24.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:23 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi192 to 5696M 2024-03-26T02:58:24.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:23 smithi192 ceph-mon[33110]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T02:58:24.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:23 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:58:24.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:23 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:58:24.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:23 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:24.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi192 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi192 to 5696M 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[29666]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi192 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi192 to 5696M 2024-03-26T02:58:24.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[37920]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T02:58:24.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:58:24.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:58:24.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:23 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:25.191 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:24 smithi192 ceph-mon[33110]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:25.191 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:24 smithi192 ceph-mon[33110]: from='client.24313 v1:172.21.15.192:0/4147189297' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:24 smithi066 ceph-mon[29666]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:24 smithi066 ceph-mon[29666]: from='client.24313 v1:172.21.15.192:0/4147189297' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:24 smithi066 ceph-mon[37920]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:24 smithi066 ceph-mon[37920]: from='client.24313 v1:172.21.15.192:0/4147189297' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:26.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:25 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:58:26.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:25 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:58:26.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:25 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:58:26.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:25 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:58:26.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:25 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:58:26.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:25 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:58:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/1607611493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]: dispatch 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]: dispatch 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]': finished 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:27.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:26 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/2568478795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:58:27.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/1607611493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]: dispatch 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]: dispatch 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]': finished 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/2568478795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T02:58:27.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/1607611493' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]: dispatch 2024-03-26T02:58:27.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]: dispatch 2024-03-26T02:58:27.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b"}]': finished 2024-03-26T02:58:27.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T02:58:27.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:27.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:26 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/2568478795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:58:29.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:28 smithi192 ceph-mon[33110]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-26T02:58:29.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:28 smithi066 ceph-mon[29666]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-26T02:58:29.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:28 smithi066 ceph-mon[37920]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-26T02:58:30.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:29 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:58:30.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:29 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:58:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:29 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:58:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:29 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:58:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:29 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:58:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:29 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:58:31.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:30 smithi192 ceph-mon[33110]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-26T02:58:31.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:30 smithi192 ceph-mon[33110]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T02:58:31.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:30 smithi066 ceph-mon[29666]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-26T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:30 smithi066 ceph-mon[29666]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T02:58:31.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:30 smithi066 ceph-mon[37920]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-26T02:58:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:30 smithi066 ceph-mon[37920]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T02:58:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:33.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:32 smithi192 ceph-mon[33110]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:33.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:32 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:58:33.229 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:32 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:33.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:32 smithi066 ceph-mon[29666]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:58:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:32 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:32 smithi066 ceph-mon[37920]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:32 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:58:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:32 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:34.152 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:33 smithi192 ceph-mon[33110]: Deploying daemon osd.7 on smithi192 2024-03-26T02:58:34.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:33 smithi066 ceph-mon[29666]: Deploying daemon osd.7 on smithi192 2024-03-26T02:58:34.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:33 smithi066 ceph-mon[37920]: Deploying daemon osd.7 on smithi192 2024-03-26T02:58:35.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:34 smithi192 ceph-mon[33110]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:35.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:34 smithi066 ceph-mon[29666]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:35.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:34 smithi066 ceph-mon[37920]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:37.167 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:36 smithi192 ceph-mon[33110]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:37.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:36 smithi066 ceph-mon[29666]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:36 smithi066 ceph-mon[37920]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:39.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[37920]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[29666]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:38 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:39.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:38 smithi192 ceph-mon[33110]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:39.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:38 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:39.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:38 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:39.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:38 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:40.901 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:40 smithi192 ceph-mon[33110]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:41.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:40 smithi066 ceph-mon[29666]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:41.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:40 smithi066 ceph-mon[37920]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:41.633 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2024-03-26T02:58:41.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:41 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:41.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:41 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:41.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:41 smithi192 ceph-mon[33110]: from='osd.7 v1:172.21.15.192:6812/3100313121' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:58:41.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:41 smithi192 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:58:41.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:41 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:41.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:41 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[29666]: from='osd.7 v1:172.21.15.192:6812/3100313121' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[29666]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[37920]: from='osd.7 v1:172.21.15.192:6812/3100313121' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[37920]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:58:42.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:41 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:42.302 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T02:58:42.304 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T02:58:42.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T02:58:42.566 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: from='osd.7 v1:172.21.15.192:6812/3100313121' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:42.927 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:42.928 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T02:58:42.870+0000 7f87bbcfc700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T02:58:42.928 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 02:58:42 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T02:58:42.876+0000 7f87b7aef700 -1 osd.7 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:58:43.020 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:43.020 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:58:43.020 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:58:43.020 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: from='osd.7 v1:172.21.15.192:6812/3100313121' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[37920]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:43.021 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: from='osd.7 v1:172.21.15.192:6812/3100313121' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:43.022 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:42 smithi066 ceph-mon[29666]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-26T02:58:44.072 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:58:44.072 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: osd.7 v1:172.21.15.192:6812/3100313121 boot 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.073 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: osd.7 v1:172.21.15.192:6812/3100313121 boot 2024-03-26T02:58:44.074 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:58:44.074 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:43 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-26T02:58:44.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T02:58:44.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: osd.7 v1:172.21.15.192:6812/3100313121 boot 2024-03-26T02:58:44.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:58:44.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:43 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:58:44.491 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:58:45.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:58:45.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:58:45.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[29666]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/69045683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[29666]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[37920]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/69045683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:58:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:44 smithi066 ceph-mon[37920]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:58:45.144 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1711421923,"num_in_osds":8,"osd_in_since":1711421906,"num_remapped_pgs":1} 2024-03-26T02:58:45.144 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:58:45.221 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:44 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:58:45.221 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:44 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:58:45.221 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:44 smithi192 ceph-mon[33110]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:58:45.221 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/69045683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:58:45.221 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:44 smithi192 ceph-mon[33110]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:58:45.406 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: Detected new or changed devices on smithi192 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi192 to 4272M 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[29666]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:58:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: Detected new or changed devices on smithi192 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi192 to 4272M 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:46.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:46.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:46.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:46 smithi066 ceph-mon[37920]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:58:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: Detected new or changed devices on smithi192 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi192 to 4272M 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:46.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:46.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:46.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:46.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:46 smithi192 ceph-mon[33110]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:58:47.374 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:58:47.374 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":61,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","created":"2024-03-26T02:53:28.887014+0000","modified":"2024-03-26T02:58:46.874917+0000","last_up_change":"2024-03-26T02:58:43.841339+0000","last_in_change":"2024-03-26T02:58:26.140451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:56:43.176262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90c9bf50-0b39-4fd1-9f00-b02800f07bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6801","nonce":1028658447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6802","nonce":1028658447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6804","nonce":1028658447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6803","nonce":1028658447}]},"public_addr":"172.21.15.66:6801/1028658447","cluster_addr":"172.21.15.66:6802/1028658447","heartbeat_back_addr":"172.21.15.66:6804/1028658447","heartbeat_front_addr":"172.21.15.66:6803/1028658447","state":["exists","up"]},{"osd":1,"uuid":"b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6805","nonce":3170415936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6806","nonce":3170415936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6808","nonce":3170415936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6807","nonce":3170415936}]},"public_addr":"172.21.15.66:6805/3170415936","cluster_addr":"172.21.15.66:6806/3170415936","heartbeat_back_addr":"172.21.15.66:6808/3170415936","heartbeat_front_addr":"172.21.15.66:6807/3170415936","state":["exists","up"]},{"osd":2,"uuid":"76639ea3-00c7-4693-9eb2-3421f234283b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6809","nonce":2520642381}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6810","nonce":2520642381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6812","nonce":2520642381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6811","nonce":2520642381}]},"public_addr":"172.21.15.66:6809/2520642381","cluster_addr":"172.21.15.66:6810/2520642381","heartbeat_back_addr":"172.21.15.66:6812/2520642381","heartbeat_front_addr":"172.21.15.66:6811/2520642381","state":["exists","up"]},{"osd":3,"uuid":"2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6813","nonce":2366374823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6814","nonce":2366374823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6816","nonce":2366374823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6815","nonce":2366374823}]},"public_addr":"172.21.15.66:6813/2366374823","cluster_addr":"172.21.15.66:6814/2366374823","heartbeat_back_addr":"172.21.15.66:6816/2366374823","heartbeat_front_addr":"172.21.15.66:6815/2366374823","state":["exists","up"]},{"osd":4,"uuid":"4fae794b-e89c-4931-9ed3-0cc32b445883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":1766416869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":1766416869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":1766416869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":1766416869}]},"public_addr":"172.21.15.192:6800/1766416869","cluster_addr":"172.21.15.192:6801/1766416869","heartbeat_back_addr":"172.21.15.192:6803/1766416869","heartbeat_front_addr":"172.21.15.192:6802/1766416869","state":["exists","up"]},{"osd":5,"uuid":"04b178c3-9be8-466c-822c-0acfca8a92b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":2546913262}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":2546913262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":2546913262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":2546913262}]},"public_addr":"172.21.15.192:6804/2546913262","cluster_addr":"172.21.15.192:6805/2546913262","heartbeat_back_addr":"172.21.15.192:6807/2546913262","heartbeat_front_addr":"172.21.15.192:6806/2546913262","state":["exists","up"]},{"osd":6,"uuid":"fc63b52a-e4fe-4396-bf54-2e38696aedcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":4039268292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":4039268292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":4039268292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":4039268292}]},"public_addr":"172.21.15.192:6808/4039268292","cluster_addr":"172.21.15.192:6809/4039268292","heartbeat_back_addr":"172.21.15.192:6811/4039268292","heartbeat_front_addr":"172.21.15.192:6810/4039268292","state":["exists","up"]},{"osd":7,"uuid":"e2f4cedd-43af-4261-9f86-c4b72c97bd1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":3100313121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":3100313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":3100313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":3100313121}]},"public_addr":"172.21.15.192:6812/3100313121","cluster_addr":"172.21.15.192:6813/3100313121","heartbeat_back_addr":"172.21.15.192:6815/3100313121","heartbeat_front_addr":"172.21.15.192:6814/3100313121","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:25.302267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:57:35.184490+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:57:55.123467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:29.694738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:41.132374+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:57:52.887857+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:17.602161+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:41.861296+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:0/2719981294":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/858010278":"2024-03-27T02:54:29.078830+0000","172.21.15.66:6800/2558488368":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/2641107787":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/5961410":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/3720619020":"2024-03-27T02:54:01.878928+0000","172.21.15.66:6800/2059693071":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/952321413":"2024-03-27T02:54:29.078830+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:58:47.929 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T02:58:47.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T02:58:47.959 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:47 smithi066 ceph-mon[29666]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:58:47.959 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:47 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1632080746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:58:47.959 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:47 smithi066 ceph-mon[37920]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:58:47.959 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:47 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1632080746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:58:48.181 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:58:48.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:47 smithi192 ceph-mon[33110]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:58:48.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:47 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1632080746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:58:49.206 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:48 smithi066 ceph-mon[29666]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:49.206 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:48 smithi066 ceph-mon[29666]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:58:49.206 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:48 smithi066 ceph-mon[37920]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:49.206 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:48 smithi066 ceph-mon[37920]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:58:49.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:48 smithi192 ceph-mon[33110]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:49.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:48 smithi192 ceph-mon[33110]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:58:49.928 INFO:teuthology.orchestra.run.smithi066.stdout:pg_num: 1 2024-03-26T02:58:50.475 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi066 2024-03-26T02:58:50.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi066=foo.a' 2024-03-26T02:58:50.728 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:58:51.102 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:50 smithi192 ceph-mon[33110]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-03-26T02:58:51.102 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:50 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1202115447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:58:51.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:50 smithi066 ceph-mon[29666]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-03-26T02:58:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:50 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1202115447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:58:51.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:50 smithi066 ceph-mon[37920]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-03-26T02:58:51.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:50 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1202115447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:58:52.489 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled rgw.foo.a update... 2024-03-26T02:58:53.086 DEBUG:teuthology.orchestra.run.smithi066:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T02:58:53.089 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi192 2024-03-26T02:58:53.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.195 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.196 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:52 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:53.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:53.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:52 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:53.325 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:58:54.107 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:53 smithi192 ceph-mon[33110]: from='client.24358 v1:172.21.15.192:0/3500535064' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi066=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:54.107 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:53 smithi192 ceph-mon[33110]: Saving service rgw.foo.a spec with placement smithi066=foo.a;count:1 2024-03-26T02:58:54.107 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:53 smithi192 ceph-mon[33110]: Deploying daemon rgw.foo.a on smithi066 2024-03-26T02:58:54.180 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:53 smithi066 ceph-mon[37920]: from='client.24358 v1:172.21.15.192:0/3500535064' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi066=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:54.180 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:53 smithi066 ceph-mon[37920]: Saving service rgw.foo.a spec with placement smithi066=foo.a;count:1 2024-03-26T02:58:54.180 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:53 smithi066 ceph-mon[37920]: Deploying daemon rgw.foo.a on smithi066 2024-03-26T02:58:54.180 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:53 smithi066 ceph-mon[29666]: from='client.24358 v1:172.21.15.192:0/3500535064' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi066=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:54.181 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:53 smithi066 ceph-mon[29666]: Saving service rgw.foo.a spec with placement smithi066=foo.a;count:1 2024-03-26T02:58:54.181 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:53 smithi066 ceph-mon[29666]: Deploying daemon rgw.foo.a on smithi066 2024-03-26T02:58:54.859 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 02:58:54 smithi066 podman[64504]: 2024-03-26 02:58:54.678195866 +0000 UTC m=+0.018345776 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:58:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:54 smithi192 ceph-mon[33110]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T02:58:55.210 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:54 smithi066 ceph-mon[29666]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T02:58:55.214 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:54 smithi066 ceph-mon[37920]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T02:58:55.215 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 02:58:55 smithi066 podman[64504]: 2024-03-26 02:58:55.036156397 +0000 UTC m=+0.376306300 container init cd8028a425d08a16e1c4f8de77bca651e82d24c1740de5dd35edf64a5114d471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T02:58:55.215 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 02:58:55 smithi066 podman[64504]: 2024-03-26 02:58:55.039203497 +0000 UTC m=+0.379353402 container start cd8028a425d08a16e1c4f8de77bca651e82d24c1740de5dd35edf64a5114d471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:58:55.608 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 02:58:55 smithi066 bash[64504]: cd8028a425d08a16e1c4f8de77bca651e82d24c1740de5dd35edf64a5114d471 2024-03-26T02:58:55.608 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 02:58:55 smithi066 systemd[1]: Started Ceph rgw.foo.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:58:55.911 INFO:teuthology.orchestra.run.smithi192.stderr:pool 'datapool' created 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/691162394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.191 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/691162394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:55 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:56.230 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/691162394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:56.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:55 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:58:56.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T02:58:56.773 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:58:57.184 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-26T02:58:57.184 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:58:57.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:58:57.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: Saving service rgw.foo.a spec with placement smithi066=foo.a;count:1 2024-03-26T02:58:57.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T02:58:57.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:58:57.185 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1987075635' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: Saving service rgw.foo.a spec with placement smithi066=foo.a;count:1 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1987075635' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:58:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:58:57.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: Saving service rgw.foo.a spec with placement smithi066=foo.a;count:1 2024-03-26T02:58:57.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T02:58:57.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:58:57.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1987075635' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1987075635' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:58.233 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:57 smithi192 ceph-mon[33110]: Checking dashboard <-> RGW credentials 2024-03-26T02:58:58.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1987075635' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[29666]: Checking dashboard <-> RGW credentials 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1987075635' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:58:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:58.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:58:58.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:58:58.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:58:58.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:57 smithi066 ceph-mon[37920]: Checking dashboard <-> RGW credentials 2024-03-26T02:58:59.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: pgmap v188: 36 pgs: 32 unknown, 4 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/2985800623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:58:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:58 smithi192 ceph-mon[33110]: from='client.24415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:58:59.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: pgmap v188: 36 pgs: 32 unknown, 4 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:58:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:58:59.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/2985800623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[29666]: from='client.24415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: pgmap v188: 36 pgs: 32 unknown, 4 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/2985800623' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:58:59.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:58 smithi066 ceph-mon[37920]: from='client.24415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:59:00.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='client.24415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:58:59 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='client.24415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:59:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='client.24415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:00.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:58:59 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:59:01.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:00 smithi192 ceph-mon[33110]: pgmap v191: 68 pgs: 13 creating+peering, 25 active+clean, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:59:01.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:00 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:59:01.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:00 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T02:59:01.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:59:01.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:00 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:59:01.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:00 smithi192 ceph-mon[33110]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:59:01.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[29666]: pgmap v191: 68 pgs: 13 creating+peering, 25 active+clean, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[29666]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[37920]: pgmap v191: 68 pgs: 13 creating+peering, 25 active+clean, 30 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:59:01.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:00 smithi066 ceph-mon[37920]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:59:01.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.192 --placement '1;smithi192=iscsi.a' 2024-03-26T02:59:01.769 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:59:03.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:02 smithi192 ceph-mon[33110]: pgmap v194: 100 pgs: 17 creating+peering, 50 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T02:59:03.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:02 smithi192 ceph-mon[33110]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:59:03.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:02 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:02 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.062 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:02 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[29666]: pgmap v194: 100 pgs: 17 creating+peering, 50 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[29666]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[37920]: pgmap v194: 100 pgs: 17 creating+peering, 50 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[37920]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:02 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:59:03.512 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled iscsi.datapool update... 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.125 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:03 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:04.162 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T02:59:04.162 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:59:04.163 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.202 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2004587607' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:59:04.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:03 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:04.209 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:59:04.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T02:59:04.242 DEBUG:teuthology.orchestra.run.smithi192:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T02:59:04.284 INFO:tasks.cephadm:Adding prometheus.a on smithi192 2024-03-26T02:59:04.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi192=a' 2024-03-26T02:59:04.574 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:59:04.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: pgmap v197: 132 pgs: 1 creating+activating, 17 creating+peering, 69 active+clean, 45 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T02:59:04.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='client.24421 v1:172.21.15.192:0/3610717557' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:04.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2024-03-26T02:59:04.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:59:04.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:59:04.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:59:04.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:04.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:04.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:04.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:59:04.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:59:04.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:04 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: pgmap v197: 132 pgs: 1 creating+activating, 17 creating+peering, 69 active+clean, 45 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='client.24421 v1:172.21.15.192:0/3610717557' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: pgmap v197: 132 pgs: 1 creating+activating, 17 creating+peering, 69 active+clean, 45 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='client.24421 v1:172.21.15.192:0/3610717557' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/149986303' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:59:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='client.34346 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:59:05.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:04 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:59:06.227 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:05 smithi192 ceph-mon[33110]: Deploying daemon iscsi.iscsi.a on smithi192 2024-03-26T02:59:06.336 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled prometheus update... 2024-03-26T02:59:06.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:05 smithi066 ceph-mon[29666]: Deploying daemon iscsi.iscsi.a on smithi192 2024-03-26T02:59:06.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:05 smithi066 ceph-mon[37920]: Deploying daemon iscsi.iscsi.a on smithi192 2024-03-26T02:59:07.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:06 smithi192 ceph-mon[33110]: pgmap v199: 132 pgs: 1 creating+activating, 9 creating+peering, 117 active+clean, 5 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 64 op/s 2024-03-26T02:59:07.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:06 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:07.311 DEBUG:teuthology.orchestra.run.smithi192:prometheus.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T02:59:07.314 INFO:tasks.cephadm:Adding node-exporter.a on smithi066 2024-03-26T02:59:07.314 INFO:tasks.cephadm:Adding node-exporter.b on smithi192 2024-03-26T02:59:07.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi066=a;smithi192=b' 2024-03-26T02:59:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:06 smithi066 ceph-mon[29666]: pgmap v199: 132 pgs: 1 creating+activating, 9 creating+peering, 117 active+clean, 5 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 64 op/s 2024-03-26T02:59:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:06 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:06 smithi066 ceph-mon[37920]: pgmap v199: 132 pgs: 1 creating+activating, 9 creating+peering, 117 active+clean, 5 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 1.9 KiB/s wr, 64 op/s 2024-03-26T02:59:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:06 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:07.770 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:59:08.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:07 smithi192 ceph-mon[33110]: from='client.24442 v1:172.21.15.192:0/2386835916' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:08.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:07 smithi192 ceph-mon[33110]: Saving service prometheus spec with placement smithi192=a;count:1 2024-03-26T02:59:08.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:07 smithi066 ceph-mon[29666]: from='client.24442 v1:172.21.15.192:0/2386835916' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:08.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:07 smithi066 ceph-mon[29666]: Saving service prometheus spec with placement smithi192=a;count:1 2024-03-26T02:59:08.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:07 smithi066 ceph-mon[37920]: from='client.24442 v1:172.21.15.192:0/2386835916' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:08.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:07 smithi066 ceph-mon[37920]: Saving service prometheus spec with placement smithi192=a;count:1 2024-03-26T02:59:09.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:08 smithi192 ceph-mon[33110]: pgmap v200: 132 pgs: 1 creating+activating, 131 active+clean; 580 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 3.7 KiB/s wr, 154 op/s 2024-03-26T02:59:09.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:08 smithi192 ceph-mon[33110]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:59:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:08 smithi066 ceph-mon[29666]: pgmap v200: 132 pgs: 1 creating+activating, 131 active+clean; 580 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 3.7 KiB/s wr, 154 op/s 2024-03-26T02:59:09.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:08 smithi066 ceph-mon[29666]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:59:09.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:08 smithi066 ceph-mon[37920]: pgmap v200: 132 pgs: 1 creating+activating, 131 active+clean; 580 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 3.7 KiB/s wr, 154 op/s 2024-03-26T02:59:09.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:08 smithi066 ceph-mon[37920]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:59:09.752 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled node-exporter update... 2024-03-26T02:59:10.456 DEBUG:teuthology.orchestra.run.smithi066:node-exporter.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T02:59:10.459 DEBUG:teuthology.orchestra.run.smithi192:node-exporter.b> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T02:59:10.462 INFO:tasks.cephadm:Adding alertmanager.a on smithi066 2024-03-26T02:59:10.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi066=a' 2024-03-26T02:59:11.058 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 6.1 KiB/s wr, 259 op/s 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: from='client.24448 v1:172.21.15.192:0/344361618' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi066=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: Saving service node-exporter spec with placement smithi066=a;smithi192=b;count:2 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.059 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:10 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 6.1 KiB/s wr, 259 op/s 2024-03-26T02:59:11.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: from='client.24448 v1:172.21.15.192:0/344361618' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi066=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: Saving service node-exporter spec with placement smithi066=a;smithi192=b;count:2 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s rd, 6.1 KiB/s wr, 259 op/s 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: from='client.24448 v1:172.21.15.192:0/344361618' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi066=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: Saving service node-exporter spec with placement smithi066=a;smithi192=b;count:2 2024-03-26T02:59:11.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:10 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:11.239 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:59:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:11 smithi192 ceph-mon[33110]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T02:59:11.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:11 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:59:11.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/1927001832' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:59:12.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:11 smithi066 ceph-mon[29666]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T02:59:12.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:11 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:59:12.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/1927001832' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:59:12.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:11 smithi066 ceph-mon[37920]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T02:59:12.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:11 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:59:12.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/1927001832' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:59:13.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:12 smithi066 ceph-mon[29666]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 200 op/s 2024-03-26T02:59:13.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:12 smithi066 ceph-mon[37920]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 200 op/s 2024-03-26T02:59:13.154 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:12 smithi192 ceph-mon[33110]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 200 op/s 2024-03-26T02:59:13.190 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled alertmanager update... 2024-03-26T02:59:13.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:13 smithi192 ceph-mon[33110]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:59:13.851 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:13 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:14.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:13 smithi066 ceph-mon[29666]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:59:14.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:13 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:13 smithi066 ceph-mon[37920]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:59:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:13 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:14.152 DEBUG:teuthology.orchestra.run.smithi066:alertmanager.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T02:59:14.155 INFO:tasks.cephadm:Adding grafana.a on smithi192 2024-03-26T02:59:14.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi192=a' 2024-03-26T02:59:15.057 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:59:15.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[29666]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 3.9 KiB/s wr, 175 op/s 2024-03-26T02:59:15.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[29666]: from='client.24484 v1:172.21.15.192:0/1147552608' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:15.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[29666]: Saving service alertmanager spec with placement smithi066=a;count:1 2024-03-26T02:59:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:15.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[37920]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 3.9 KiB/s wr, 175 op/s 2024-03-26T02:59:15.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[37920]: from='client.24484 v1:172.21.15.192:0/1147552608' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:15.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[37920]: Saving service alertmanager spec with placement smithi066=a;count:1 2024-03-26T02:59:15.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:14 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:15.179 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:14 smithi192 ceph-mon[33110]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 3.9 KiB/s wr, 175 op/s 2024-03-26T02:59:15.179 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:14 smithi192 ceph-mon[33110]: from='client.24484 v1:172.21.15.192:0/1147552608' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi066=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:15.179 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:14 smithi192 ceph-mon[33110]: Saving service alertmanager spec with placement smithi066=a;count:1 2024-03-26T02:59:15.180 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:14 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.044 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.045 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.045 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:59:16.045 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:59:16.045 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.045 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:59:16.045 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:15 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:59:16.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:59:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:15 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:59:16.768 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled grafana update... 2024-03-26T02:59:17.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[29666]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 3.1 KiB/s wr, 134 op/s 2024-03-26T02:59:17.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[29666]: Deploying daemon prometheus.a on smithi192 2024-03-26T02:59:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[29666]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:59:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:17.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[37920]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 3.1 KiB/s wr, 134 op/s 2024-03-26T02:59:17.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[37920]: Deploying daemon prometheus.a on smithi192 2024-03-26T02:59:17.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[37920]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:59:17.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:17 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:17 smithi192 ceph-mon[33110]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 3.1 KiB/s wr, 134 op/s 2024-03-26T02:59:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:17 smithi192 ceph-mon[33110]: Deploying daemon prometheus.a on smithi192 2024-03-26T02:59:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:17 smithi192 ceph-mon[33110]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:59:17.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:17 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:18.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:18 smithi192 ceph-mon[33110]: from='client.24490 v1:172.21.15.192:0/2313819433' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:18.218 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:18 smithi192 ceph-mon[33110]: Saving service grafana spec with placement smithi192=a;count:1 2024-03-26T02:59:18.247 DEBUG:teuthology.orchestra.run.smithi192:grafana.a> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T02:59:18.249 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T02:59:18.250 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:59:18.272 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:18 smithi066 ceph-mon[29666]: from='client.24490 v1:172.21.15.192:0/2313819433' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:18.272 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:18 smithi066 ceph-mon[29666]: Saving service grafana spec with placement smithi192=a;count:1 2024-03-26T02:59:18.272 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:18 smithi066 ceph-mon[37920]: from='client.24490 v1:172.21.15.192:0/2313819433' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:59:18.272 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:18 smithi066 ceph-mon[37920]: Saving service grafana spec with placement smithi192=a;count:1 2024-03-26T02:59:18.515 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:19.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[37920]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 370 B/s rd, 247 B/s wr, 1 op/s 2024-03-26T02:59:19.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:59:19.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:59:19.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[37920]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:59:19.329 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[29666]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 370 B/s rd, 247 B/s wr, 1 op/s 2024-03-26T02:59:19.329 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:59:19.329 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:59:19.329 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:19 smithi066 ceph-mon[29666]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:59:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:19 smithi192 ceph-mon[33110]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 370 B/s rd, 247 B/s wr, 1 op/s 2024-03-26T02:59:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:19 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:59:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:19 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:59:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:19 smithi192 ceph-mon[33110]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:59:20.454 INFO:teuthology.orchestra.run.smithi066.stdout:[client.0] 2024-03-26T02:59:20.455 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQAIOgJmSTTVGhAAGBJnW5BUXImLNPICo3/Tmg== 2024-03-26T02:59:21.041 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-26T02:59:21.042 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T02:59:21.042 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T02:59:21.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:21 smithi066 ceph-mon[29666]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-03-26T02:59:21.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:21 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3165707267' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:21.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:21 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3165707267' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:59:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:21 smithi066 ceph-mon[37920]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-03-26T02:59:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:21 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3165707267' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:21 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3165707267' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:59:21.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:59:21.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:21 smithi192 ceph-mon[33110]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 127 B/s wr, 2 op/s 2024-03-26T02:59:21.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:21 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3165707267' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:21.161 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:21 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3165707267' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:59:22.313 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:59:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:22 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:22 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:22 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:23.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:23 smithi066 ceph-mon[29666]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:59:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:23 smithi066 ceph-mon[37920]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:59:23.363 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:23 smithi192 ceph-mon[33110]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:59:24.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:24 smithi066 ceph-mon[29666]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:59:24.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:24 smithi066 ceph-mon[37920]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:59:24.376 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:24 smithi192 ceph-mon[33110]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:59:25.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:25 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:59:25.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:25 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:59:25.284 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:25 smithi192 ceph-mon[33110]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:59:25.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:25 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:59:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:25 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:59:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:25 smithi066 ceph-mon[37920]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:59:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:25 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:59:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:25 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:59:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:25 smithi066 ceph-mon[29666]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:59:26.046 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-03-26T02:59:26.047 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAOOgJmZwyCAhAAPW9/6VfvYYcRSyfGZrFbmw== 2024-03-26T02:59:26.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:26 smithi066 ceph-mon[29666]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:59:26.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:26 smithi066 ceph-mon[37920]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:59:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:26 smithi192 ceph-mon[33110]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:59:26.771 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-26T02:59:26.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T02:59:26.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T02:59:26.855 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T02:59:26.855 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T02:59:26.855 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T02:59:27.122 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:27.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.192:0/3294629832' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:27.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:27 smithi066 ceph-mon[29666]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:27.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:27 smithi066 ceph-mon[29666]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:59:27.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.192:0/3294629832' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:27.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:27 smithi066 ceph-mon[37920]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:27.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:27 smithi066 ceph-mon[37920]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:59:27.264 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.192:0/3294629832' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:27.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:27 smithi192 ceph-mon[33110]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:59:27.265 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:27 smithi192 ceph-mon[33110]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:59:28.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:28 smithi066 ceph-mon[29666]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-03-26T02:59:28.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:28 smithi066 ceph-mon[37920]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-03-26T02:59:28.436 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:28 smithi192 ceph-mon[33110]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-03-26T02:59:28.737 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:28 smithi192 systemd[1]: Starting Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T02:59:29.037 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:59:29.092 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:28 smithi192 podman[61957]: 2024-03-26 02:59:28.736764218 +0000 UTC m=+0.018582981 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T02:59:29.092 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:28 smithi192 podman[61957]: 2024-03-26 02:59:28.856421437 +0000 UTC m=+0.138240193 container create c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:29.092 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 podman[61957]: 2024-03-26 02:59:29.092162536 +0000 UTC m=+0.373981290 container init c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 podman[61957]: 2024-03-26 02:59:29.094825247 +0000 UTC m=+0.376644001 container start c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.118Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.118Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.118Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi192 (none))" 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.118Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.118Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:59:29.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.119Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.119Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.121Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.121Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.899µs 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=16.763µs wal_replay_duration=198.957µs wbl_replay_duration=114ns total_replay_duration=233.365µs 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.123Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.144Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.429377ms db_storage=767ns remote_storage=1.157µs web_handler=613ns query_engine=499ns scrape=374.958µs scrape_sd=73.985µs notify=690ns notify_sd=1.89µs rules=19.696649ms tracing=6.653µs 2024-03-26T02:59:29.489 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.144Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T02:59:29.490 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T02:59:29.144Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T02:59:29.490 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 bash[61957]: c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e 2024-03-26T02:59:29.490 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 02:59:29 smithi192 systemd[1]: Started Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T02:59:29.711 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6800","nonce":2473255846}]},"active_addr":"172.21.15.66:6800/2473255846","active_change":"2024-03-26T02:54:29.078946+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.66:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":2101239378}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3105618869}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3359773304}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":1321389422}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.66:0","nonce":3068874001}]}]} 2024-03-26T02:59:29.715 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T02:59:29.715 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T02:59:29.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:59:29.971 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:30.093 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/521167096' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.094 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:59:30.095 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/521167096' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:59:30.095 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-26T02:59:30.095 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-26T02:59:30.095 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T02:59:30.095 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T02:59:30.095 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.096 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.096 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.096 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:29 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:59:30.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/521167096' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:59:30.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' 2024-03-26T02:59:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:29 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:59:30.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ignoring --setuser ceph since I am not root 2024-03-26T02:59:30.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ignoring --setgroup ceph since I am not root 2024-03-26T02:59:30.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: ignoring --setuser ceph since I am not root 2024-03-26T02:59:30.738 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: ignoring --setgroup ceph since I am not root 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:30.876+0000 7fa32d9d3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:30.976+0000 7fa32d9d3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T02:59:30.978 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[37920]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:59:30.979 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:30 smithi066 ceph-mon[29666]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T02:59:31.237 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:30.822+0000 7fcfd4211200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:59:31.237 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:30.923+0000 7fcfd4211200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: from='mgr.14150 v1:172.21.15.66:0/4192186223' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:59:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:30 smithi192 ceph-mon[33110]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T02:59:31.865 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:31.557+0000 7fcfd4211200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:59:31.866 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:31.648+0000 7fcfd4211200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:59:31.866 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:31.788+0000 7fcfd4211200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:59:31.915 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:31 smithi066 ceph-mon[29666]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:59:31.916 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:31 smithi066 ceph-mon[37920]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:59:31.916 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:31 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:31.662+0000 7fa32d9d3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:59:31.916 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:31 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:31.763+0000 7fa32d9d3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:59:31.916 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:31 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:31.914+0000 7fa32d9d3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:59:31.967 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:59:31.968 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":77,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","created":"2024-03-26T02:53:28.887014+0000","modified":"2024-03-26T02:59:30.857409+0000","last_up_change":"2024-03-26T02:58:43.841339+0000","last_in_change":"2024-03-26T02:58:26.140451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:56:43.176262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T02:58:55.077781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T02:58:55.118069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:58:56.987641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:58:58.940665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:59:01.006117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90c9bf50-0b39-4fd1-9f00-b02800f07bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6801","nonce":1028658447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6802","nonce":1028658447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6804","nonce":1028658447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6803","nonce":1028658447}]},"public_addr":"172.21.15.66:6801/1028658447","cluster_addr":"172.21.15.66:6802/1028658447","heartbeat_back_addr":"172.21.15.66:6804/1028658447","heartbeat_front_addr":"172.21.15.66:6803/1028658447","state":["exists","up"]},{"osd":1,"uuid":"b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6805","nonce":3170415936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6806","nonce":3170415936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6808","nonce":3170415936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6807","nonce":3170415936}]},"public_addr":"172.21.15.66:6805/3170415936","cluster_addr":"172.21.15.66:6806/3170415936","heartbeat_back_addr":"172.21.15.66:6808/3170415936","heartbeat_front_addr":"172.21.15.66:6807/3170415936","state":["exists","up"]},{"osd":2,"uuid":"76639ea3-00c7-4693-9eb2-3421f234283b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6809","nonce":2520642381}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6810","nonce":2520642381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6812","nonce":2520642381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6811","nonce":2520642381}]},"public_addr":"172.21.15.66:6809/2520642381","cluster_addr":"172.21.15.66:6810/2520642381","heartbeat_back_addr":"172.21.15.66:6812/2520642381","heartbeat_front_addr":"172.21.15.66:6811/2520642381","state":["exists","up"]},{"osd":3,"uuid":"2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6813","nonce":2366374823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6814","nonce":2366374823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6816","nonce":2366374823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6815","nonce":2366374823}]},"public_addr":"172.21.15.66:6813/2366374823","cluster_addr":"172.21.15.66:6814/2366374823","heartbeat_back_addr":"172.21.15.66:6816/2366374823","heartbeat_front_addr":"172.21.15.66:6815/2366374823","state":["exists","up"]},{"osd":4,"uuid":"4fae794b-e89c-4931-9ed3-0cc32b445883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":1766416869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":1766416869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":1766416869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":1766416869}]},"public_addr":"172.21.15.192:6800/1766416869","cluster_addr":"172.21.15.192:6801/1766416869","heartbeat_back_addr":"172.21.15.192:6803/1766416869","heartbeat_front_addr":"172.21.15.192:6802/1766416869","state":["exists","up"]},{"osd":5,"uuid":"04b178c3-9be8-466c-822c-0acfca8a92b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":2546913262}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":2546913262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":2546913262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":2546913262}]},"public_addr":"172.21.15.192:6804/2546913262","cluster_addr":"172.21.15.192:6805/2546913262","heartbeat_back_addr":"172.21.15.192:6807/2546913262","heartbeat_front_addr":"172.21.15.192:6806/2546913262","state":["exists","up"]},{"osd":6,"uuid":"fc63b52a-e4fe-4396-bf54-2e38696aedcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":4039268292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":4039268292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":4039268292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":4039268292}]},"public_addr":"172.21.15.192:6808/4039268292","cluster_addr":"172.21.15.192:6809/4039268292","heartbeat_back_addr":"172.21.15.192:6811/4039268292","heartbeat_front_addr":"172.21.15.192:6810/4039268292","state":["exists","up"]},{"osd":7,"uuid":"e2f4cedd-43af-4261-9f86-c4b72c97bd1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":3100313121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":3100313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":3100313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":3100313121}]},"public_addr":"172.21.15.192:6812/3100313121","cluster_addr":"172.21.15.192:6813/3100313121","heartbeat_back_addr":"172.21.15.192:6815/3100313121","heartbeat_front_addr":"172.21.15.192:6814/3100313121","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:25.302267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:55.209947+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:59:17.632780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:29.694738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:41.132374+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:59:15.465786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:59:23.692713+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:41.861296+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":7,"to":4}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:0/2719981294":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/858010278":"2024-03-27T02:54:29.078830+0000","172.21.15.66:6800/2558488368":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/2641107787":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/5961410":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/3720619020":"2024-03-27T02:54:01.878928+0000","172.21.15.66:6800/2059693071":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/952321413":"2024-03-27T02:54:29.078830+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:59:32.230 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:31 smithi192 ceph-mon[33110]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:59:32.230 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:31.882+0000 7fcfd4211200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:59:32.230 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:31 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:31.966+0000 7fcfd4211200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:59:32.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.021+0000 7fa32d9d3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:59:32.358 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.109+0000 7fa32d9d3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:59:32.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:32.229+0000 7fcfd4211200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:59:32.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:32.337+0000 7fcfd4211200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:59:32.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:32.424+0000 7fcfd4211200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:59:32.625 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.392+0000 7fa32d9d3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:59:32.625 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.506+0000 7fa32d9d3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:59:32.625 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.599+0000 7fa32d9d3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:59:32.653 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T02:59:32.654 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:59:32.866 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:32.518+0000 7fcfd4211200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:59:32.866 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:32.694+0000 7fcfd4211200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:59:32.899 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2135994860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:59:32.899 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2135994860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:59:32.899 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.702+0000 7fa32d9d3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:59:32.899 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:32.897+0000 7fa32d9d3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:59:32.917 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:33.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2135994860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:59:33.987 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:33.653+0000 7fcfd4211200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:59:33.987 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:33.783+0000 7fcfd4211200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:59:34.251 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:33.945+0000 7fa32d9d3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:59:34.251 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:34.092+0000 7fa32d9d3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:59:34.407 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:34.119+0000 7fcfd4211200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:59:34.407 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:34.208+0000 7fcfd4211200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:59:34.547 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:34.446+0000 7fa32d9d3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:59:34.547 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:34.546+0000 7fa32d9d3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:59:34.717 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T02:59:34.717 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":78,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","created":"2024-03-26T02:53:28.887014+0000","modified":"2024-03-26T02:59:33.853272+0000","last_up_change":"2024-03-26T02:58:43.841339+0000","last_in_change":"2024-03-26T02:58:26.140451+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:56:43.176262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T02:58:55.077781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T02:58:55.118069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:58:56.987641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:58:58.940665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:59:01.006117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90c9bf50-0b39-4fd1-9f00-b02800f07bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6801","nonce":1028658447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6802","nonce":1028658447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6804","nonce":1028658447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6803","nonce":1028658447}]},"public_addr":"172.21.15.66:6801/1028658447","cluster_addr":"172.21.15.66:6802/1028658447","heartbeat_back_addr":"172.21.15.66:6804/1028658447","heartbeat_front_addr":"172.21.15.66:6803/1028658447","state":["exists","up"]},{"osd":1,"uuid":"b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6805","nonce":3170415936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6806","nonce":3170415936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6808","nonce":3170415936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6807","nonce":3170415936}]},"public_addr":"172.21.15.66:6805/3170415936","cluster_addr":"172.21.15.66:6806/3170415936","heartbeat_back_addr":"172.21.15.66:6808/3170415936","heartbeat_front_addr":"172.21.15.66:6807/3170415936","state":["exists","up"]},{"osd":2,"uuid":"76639ea3-00c7-4693-9eb2-3421f234283b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6809","nonce":2520642381}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6810","nonce":2520642381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6812","nonce":2520642381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6811","nonce":2520642381}]},"public_addr":"172.21.15.66:6809/2520642381","cluster_addr":"172.21.15.66:6810/2520642381","heartbeat_back_addr":"172.21.15.66:6812/2520642381","heartbeat_front_addr":"172.21.15.66:6811/2520642381","state":["exists","up"]},{"osd":3,"uuid":"2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6813","nonce":2366374823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6814","nonce":2366374823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6816","nonce":2366374823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6815","nonce":2366374823}]},"public_addr":"172.21.15.66:6813/2366374823","cluster_addr":"172.21.15.66:6814/2366374823","heartbeat_back_addr":"172.21.15.66:6816/2366374823","heartbeat_front_addr":"172.21.15.66:6815/2366374823","state":["exists","up"]},{"osd":4,"uuid":"4fae794b-e89c-4931-9ed3-0cc32b445883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":1766416869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":1766416869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":1766416869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":1766416869}]},"public_addr":"172.21.15.192:6800/1766416869","cluster_addr":"172.21.15.192:6801/1766416869","heartbeat_back_addr":"172.21.15.192:6803/1766416869","heartbeat_front_addr":"172.21.15.192:6802/1766416869","state":["exists","up"]},{"osd":5,"uuid":"04b178c3-9be8-466c-822c-0acfca8a92b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":2546913262}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":2546913262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":2546913262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":2546913262}]},"public_addr":"172.21.15.192:6804/2546913262","cluster_addr":"172.21.15.192:6805/2546913262","heartbeat_back_addr":"172.21.15.192:6807/2546913262","heartbeat_front_addr":"172.21.15.192:6806/2546913262","state":["exists","up"]},{"osd":6,"uuid":"fc63b52a-e4fe-4396-bf54-2e38696aedcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":4039268292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":4039268292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":4039268292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":4039268292}]},"public_addr":"172.21.15.192:6808/4039268292","cluster_addr":"172.21.15.192:6809/4039268292","heartbeat_back_addr":"172.21.15.192:6811/4039268292","heartbeat_front_addr":"172.21.15.192:6810/4039268292","state":["exists","up"]},{"osd":7,"uuid":"e2f4cedd-43af-4261-9f86-c4b72c97bd1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":3100313121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":3100313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":3100313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":3100313121}]},"public_addr":"172.21.15.192:6812/3100313121","cluster_addr":"172.21.15.192:6813/3100313121","heartbeat_back_addr":"172.21.15.192:6815/3100313121","heartbeat_front_addr":"172.21.15.192:6814/3100313121","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:25.302267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:55.209947+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:59:17.632780+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:29.694738+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:41.132374+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:59:15.465786+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:59:23.692713+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:58:41.861296+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":7,"to":4}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:0/2719981294":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/858010278":"2024-03-27T02:54:29.078830+0000","172.21.15.66:6800/2558488368":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/2641107787":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/5961410":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/3720619020":"2024-03-27T02:54:01.878928+0000","172.21.15.66:6800/2059693071":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/952321413":"2024-03-27T02:54:29.078830+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:59:34.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:34.406+0000 7fcfd4211200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:59:34.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:34.591+0000 7fcfd4211200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:59:34.855 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:34.761+0000 7fa32d9d3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:59:35.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-mon[29666]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:59:35.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3147416062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:59:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-mon[37920]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:59:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3147416062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:59:35.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:34 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:34.961+0000 7fa32d9d3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:59:35.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-mon[33110]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:59:35.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3147416062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:59:35.140 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:34.877+0000 7fcfd4211200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:59:35.389 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T02:59:35.389 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T02:59:35.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T02:59:35.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T02:59:35.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T02:59:35.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T02:59:35.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T02:59:35.395 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T02:59:35.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:35 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:35.138+0000 7fcfd4211200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:59:35.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:35 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:35.375+0000 7fcfd4211200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:59:35.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:35 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:35.265+0000 7fa32d9d3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:59:35.942 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.949 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.951 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.951 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.951 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.955 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.959 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:35.960 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:36.057 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:35 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:35.631+0000 7fa32d9d3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:59:36.318 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:36 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:36.056+0000 7fa32d9d3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:59:36.487 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:36 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:36.170+0000 7fcfd4211200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:59:37.060 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:36 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:36.742+0000 7fcfd4211200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:59:37.060 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:36 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:36.839+0000 7fcfd4211200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:59:37.073 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:36 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:36.933+0000 7fa32d9d3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:37.059+0000 7fcfd4211200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: [26/Mar/2024:02:59:37] ENGINE Bus STARTING 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: CherryPy Checker: 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: The Application mounted at '' has an empty config. 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: [26/Mar/2024:02:59:37] ENGINE Serving on http://:::9283 2024-03-26T02:59:37.488 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x[34987]: [26/Mar/2024:02:59:37] ENGINE Bus STARTED 2024-03-26T02:59:37.723 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:37.544+0000 7fa32d9d3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:59:37.723 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:37.647+0000 7fa32d9d3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:59:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T02:59:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T02:59:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: Standby manager daemon x restarted 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: Standby manager daemon x started 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:59:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:37 smithi192 ceph-mon[33110]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: Standby manager daemon x restarted 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: Standby manager daemon x started 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:59:37.993 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[37920]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: Standby manager daemon x restarted 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: Standby manager daemon x started 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:59:37.994 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:59:37.995 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-mon[29666]: from='mgr.? v1:172.21.15.192:0/543874328' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:59:37.995 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:37.879+0000 7fa32d9d3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:59:37.995 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:02:59:37] ENGINE Bus STARTING 2024-03-26T02:59:38.247 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: CherryPy Checker: 2024-03-26T02:59:38.247 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: The Application mounted at '' has an empty config. 2024-03-26T02:59:38.247 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T02:59:38.247 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:02:59:38] ENGINE Serving on http://:::9283 2024-03-26T02:59:38.247 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:02:59:38] ENGINE Bus STARTED 2024-03-26T02:59:38.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: Active manager daemon y restarted 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: Activating manager daemon y 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: mgrmap e19: y(active, starting, since 0.00834516s), standbys: x 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:59:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:59:38.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:59:38.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: Manager daemon y is now available 2024-03-26T02:59:38.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:59:38.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:38.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:59:38.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:59:38.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:59:39.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: Active manager daemon y restarted 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: Activating manager daemon y 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: mgrmap e19: y(active, starting, since 0.00834516s), standbys: x 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:59:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: Manager daemon y is now available 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:59:39.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:59:39.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: Active manager daemon y restarted 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: Activating manager daemon y 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: mgrmap e19: y(active, starting, since 0.00834516s), standbys: x 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:59:39.112 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: Manager daemon y is now available 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:59:39.113 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:59:40.090 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:59:40.090 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: [26/Mar/2024:02:59:38] ENGINE Bus STARTING 2024-03-26T02:59:40.090 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: mgrmap e20: y(active, since 1.01826s), standbys: x 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: [26/Mar/2024:02:59:39] ENGINE Serving on https://172.21.15.66:7150 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: [26/Mar/2024:02:59:39] ENGINE Serving on http://172.21.15.66:8765 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: [26/Mar/2024:02:59:39] ENGINE Bus STARTED 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: [26/Mar/2024:02:59:38] ENGINE Bus STARTING 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: mgrmap e20: y(active, since 1.01826s), standbys: x 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:40.091 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:40.092 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: [26/Mar/2024:02:59:39] ENGINE Serving on https://172.21.15.66:7150 2024-03-26T02:59:40.092 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: [26/Mar/2024:02:59:39] ENGINE Serving on http://172.21.15.66:8765 2024-03-26T02:59:40.092 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: [26/Mar/2024:02:59:39] ENGINE Bus STARTED 2024-03-26T02:59:40.092 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T02:59:40.092 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:39 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: [26/Mar/2024:02:59:38] ENGINE Bus STARTING 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: mgrmap e20: y(active, since 1.01826s), standbys: x 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: [26/Mar/2024:02:59:39] ENGINE Serving on https://172.21.15.66:7150 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: [26/Mar/2024:02:59:39] ENGINE Serving on http://172.21.15.66:8765 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: [26/Mar/2024:02:59:39] ENGINE Bus STARTED 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T02:59:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:39 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T02:59:41.186 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:40 smithi066 ceph-mon[29666]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:41.186 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:40 smithi066 ceph-mon[29666]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:59:41.186 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:40 smithi066 ceph-mon[29666]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-26T02:59:41.187 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:40 smithi066 ceph-mon[37920]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:41.187 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:40 smithi066 ceph-mon[37920]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:59:41.187 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:40 smithi066 ceph-mon[37920]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-26T02:59:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:40 smithi192 ceph-mon[33110]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:40 smithi192 ceph-mon[33110]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:59:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:40 smithi192 ceph-mon[33110]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-26T02:59:42.553 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:42.553 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:42.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:42.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi192 to 3760M 2024-03-26T02:59:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi192 to 3760M 2024-03-26T02:59:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T02:59:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi192 to 3760M 2024-03-26T02:59:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:43 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T02:59:44.089 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:02:59:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:59:45.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:44 smithi066 ceph-mon[29666]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:59:45.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:44 smithi066 ceph-mon[29666]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:44 smithi066 ceph-mon[37920]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:59:45.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:44 smithi066 ceph-mon[37920]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:44 smithi192 ceph-mon[33110]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:59:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:44 smithi192 ceph-mon[33110]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:59:45.420 INFO:teuthology.orchestra.run.smithi066.stdout:249108103182 2024-03-26T02:59:45.421 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T02:59:46.231 INFO:teuthology.orchestra.run.smithi066.stdout:77309411364 2024-03-26T02:59:46.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T02:59:47.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:46 smithi192 ceph-mon[33110]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:59:47.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:46 smithi066 ceph-mon[29666]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:59:47.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:46 smithi066 ceph-mon[37920]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:59:47.438 INFO:teuthology.orchestra.run.smithi066.stdout:55834574891 2024-03-26T02:59:47.439 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T02:59:48.633 INFO:teuthology.orchestra.run.smithi066.stdout:34359738416 2024-03-26T02:59:48.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T02:59:49.210 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:48 smithi066 ceph-mon[29666]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T02:59:49.210 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:48 smithi066 ceph-mon[37920]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T02:59:49.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:48 smithi192 ceph-mon[33110]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T02:59:49.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 02:59:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T02:59:49.604 INFO:teuthology.orchestra.run.smithi066.stdout:133143986203 2024-03-26T02:59:49.604 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T02:59:50.131 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:49 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:50.131 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:49 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:49 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:59:50.857 INFO:teuthology.orchestra.run.smithi066.stdout:206158430226 2024-03-26T02:59:50.858 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T02:59:51.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:50 smithi066 ceph-mon[29666]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:51.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:50 smithi066 ceph-mon[37920]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:50 smithi192 ceph-mon[33110]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:52.165 INFO:teuthology.orchestra.run.smithi066.stdout:111669149730 2024-03-26T02:59:52.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T02:59:52.594 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:52.594 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:52.595 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:52.596 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:53.061 INFO:teuthology.orchestra.run.smithi066.stdout:171798691865 2024-03-26T02:59:53.062 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T02:59:53.084 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:53.086 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:53.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:52 smithi192 ceph-mon[33110]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:53.297 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:52 smithi066 ceph-mon[37920]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:53.297 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:52 smithi066 ceph-mon[29666]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:53.949 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 02:59:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:02:59:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:59:54.201 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:59:54.202 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:59:54.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:59:55.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:54 smithi066 ceph-mon[29666]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:55.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:54 smithi066 ceph-mon[37920]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:54 smithi192 ceph-mon[33110]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:59:57.044 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:57.153 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:56 smithi066 ceph-mon[29666]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:59:57.153 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:56 smithi066 ceph-mon[37920]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:59:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:56 smithi192 ceph-mon[33110]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:59:59.158 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[29666]: 2.d deep-scrub starts 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[29666]: 2.d deep-scrub ok 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[29666]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/338496425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[37920]: 2.d deep-scrub starts 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[37920]: 2.d deep-scrub ok 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[37920]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:59:59.159 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 02:59:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/338496425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:59:59.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:58 smithi192 ceph-mon[33110]: 2.d deep-scrub starts 2024-03-26T02:59:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:58 smithi192 ceph-mon[33110]: 2.d deep-scrub ok 2024-03-26T02:59:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:58 smithi192 ceph-mon[33110]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:59:59.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 02:59:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/338496425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:59:59.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 02:59:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T02:59:59.622 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:59:59.756 INFO:teuthology.orchestra.run.smithi066.stdout:206158430228 2024-03-26T03:00:00.051 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:00:00.051 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:00:00.051 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:00:00.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:00 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:00:00.326 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430226 got 206158430228 for osd.6 2024-03-26T03:00:00.326 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:00 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:00:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:00 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:00:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:00 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:00:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:00 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:00:01.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[29666]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[29666]: overall HEALTH_OK 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[37920]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[37920]: overall HEALTH_OK 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:00:01.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:01 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:00:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:01 smithi192 ceph-mon[33110]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:01 smithi192 ceph-mon[33110]: overall HEALTH_OK 2024-03-26T03:00:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:01 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:00:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:01 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:00:02.701 INFO:teuthology.orchestra.run.smithi066.stdout:55834574894 2024-03-26T03:00:03.246 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:03.246 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:00:03.246 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4110901499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: 2.1b scrub starts 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: 2.1b scrub ok 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4110901499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: 2.1b scrub starts 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: 2.1b scrub ok 2024-03-26T03:00:03.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:00:03.248 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:00:03.375 INFO:teuthology.orchestra.run.smithi066.stdout:34359738419 2024-03-26T03:00:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:00:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:00:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4110901499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:00:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: 2.1b scrub starts 2024-03-26T03:00:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: 2.1b scrub ok 2024-03-26T03:00:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:00:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:03 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:00:04.085 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:00:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:00:04.085 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1866702512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:00:04.085 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4098392347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:00:04.085 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:00:04.085 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:00:04.086 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1866702512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:00:04.086 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4098392347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:00:04.086 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:00:04.086 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:04 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:00:04.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574894 for osd.1 2024-03-26T03:00:04.111 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1866702512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:00:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4098392347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:00:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:04 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:00:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:04 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:00:04.526 INFO:teuthology.orchestra.run.smithi066.stdout:77309411369 2024-03-26T03:00:04.914 INFO:teuthology.orchestra.run.smithi066.stdout:249108103185 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:00:05.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2418245878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:00:05.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:05 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2418245878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:00:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:00:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:00:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:00:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:00:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:00:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:00:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:05 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2418245878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:00:06.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[29666]: 4.1c scrub starts 2024-03-26T03:00:06.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[29666]: 4.1c scrub ok 2024-03-26T03:00:06.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:00:06.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:00:06.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[37920]: 4.1c scrub starts 2024-03-26T03:00:06.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[37920]: 4.1c scrub ok 2024-03-26T03:00:06.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:00:06.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:06 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:00:06.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103182 got 249108103185 for osd.7 2024-03-26T03:00:06.351 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:06.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:06 smithi192 ceph-mon[33110]: 4.1c scrub starts 2024-03-26T03:00:06.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:06 smithi192 ceph-mon[33110]: 4.1c scrub ok 2024-03-26T03:00:06.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:06 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:00:06.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:06 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:00:06.997 INFO:teuthology.orchestra.run.smithi066.stdout:111669149733 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[29666]: 4.17 scrub starts 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[29666]: 4.17 scrub ok 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[29666]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2048951188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:00:07.232 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[37920]: 4.17 scrub starts 2024-03-26T03:00:07.233 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[37920]: 4.17 scrub ok 2024-03-26T03:00:07.233 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:00:07.233 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:00:07.233 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[37920]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:07.233 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2048951188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:00:07.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-03-26T03:00:07.257 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:07 smithi192 ceph-mon[33110]: 4.17 scrub starts 2024-03-26T03:00:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:07 smithi192 ceph-mon[33110]: 4.17 scrub ok 2024-03-26T03:00:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:07 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:00:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:07 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:00:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:07 smithi192 ceph-mon[33110]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2048951188' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:00:07.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149733 for osd.3 2024-03-26T03:00:07.888 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:08.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 4.1a scrub starts 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 4.1a scrub ok 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 5.17 scrub starts 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[29666]: 5.17 scrub ok 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 4.1a scrub starts 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 4.1a scrub ok 2024-03-26T03:00:08.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 5.17 scrub starts 2024-03-26T03:00:08.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:08 smithi066 ceph-mon[37920]: 5.17 scrub ok 2024-03-26T03:00:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 4.1a scrub starts 2024-03-26T03:00:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 4.1a scrub ok 2024-03-26T03:00:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:00:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:00:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:00:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:00:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 5.17 scrub starts 2024-03-26T03:00:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-mon[33110]: 5.17 scrub ok 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: 4.0 deep-scrub starts 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: 4.0 deep-scrub ok 2024-03-26T03:00:09.144 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[29666]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: 4.0 deep-scrub starts 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: 4.0 deep-scrub ok 2024-03-26T03:00:09.145 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:09 smithi066 ceph-mon[37920]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T03:00:09.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:00:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:00:09.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: 4.0 deep-scrub starts 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: 4.0 deep-scrub ok 2024-03-26T03:00:09.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:09 smithi192 ceph-mon[33110]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T03:00:09.573 INFO:teuthology.orchestra.run.smithi066.stdout:133143986208 2024-03-26T03:00:09.707 INFO:teuthology.orchestra.run.smithi066.stdout:171798691868 2024-03-26T03:00:10.049 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411369 for osd.2 2024-03-26T03:00:10.049 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:10.293 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:00:10.293 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:00:10.293 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: 4.c scrub starts 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: 4.c scrub ok 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4282557380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:00:10.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1753047012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: 4.c scrub starts 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: 4.c scrub ok 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:10.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4282557380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:00:10.296 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:10 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1753047012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:00:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:00:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:00:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: 4.c scrub starts 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: 4.c scrub ok 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4282557380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:00:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:10 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1753047012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:00:10.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986208 for osd.4 2024-03-26T03:00:10.665 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: 6.16 scrub starts 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: 6.16 scrub ok 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:00:11.315 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: 6.16 scrub starts 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: 6.16 scrub ok 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:00:11.316 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:11 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:00:11.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691865 got 171798691868 for osd.5 2024-03-26T03:00:11.343 DEBUG:teuthology.parallel:result is None 2024-03-26T03:00:11.343 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T03:00:11.344 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:00:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: 6.16 scrub starts 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: 6.16 scrub ok 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:00:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:11 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:00:11.607 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:12.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: 6.0 scrub starts 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: 6.0 scrub ok 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: 6.0 scrub starts 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: 6.0 scrub ok 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T03:00:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:00:12.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:00:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: 6.0 scrub starts 2024-03-26T03:00:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: 6.0 scrub ok 2024-03-26T03:00:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:00:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi066 to 3760M 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:13.040 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:00:13.041 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi066 to 3760M 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.042 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.043 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.043 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:00:13.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:00:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi066 to 3760M 2024-03-26T03:00:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi066:/etc/ceph/ceph.conf 2024-03-26T03:00:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-26T03:00:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:00:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:00:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:13.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:13.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi066:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:13.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: Updating smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:00:13.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:13.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:00:13.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:13 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:00:13.499 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:00:13.501 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:00:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:00:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:00:14.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: Deploying daemon alertmanager.a on smithi066 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: 5.1e deep-scrub starts 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: 5.1e deep-scrub ok 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: 4.4 deep-scrub starts 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: 4.4 deep-scrub ok 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[37920]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: Deploying daemon alertmanager.a on smithi066 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: 5.1e deep-scrub starts 2024-03-26T03:00:14.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: 5.1e deep-scrub ok 2024-03-26T03:00:14.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: 4.4 deep-scrub starts 2024-03-26T03:00:14.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: 4.4 deep-scrub ok 2024-03-26T03:00:14.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:14 smithi066 ceph-mon[29666]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T03:00:14.338 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-26T03:00:11.918386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1285,"num_read_kb":1099,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":805,"ondisk_log_size":805,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236252,"kb_used_data":7636,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724964,"statfs":{"total":767960285184,"available":767718363136,"internally_reserved":0,"allocated":7819264,"data_stored":4365436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001665"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913228+0000","last_change":"2024-03-26T02:59:03.208828+0000","last_active":"2024-03-26T03:00:08.913228+0000","last_peered":"2024-03-26T03:00:08.913228+0000","last_clean":"2024-03-26T03:00:08.913228+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:00:08.913228+0000","last_undegraded":"2024-03-26T03:00:08.913228+0000","last_fullsized":"2024-03-26T03:00:08.913228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.769121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.864763+0000","last_change":"2024-03-26T02:59:00.943722+0000","last_active":"2024-03-26T03:00:08.864763+0000","last_peered":"2024-03-26T03:00:08.864763+0000","last_clean":"2024-03-26T03:00:08.864763+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:00:08.864763+0000","last_undegraded":"2024-03-26T03:00:08.864763+0000","last_fullsized":"2024-03-26T03:00:08.864763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.322632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913842+0000","last_change":"2024-03-26T02:58:56.920384+0000","last_active":"2024-03-26T03:00:08.913842+0000","last_peered":"2024-03-26T03:00:08.913842+0000","last_clean":"2024-03-26T03:00:08.913842+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:00:08.913842+0000","last_undegraded":"2024-03-26T03:00:08.913842+0000","last_fullsized":"2024-03-26T03:00:08.913842+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.399933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":123,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912811+0000","last_change":"2024-03-26T02:58:58.929759+0000","last_active":"2024-03-26T03:00:08.912811+0000","last_peered":"2024-03-26T03:00:08.912811+0000","last_clean":"2024-03-26T03:00:08.912811+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:00:08.912811+0000","last_undegraded":"2024-03-26T03:00:08.912811+0000","last_fullsized":"2024-03-26T03:00:08.912811+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.022510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.864921+0000","last_change":"2024-03-26T02:59:02.945015+0000","last_active":"2024-03-26T03:00:08.864921+0000","last_peered":"2024-03-26T03:00:08.864921+0000","last_clean":"2024-03-26T03:00:08.864921+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:00:08.864921+0000","last_undegraded":"2024-03-26T03:00:08.864921+0000","last_fullsized":"2024-03-26T03:00:08.864921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.068015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868400+0000","last_change":"2024-03-26T02:59:00.932288+0000","last_active":"2024-03-26T03:00:08.868400+0000","last_peered":"2024-03-26T03:00:08.868400+0000","last_clean":"2024-03-26T03:00:08.868400+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:00:08.868400+0000","last_undegraded":"2024-03-26T03:00:08.868400+0000","last_fullsized":"2024-03-26T03:00:08.868400+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.996531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914046+0000","last_change":"2024-03-26T02:58:56.920528+0000","last_active":"2024-03-26T03:00:08.914046+0000","last_peered":"2024-03-26T03:00:08.914046+0000","last_clean":"2024-03-26T03:00:08.914046+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:00:08.914046+0000","last_undegraded":"2024-03-26T03:00:08.914046+0000","last_fullsized":"2024-03-26T03:00:08.914046+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.721957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865038+0000","last_change":"2024-03-26T02:58:58.927137+0000","last_active":"2024-03-26T03:00:08.865038+0000","last_peered":"2024-03-26T03:00:08.865038+0000","last_clean":"2024-03-26T03:00:08.865038+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:00:08.865038+0000","last_undegraded":"2024-03-26T03:00:08.865038+0000","last_fullsized":"2024-03-26T03:00:08.865038+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:11.385943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914383+0000","last_change":"2024-03-26T02:59:02.943093+0000","last_active":"2024-03-26T03:00:08.914383+0000","last_peered":"2024-03-26T03:00:08.914383+0000","last_clean":"2024-03-26T03:00:08.914383+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:00:08.914383+0000","last_undegraded":"2024-03-26T03:00:08.914383+0000","last_fullsized":"2024-03-26T03:00:08.914383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:11.603704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914142+0000","last_change":"2024-03-26T02:59:00.937245+0000","last_active":"2024-03-26T03:00:08.914142+0000","last_peered":"2024-03-26T03:00:08.914142+0000","last_clean":"2024-03-26T03:00:08.914142+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:00:08.914142+0000","last_undegraded":"2024-03-26T03:00:08.914142+0000","last_fullsized":"2024-03-26T03:00:08.914142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914690+0000","last_change":"2024-03-26T02:58:56.924320+0000","last_active":"2024-03-26T03:00:08.914690+0000","last_peered":"2024-03-26T03:00:08.914690+0000","last_clean":"2024-03-26T03:00:08.914690+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:00:08.914690+0000","last_undegraded":"2024-03-26T03:00:08.914690+0000","last_fullsized":"2024-03-26T03:00:08.914690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.644583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865350+0000","last_change":"2024-03-26T02:58:58.931435+0000","last_active":"2024-03-26T03:00:08.865350+0000","last_peered":"2024-03-26T03:00:08.865350+0000","last_clean":"2024-03-26T03:00:08.865350+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:00:08.865350+0000","last_undegraded":"2024-03-26T03:00:08.865350+0000","last_fullsized":"2024-03-26T03:00:08.865350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.301399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912620+0000","last_change":"2024-03-26T02:59:02.947270+0000","last_active":"2024-03-26T03:00:08.912620+0000","last_peered":"2024-03-26T03:00:08.912620+0000","last_clean":"2024-03-26T03:00:08.912620+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:00:08.912620+0000","last_undegraded":"2024-03-26T03:00:08.912620+0000","last_fullsized":"2024-03-26T03:00:08.912620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.504143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913844+0000","last_change":"2024-03-26T03:00:05.805098+0000","last_active":"2024-03-26T03:00:08.913844+0000","last_peered":"2024-03-26T03:00:08.913844+0000","last_clean":"2024-03-26T03:00:08.913844+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:00:08.913844+0000","last_undegraded":"2024-03-26T03:00:08.913844+0000","last_fullsized":"2024-03-26T03:00:08.913844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:05.804994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.804994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.354036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913809+0000","last_change":"2024-03-26T02:58:56.922748+0000","last_active":"2024-03-26T03:00:08.913809+0000","last_peered":"2024-03-26T03:00:08.913809+0000","last_clean":"2024-03-26T03:00:08.913809+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:00:08.913809+0000","last_undegraded":"2024-03-26T03:00:08.913809+0000","last_fullsized":"2024-03-26T03:00:08.913809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:09.938442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":96,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865238+0000","last_change":"2024-03-26T03:00:06.927566+0000","last_active":"2024-03-26T03:00:08.865238+0000","last_peered":"2024-03-26T03:00:08.865238+0000","last_clean":"2024-03-26T03:00:08.865238+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:00:08.865238+0000","last_undegraded":"2024-03-26T03:00:08.865238+0000","last_fullsized":"2024-03-26T03:00:08.865238+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:06.927439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:06.927439+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:07.212337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914063+0000","last_change":"2024-03-26T02:59:03.207222+0000","last_active":"2024-03-26T03:00:08.914063+0000","last_peered":"2024-03-26T03:00:08.914063+0000","last_clean":"2024-03-26T03:00:08.914063+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:00:08.914063+0000","last_undegraded":"2024-03-26T03:00:08.914063+0000","last_fullsized":"2024-03-26T03:00:08.914063+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.830165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865638+0000","last_change":"2024-03-26T02:59:00.943118+0000","last_active":"2024-03-26T03:00:08.865638+0000","last_peered":"2024-03-26T03:00:08.865638+0000","last_clean":"2024-03-26T03:00:08.865638+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:00:08.865638+0000","last_undegraded":"2024-03-26T03:00:08.865638+0000","last_fullsized":"2024-03-26T03:00:08.865638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:15.636738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913287+0000","last_change":"2024-03-26T03:00:02.741809+0000","last_active":"2024-03-26T03:00:08.913287+0000","last_peered":"2024-03-26T03:00:08.913287+0000","last_clean":"2024-03-26T03:00:08.913287+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:00:08.913287+0000","last_undegraded":"2024-03-26T03:00:08.913287+0000","last_fullsized":"2024-03-26T03:00:08.913287+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.741725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.741725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.926475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912603+0000","last_change":"2024-03-26T02:58:58.935294+0000","last_active":"2024-03-26T03:00:08.912603+0000","last_peered":"2024-03-26T03:00:08.912603+0000","last_clean":"2024-03-26T03:00:08.912603+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:00:08.912603+0000","last_undegraded":"2024-03-26T03:00:08.912603+0000","last_fullsized":"2024-03-26T03:00:08.912603+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:10.817677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866069+0000","last_change":"2024-03-26T02:59:03.206538+0000","last_active":"2024-03-26T03:00:08.866069+0000","last_peered":"2024-03-26T03:00:08.866069+0000","last_clean":"2024-03-26T03:00:08.866069+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:00:08.866069+0000","last_undegraded":"2024-03-26T03:00:08.866069+0000","last_fullsized":"2024-03-26T03:00:08.866069+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:10.570683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868686+0000","last_change":"2024-03-26T02:59:00.936674+0000","last_active":"2024-03-26T03:00:08.868686+0000","last_peered":"2024-03-26T03:00:08.868686+0000","last_clean":"2024-03-26T03:00:08.868686+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:00:08.868686+0000","last_undegraded":"2024-03-26T03:00:08.868686+0000","last_fullsized":"2024-03-26T03:00:08.868686+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914378+0000","last_change":"2024-03-26T02:58:56.922081+0000","last_active":"2024-03-26T03:00:08.914378+0000","last_peered":"2024-03-26T03:00:08.914378+0000","last_clean":"2024-03-26T03:00:08.914378+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:00:08.914378+0000","last_undegraded":"2024-03-26T03:00:08.914378+0000","last_fullsized":"2024-03-26T03:00:08.914378+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":82,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866372+0000","last_change":"2024-03-26T03:00:04.540480+0000","last_active":"2024-03-26T03:00:08.866372+0000","last_peered":"2024-03-26T03:00:08.866372+0000","last_clean":"2024-03-26T03:00:08.866372+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:00:08.866372+0000","last_undegraded":"2024-03-26T03:00:08.866372+0000","last_fullsized":"2024-03-26T03:00:08.866372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:04.540375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.540375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.104233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868143+0000","last_change":"2024-03-26T02:59:02.938768+0000","last_active":"2024-03-26T03:00:08.868143+0000","last_peered":"2024-03-26T03:00:08.868143+0000","last_clean":"2024-03-26T03:00:08.868143+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:00:08.868143+0000","last_undegraded":"2024-03-26T03:00:08.868143+0000","last_fullsized":"2024-03-26T03:00:08.868143+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.885569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914161+0000","last_change":"2024-03-26T02:59:00.936731+0000","last_active":"2024-03-26T03:00:08.914161+0000","last_peered":"2024-03-26T03:00:08.914161+0000","last_clean":"2024-03-26T03:00:08.914161+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:00:08.914161+0000","last_undegraded":"2024-03-26T03:00:08.914161+0000","last_fullsized":"2024-03-26T03:00:08.914161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912852+0000","last_change":"2024-03-26T02:58:56.922782+0000","last_active":"2024-03-26T03:00:08.912852+0000","last_peered":"2024-03-26T03:00:08.912852+0000","last_clean":"2024-03-26T03:00:08.912852+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:00:08.912852+0000","last_undegraded":"2024-03-26T03:00:08.912852+0000","last_fullsized":"2024-03-26T03:00:08.912852+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:22.137252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914555+0000","last_change":"2024-03-26T02:58:58.929170+0000","last_active":"2024-03-26T03:00:08.914555+0000","last_peered":"2024-03-26T03:00:08.914555+0000","last_clean":"2024-03-26T03:00:08.914555+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:00:08.914555+0000","last_undegraded":"2024-03-26T03:00:08.914555+0000","last_fullsized":"2024-03-26T03:00:08.914555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.666367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914046+0000","last_change":"2024-03-26T02:59:02.943645+0000","last_active":"2024-03-26T03:00:08.914046+0000","last_peered":"2024-03-26T03:00:08.914046+0000","last_clean":"2024-03-26T03:00:08.914046+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:00:08.914046+0000","last_undegraded":"2024-03-26T03:00:08.914046+0000","last_fullsized":"2024-03-26T03:00:08.914046+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914666+0000","last_change":"2024-03-26T02:59:00.939809+0000","last_active":"2024-03-26T03:00:08.914666+0000","last_peered":"2024-03-26T03:00:08.914666+0000","last_clean":"2024-03-26T03:00:08.914666+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:00:08.914666+0000","last_undegraded":"2024-03-26T03:00:08.914666+0000","last_fullsized":"2024-03-26T03:00:08.914666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:11.743016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913041+0000","last_change":"2024-03-26T02:58:56.924883+0000","last_active":"2024-03-26T03:00:08.913041+0000","last_peered":"2024-03-26T03:00:08.913041+0000","last_clean":"2024-03-26T03:00:08.913041+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:00:08.913041+0000","last_undegraded":"2024-03-26T03:00:08.913041+0000","last_fullsized":"2024-03-26T03:00:08.913041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914104+0000","last_change":"2024-03-26T02:58:58.945932+0000","last_active":"2024-03-26T03:00:08.914104+0000","last_peered":"2024-03-26T03:00:08.914104+0000","last_clean":"2024-03-26T03:00:08.914104+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:00:08.914104+0000","last_undegraded":"2024-03-26T03:00:08.914104+0000","last_fullsized":"2024-03-26T03:00:08.914104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.665509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913514+0000","last_change":"2024-03-26T02:59:03.207173+0000","last_active":"2024-03-26T03:00:08.913514+0000","last_peered":"2024-03-26T03:00:08.913514+0000","last_clean":"2024-03-26T03:00:08.913514+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:00:08.913514+0000","last_undegraded":"2024-03-26T03:00:08.913514+0000","last_fullsized":"2024-03-26T03:00:08.913514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.434157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913233+0000","last_change":"2024-03-26T03:00:04.453033+0000","last_active":"2024-03-26T03:00:08.913233+0000","last_peered":"2024-03-26T03:00:08.913233+0000","last_clean":"2024-03-26T03:00:08.913233+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:00:08.913233+0000","last_undegraded":"2024-03-26T03:00:08.913233+0000","last_fullsized":"2024-03-26T03:00:08.913233+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.452915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.452915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.497308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914922+0000","last_change":"2024-03-26T02:58:56.925003+0000","last_active":"2024-03-26T03:00:08.914922+0000","last_peered":"2024-03-26T03:00:08.914922+0000","last_clean":"2024-03-26T03:00:08.914922+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:00:08.914922+0000","last_undegraded":"2024-03-26T03:00:08.914922+0000","last_fullsized":"2024-03-26T03:00:08.914922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:15.012511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"71'17","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913181+0000","last_change":"2024-03-26T02:58:58.945243+0000","last_active":"2024-03-26T03:00:08.913181+0000","last_peered":"2024-03-26T03:00:08.913181+0000","last_clean":"2024-03-26T03:00:08.913181+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:00:08.913181+0000","last_undegraded":"2024-03-26T03:00:08.913181+0000","last_fullsized":"2024-03-26T03:00:08.913181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.347219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912863+0000","last_change":"2024-03-26T02:59:02.941127+0000","last_active":"2024-03-26T03:00:08.912863+0000","last_peered":"2024-03-26T03:00:08.912863+0000","last_clean":"2024-03-26T03:00:08.912863+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:00:08.912863+0000","last_undegraded":"2024-03-26T03:00:08.912863+0000","last_fullsized":"2024-03-26T03:00:08.912863+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912451+0000","last_change":"2024-03-26T02:59:00.940480+0000","last_active":"2024-03-26T03:00:08.912451+0000","last_peered":"2024-03-26T03:00:08.912451+0000","last_clean":"2024-03-26T03:00:08.912451+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:00:08.912451+0000","last_undegraded":"2024-03-26T03:00:08.912451+0000","last_fullsized":"2024-03-26T03:00:08.912451+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:10.383820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915456+0000","last_change":"2024-03-26T03:00:02.803641+0000","last_active":"2024-03-26T03:00:08.915456+0000","last_peered":"2024-03-26T03:00:08.915456+0000","last_clean":"2024-03-26T03:00:08.915456+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:00:08.915456+0000","last_undegraded":"2024-03-26T03:00:08.915456+0000","last_fullsized":"2024-03-26T03:00:08.915456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.803511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.803511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.142559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":55,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912595+0000","last_change":"2024-03-26T02:58:58.935611+0000","last_active":"2024-03-26T03:00:08.912595+0000","last_peered":"2024-03-26T03:00:08.912595+0000","last_clean":"2024-03-26T03:00:08.912595+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:00:08.912595+0000","last_undegraded":"2024-03-26T03:00:08.912595+0000","last_fullsized":"2024-03-26T03:00:08.912595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912660+0000","last_change":"2024-03-26T02:59:02.945112+0000","last_active":"2024-03-26T03:00:08.912660+0000","last_peered":"2024-03-26T03:00:08.912660+0000","last_clean":"2024-03-26T03:00:08.912660+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:00:08.912660+0000","last_undegraded":"2024-03-26T03:00:08.912660+0000","last_fullsized":"2024-03-26T03:00:08.912660+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.165290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867877+0000","last_change":"2024-03-26T02:59:00.936834+0000","last_active":"2024-03-26T03:00:08.867877+0000","last_peered":"2024-03-26T03:00:08.867877+0000","last_clean":"2024-03-26T03:00:08.867877+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:00:08.867877+0000","last_undegraded":"2024-03-26T03:00:08.867877+0000","last_fullsized":"2024-03-26T03:00:08.867877+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.104445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868479+0000","last_change":"2024-03-26T03:00:02.224750+0000","last_active":"2024-03-26T03:00:08.868479+0000","last_peered":"2024-03-26T03:00:08.868479+0000","last_clean":"2024-03-26T03:00:08.868479+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:00:08.868479+0000","last_undegraded":"2024-03-26T03:00:08.868479+0000","last_fullsized":"2024-03-26T03:00:08.868479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.224639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.224639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.778744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865491+0000","last_change":"2024-03-26T02:58:58.925892+0000","last_active":"2024-03-26T03:00:08.865491+0000","last_peered":"2024-03-26T03:00:08.865491+0000","last_clean":"2024-03-26T03:00:08.865491+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:00:08.865491+0000","last_undegraded":"2024-03-26T03:00:08.865491+0000","last_fullsized":"2024-03-26T03:00:08.865491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:22.434368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912224+0000","last_change":"2024-03-26T02:59:02.948086+0000","last_active":"2024-03-26T03:00:08.912224+0000","last_peered":"2024-03-26T03:00:08.912224+0000","last_clean":"2024-03-26T03:00:08.912224+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:00:08.912224+0000","last_undegraded":"2024-03-26T03:00:08.912224+0000","last_fullsized":"2024-03-26T03:00:08.912224+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:21.896674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913428+0000","last_change":"2024-03-26T02:59:00.933280+0000","last_active":"2024-03-26T03:00:08.913428+0000","last_peered":"2024-03-26T03:00:08.913428+0000","last_clean":"2024-03-26T03:00:08.913428+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:00:08.913428+0000","last_undegraded":"2024-03-26T03:00:08.913428+0000","last_fullsized":"2024-03-26T03:00:08.913428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914356+0000","last_change":"2024-03-26T03:00:07.455788+0000","last_active":"2024-03-26T03:00:08.914356+0000","last_peered":"2024-03-26T03:00:08.914356+0000","last_clean":"2024-03-26T03:00:08.914356+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:00:08.914356+0000","last_undegraded":"2024-03-26T03:00:08.914356+0000","last_fullsized":"2024-03-26T03:00:08.914356+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.455691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.455691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.746322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868303+0000","last_change":"2024-03-26T02:58:58.928475+0000","last_active":"2024-03-26T03:00:08.868303+0000","last_peered":"2024-03-26T03:00:08.868303+0000","last_clean":"2024-03-26T03:00:08.868303+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:00:08.868303+0000","last_undegraded":"2024-03-26T03:00:08.868303+0000","last_fullsized":"2024-03-26T03:00:08.868303+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.922564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866218+0000","last_change":"2024-03-26T02:59:02.944972+0000","last_active":"2024-03-26T03:00:08.866218+0000","last_peered":"2024-03-26T03:00:08.866218+0000","last_clean":"2024-03-26T03:00:08.866218+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:00:08.866218+0000","last_undegraded":"2024-03-26T03:00:08.866218+0000","last_fullsized":"2024-03-26T03:00:08.866218+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.212361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913026+0000","last_change":"2024-03-26T02:59:00.933699+0000","last_active":"2024-03-26T03:00:08.913026+0000","last_peered":"2024-03-26T03:00:08.913026+0000","last_clean":"2024-03-26T03:00:08.913026+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:00:08.913026+0000","last_undegraded":"2024-03-26T03:00:08.913026+0000","last_fullsized":"2024-03-26T03:00:08.913026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.593836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913674+0000","last_change":"2024-03-26T02:58:56.914857+0000","last_active":"2024-03-26T03:00:08.913674+0000","last_peered":"2024-03-26T03:00:08.913674+0000","last_clean":"2024-03-26T03:00:08.913674+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:00:08.913674+0000","last_undegraded":"2024-03-26T03:00:08.913674+0000","last_fullsized":"2024-03-26T03:00:08.913674+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.500241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915542+0000","last_change":"2024-03-26T02:58:58.945768+0000","last_active":"2024-03-26T03:00:08.915542+0000","last_peered":"2024-03-26T03:00:08.915542+0000","last_clean":"2024-03-26T03:00:08.915542+0000","last_became_active":"2024-03-26T02:58:58.945477+0000","last_became_peered":"2024-03-26T02:58:58.945477+0000","last_unstale":"2024-03-26T03:00:08.915542+0000","last_undegraded":"2024-03-26T03:00:08.915542+0000","last_fullsized":"2024-03-26T03:00:08.915542+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.280772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913716+0000","last_change":"2024-03-26T02:59:02.948593+0000","last_active":"2024-03-26T03:00:08.913716+0000","last_peered":"2024-03-26T03:00:08.913716+0000","last_clean":"2024-03-26T03:00:08.913716+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:00:08.913716+0000","last_undegraded":"2024-03-26T03:00:08.913716+0000","last_fullsized":"2024-03-26T03:00:08.913716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915417+0000","last_change":"2024-03-26T02:59:00.937123+0000","last_active":"2024-03-26T03:00:08.915417+0000","last_peered":"2024-03-26T03:00:08.915417+0000","last_clean":"2024-03-26T03:00:08.915417+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:00:08.915417+0000","last_undegraded":"2024-03-26T03:00:08.915417+0000","last_fullsized":"2024-03-26T03:00:08.915417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913728+0000","last_change":"2024-03-26T02:58:56.928271+0000","last_active":"2024-03-26T03:00:08.913728+0000","last_peered":"2024-03-26T03:00:08.913728+0000","last_clean":"2024-03-26T03:00:08.913728+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:00:08.913728+0000","last_undegraded":"2024-03-26T03:00:08.913728+0000","last_fullsized":"2024-03-26T03:00:08.913728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"71'22","reported_seq":88,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913241+0000","last_change":"2024-03-26T02:58:58.946522+0000","last_active":"2024-03-26T03:00:08.913241+0000","last_peered":"2024-03-26T03:00:08.913241+0000","last_clean":"2024-03-26T03:00:08.913241+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:00:08.913241+0000","last_undegraded":"2024-03-26T03:00:08.913241+0000","last_fullsized":"2024-03-26T03:00:08.913241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:11.393561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912678+0000","last_change":"2024-03-26T02:59:03.208607+0000","last_active":"2024-03-26T03:00:08.912678+0000","last_peered":"2024-03-26T03:00:08.912678+0000","last_clean":"2024-03-26T03:00:08.912678+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:00:08.912678+0000","last_undegraded":"2024-03-26T03:00:08.912678+0000","last_fullsized":"2024-03-26T03:00:08.912678+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.254105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915406+0000","last_change":"2024-03-26T02:59:00.930582+0000","last_active":"2024-03-26T03:00:08.915406+0000","last_peered":"2024-03-26T03:00:08.915406+0000","last_clean":"2024-03-26T03:00:08.915406+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:00:08.915406+0000","last_undegraded":"2024-03-26T03:00:08.915406+0000","last_fullsized":"2024-03-26T03:00:08.915406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912561+0000","last_change":"2024-03-26T03:00:04.420607+0000","last_active":"2024-03-26T03:00:08.912561+0000","last_peered":"2024-03-26T03:00:08.912561+0000","last_clean":"2024-03-26T03:00:08.912561+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:00:08.912561+0000","last_undegraded":"2024-03-26T03:00:08.912561+0000","last_fullsized":"2024-03-26T03:00:08.912561+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.420504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.420504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.414672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":94,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914270+0000","last_change":"2024-03-26T03:00:05.814094+0000","last_active":"2024-03-26T03:00:08.914270+0000","last_peered":"2024-03-26T03:00:08.914270+0000","last_clean":"2024-03-26T03:00:08.914270+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:00:08.914270+0000","last_undegraded":"2024-03-26T03:00:08.914270+0000","last_fullsized":"2024-03-26T03:00:08.914270+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:00:05.813958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.813958+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.910237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866072+0000","last_change":"2024-03-26T02:59:02.942788+0000","last_active":"2024-03-26T03:00:08.866072+0000","last_peered":"2024-03-26T03:00:08.866072+0000","last_clean":"2024-03-26T03:00:08.866072+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:00:08.866072+0000","last_undegraded":"2024-03-26T03:00:08.866072+0000","last_fullsized":"2024-03-26T03:00:08.866072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.084675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914166+0000","last_change":"2024-03-26T03:00:07.801621+0000","last_active":"2024-03-26T03:00:08.914166+0000","last_peered":"2024-03-26T03:00:08.914166+0000","last_clean":"2024-03-26T03:00:08.914166+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:00:08.914166+0000","last_undegraded":"2024-03-26T03:00:08.914166+0000","last_fullsized":"2024-03-26T03:00:08.914166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.801516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.801516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.176058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865129+0000","last_change":"2024-03-26T02:58:56.925054+0000","last_active":"2024-03-26T03:00:08.865129+0000","last_peered":"2024-03-26T03:00:08.865129+0000","last_clean":"2024-03-26T03:00:08.865129+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:00:08.865129+0000","last_undegraded":"2024-03-26T03:00:08.865129+0000","last_fullsized":"2024-03-26T03:00:08.865129+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.471451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":89,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913977+0000","last_change":"2024-03-26T02:58:58.946140+0000","last_active":"2024-03-26T03:00:08.913977+0000","last_peered":"2024-03-26T03:00:08.913977+0000","last_clean":"2024-03-26T03:00:08.913977+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:00:08.913977+0000","last_undegraded":"2024-03-26T03:00:08.913977+0000","last_fullsized":"2024-03-26T03:00:08.913977+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:21.167455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913496+0000","last_change":"2024-03-26T02:59:02.942511+0000","last_active":"2024-03-26T03:00:08.913496+0000","last_peered":"2024-03-26T03:00:08.913496+0000","last_clean":"2024-03-26T03:00:08.913496+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:00:08.913496+0000","last_undegraded":"2024-03-26T03:00:08.913496+0000","last_fullsized":"2024-03-26T03:00:08.913496+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.547464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866008+0000","last_change":"2024-03-26T02:58:56.919704+0000","last_active":"2024-03-26T03:00:08.866008+0000","last_peered":"2024-03-26T03:00:08.866008+0000","last_clean":"2024-03-26T03:00:08.866008+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:00:08.866008+0000","last_undegraded":"2024-03-26T03:00:08.866008+0000","last_fullsized":"2024-03-26T03:00:08.866008+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.046057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":117,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865207+0000","last_change":"2024-03-26T02:59:30.875750+0000","last_active":"2024-03-26T03:00:08.865207+0000","last_peered":"2024-03-26T03:00:08.865207+0000","last_clean":"2024-03-26T03:00:08.865207+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:00:08.865207+0000","last_undegraded":"2024-03-26T03:00:08.865207+0000","last_fullsized":"2024-03-26T03:00:08.865207+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:10.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865092+0000","last_change":"2024-03-26T03:00:08.469473+0000","last_active":"2024-03-26T03:00:08.865092+0000","last_peered":"2024-03-26T03:00:08.865092+0000","last_clean":"2024-03-26T03:00:08.865092+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:00:08.865092+0000","last_undegraded":"2024-03-26T03:00:08.865092+0000","last_fullsized":"2024-03-26T03:00:08.865092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:08.469321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.469321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.714692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912236+0000","last_change":"2024-03-26T02:59:02.948429+0000","last_active":"2024-03-26T03:00:08.912236+0000","last_peered":"2024-03-26T03:00:08.912236+0000","last_clean":"2024-03-26T03:00:08.912236+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:00:08.912236+0000","last_undegraded":"2024-03-26T03:00:08.912236+0000","last_fullsized":"2024-03-26T03:00:08.912236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868472+0000","last_change":"2024-03-26T02:58:56.929860+0000","last_active":"2024-03-26T03:00:08.868472+0000","last_peered":"2024-03-26T03:00:08.868472+0000","last_clean":"2024-03-26T03:00:08.868472+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:00:08.868472+0000","last_undegraded":"2024-03-26T03:00:08.868472+0000","last_fullsized":"2024-03-26T03:00:08.868472+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"71'16","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867651+0000","last_change":"2024-03-26T02:58:58.926863+0000","last_active":"2024-03-26T03:00:08.867651+0000","last_peered":"2024-03-26T03:00:08.867651+0000","last_clean":"2024-03-26T03:00:08.867651+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:00:08.867651+0000","last_undegraded":"2024-03-26T03:00:08.867651+0000","last_fullsized":"2024-03-26T03:00:08.867651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:10.907373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913855+0000","last_change":"2024-03-26T02:59:00.939537+0000","last_active":"2024-03-26T03:00:08.913855+0000","last_peered":"2024-03-26T03:00:08.913855+0000","last_clean":"2024-03-26T03:00:08.913855+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:00:08.913855+0000","last_undegraded":"2024-03-26T03:00:08.913855+0000","last_fullsized":"2024-03-26T03:00:08.913855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912544+0000","last_change":"2024-03-26T02:59:02.941781+0000","last_active":"2024-03-26T03:00:08.912544+0000","last_peered":"2024-03-26T03:00:08.912544+0000","last_clean":"2024-03-26T03:00:08.912544+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:00:08.912544+0000","last_undegraded":"2024-03-26T03:00:08.912544+0000","last_fullsized":"2024-03-26T03:00:08.912544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.196474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":52,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913875+0000","last_change":"2024-03-26T02:58:56.923680+0000","last_active":"2024-03-26T03:00:08.913875+0000","last_peered":"2024-03-26T03:00:08.913875+0000","last_clean":"2024-03-26T03:00:08.913875+0000","last_became_active":"2024-03-26T02:58:56.923317+0000","last_became_peered":"2024-03-26T02:58:56.923317+0000","last_unstale":"2024-03-26T03:00:08.913875+0000","last_undegraded":"2024-03-26T03:00:08.913875+0000","last_fullsized":"2024-03-26T03:00:08.913875+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.601353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"71'34","reported_seq":124,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868547+0000","last_change":"2024-03-26T02:58:58.931781+0000","last_active":"2024-03-26T03:00:08.868547+0000","last_peered":"2024-03-26T03:00:08.868547+0000","last_clean":"2024-03-26T03:00:08.868547+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:00:08.868547+0000","last_undegraded":"2024-03-26T03:00:08.868547+0000","last_fullsized":"2024-03-26T03:00:08.868547+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913453+0000","last_change":"2024-03-26T02:59:00.940319+0000","last_active":"2024-03-26T03:00:08.913453+0000","last_peered":"2024-03-26T03:00:08.913453+0000","last_clean":"2024-03-26T03:00:08.913453+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:00:08.913453+0000","last_undegraded":"2024-03-26T03:00:08.913453+0000","last_fullsized":"2024-03-26T03:00:08.913453+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.642394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914852+0000","last_change":"2024-03-26T02:59:03.208010+0000","last_active":"2024-03-26T03:00:08.914852+0000","last_peered":"2024-03-26T03:00:08.914852+0000","last_clean":"2024-03-26T03:00:08.914852+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:00:08.914852+0000","last_undegraded":"2024-03-26T03:00:08.914852+0000","last_fullsized":"2024-03-26T03:00:08.914852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:11.421092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":134,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867973+0000","last_change":"2024-03-26T03:00:04.289538+0000","last_active":"2024-03-26T03:00:08.867973+0000","last_peered":"2024-03-26T03:00:08.867973+0000","last_clean":"2024-03-26T03:00:08.867973+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:00:08.867973+0000","last_undegraded":"2024-03-26T03:00:08.867973+0000","last_fullsized":"2024-03-26T03:00:08.867973+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:00:04.289452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.289452+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.576447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":58,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865106+0000","last_change":"2024-03-26T02:58:56.927484+0000","last_active":"2024-03-26T03:00:08.865106+0000","last_peered":"2024-03-26T03:00:08.865106+0000","last_clean":"2024-03-26T03:00:08.865106+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:00:08.865106+0000","last_undegraded":"2024-03-26T03:00:08.865106+0000","last_fullsized":"2024-03-26T03:00:08.865106+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.094713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":88,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868028+0000","last_change":"2024-03-26T02:58:58.929597+0000","last_active":"2024-03-26T03:00:08.868028+0000","last_peered":"2024-03-26T03:00:08.868028+0000","last_clean":"2024-03-26T03:00:08.868028+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:00:08.868028+0000","last_undegraded":"2024-03-26T03:00:08.868028+0000","last_fullsized":"2024-03-26T03:00:08.868028+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865543+0000","last_change":"2024-03-26T02:59:00.935309+0000","last_active":"2024-03-26T03:00:08.865543+0000","last_peered":"2024-03-26T03:00:08.865543+0000","last_clean":"2024-03-26T03:00:08.865543+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:00:08.865543+0000","last_undegraded":"2024-03-26T03:00:08.865543+0000","last_fullsized":"2024-03-26T03:00:08.865543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:16.113547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915550+0000","last_change":"2024-03-26T02:59:02.942463+0000","last_active":"2024-03-26T03:00:08.915550+0000","last_peered":"2024-03-26T03:00:08.915550+0000","last_clean":"2024-03-26T03:00:08.915550+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:00:08.915550+0000","last_undegraded":"2024-03-26T03:00:08.915550+0000","last_fullsized":"2024-03-26T03:00:08.915550+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:09.555826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912854+0000","last_change":"2024-03-26T02:59:00.924687+0000","last_active":"2024-03-26T03:00:08.912854+0000","last_peered":"2024-03-26T03:00:08.912854+0000","last_clean":"2024-03-26T03:00:08.912854+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:00:08.912854+0000","last_undegraded":"2024-03-26T03:00:08.912854+0000","last_fullsized":"2024-03-26T03:00:08.912854+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.037055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913694+0000","last_change":"2024-03-26T03:00:03.775166+0000","last_active":"2024-03-26T03:00:08.913694+0000","last_peered":"2024-03-26T03:00:08.913694+0000","last_clean":"2024-03-26T03:00:08.913694+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:00:08.913694+0000","last_undegraded":"2024-03-26T03:00:08.913694+0000","last_fullsized":"2024-03-26T03:00:08.913694+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:03.775065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:03.775065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:04.410162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":300,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912874+0000","last_change":"2024-03-26T02:59:36.442802+0000","last_active":"2024-03-26T03:00:08.912874+0000","last_peered":"2024-03-26T03:00:08.912874+0000","last_clean":"2024-03-26T03:00:08.912874+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:00:08.912874+0000","last_undegraded":"2024-03-26T03:00:08.912874+0000","last_fullsized":"2024-03-26T03:00:08.912874+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-26T02:59:36.442703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T02:59:36.442703+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:44.641515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":86,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913619+0000","last_change":"2024-03-26T02:58:58.945716+0000","last_active":"2024-03-26T03:00:08.913619+0000","last_peered":"2024-03-26T03:00:08.913619+0000","last_clean":"2024-03-26T03:00:08.913619+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:00:08.913619+0000","last_undegraded":"2024-03-26T03:00:08.913619+0000","last_fullsized":"2024-03-26T03:00:08.913619+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:13.818786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913925+0000","last_change":"2024-03-26T02:59:00.935161+0000","last_active":"2024-03-26T03:00:08.913925+0000","last_peered":"2024-03-26T03:00:08.913925+0000","last_clean":"2024-03-26T03:00:08.913925+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:00:08.913925+0000","last_undegraded":"2024-03-26T03:00:08.913925+0000","last_fullsized":"2024-03-26T03:00:08.913925+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867709+0000","last_change":"2024-03-26T02:59:02.935205+0000","last_active":"2024-03-26T03:00:08.867709+0000","last_peered":"2024-03-26T03:00:08.867709+0000","last_clean":"2024-03-26T03:00:08.867709+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:00:08.867709+0000","last_undegraded":"2024-03-26T03:00:08.867709+0000","last_fullsized":"2024-03-26T03:00:08.867709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:08.962368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912918+0000","last_change":"2024-03-26T02:59:00.929356+0000","last_active":"2024-03-26T03:00:08.912918+0000","last_peered":"2024-03-26T03:00:08.912918+0000","last_clean":"2024-03-26T03:00:08.912918+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:00:08.912918+0000","last_undegraded":"2024-03-26T03:00:08.912918+0000","last_fullsized":"2024-03-26T03:00:08.912918+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.193414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912746+0000","last_change":"2024-03-26T02:58:56.926508+0000","last_active":"2024-03-26T03:00:08.912746+0000","last_peered":"2024-03-26T03:00:08.912746+0000","last_clean":"2024-03-26T03:00:08.912746+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:00:08.912746+0000","last_undegraded":"2024-03-26T03:00:08.912746+0000","last_fullsized":"2024-03-26T03:00:08.912746+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.271613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913909+0000","last_change":"2024-03-26T02:59:58.957331+0000","last_active":"2024-03-26T03:00:08.913909+0000","last_peered":"2024-03-26T03:00:08.913909+0000","last_clean":"2024-03-26T03:00:08.913909+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:00:08.913909+0000","last_undegraded":"2024-03-26T03:00:08.913909+0000","last_fullsized":"2024-03-26T03:00:08.913909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T02:59:58.957239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.957239+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:59.168727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915527+0000","last_change":"2024-03-26T02:59:00.935022+0000","last_active":"2024-03-26T03:00:08.915527+0000","last_peered":"2024-03-26T03:00:08.915527+0000","last_clean":"2024-03-26T03:00:08.915527+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:00:08.915527+0000","last_undegraded":"2024-03-26T03:00:08.915527+0000","last_fullsized":"2024-03-26T03:00:08.915527+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.032534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867573+0000","last_change":"2024-03-26T03:00:07.343102+0000","last_active":"2024-03-26T03:00:08.867573+0000","last_peered":"2024-03-26T03:00:08.867573+0000","last_clean":"2024-03-26T03:00:08.867573+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:00:08.867573+0000","last_undegraded":"2024-03-26T03:00:08.867573+0000","last_fullsized":"2024-03-26T03:00:08.867573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.343020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.343020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.674172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914896+0000","last_change":"2024-03-26T02:58:56.923089+0000","last_active":"2024-03-26T03:00:08.914896+0000","last_peered":"2024-03-26T03:00:08.914896+0000","last_clean":"2024-03-26T03:00:08.914896+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:00:08.914896+0000","last_undegraded":"2024-03-26T03:00:08.914896+0000","last_fullsized":"2024-03-26T03:00:08.914896+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.419614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":97,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913564+0000","last_change":"2024-03-26T02:58:58.945770+0000","last_active":"2024-03-26T03:00:08.913564+0000","last_peered":"2024-03-26T03:00:08.913564+0000","last_clean":"2024-03-26T03:00:08.913564+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:00:08.913564+0000","last_undegraded":"2024-03-26T03:00:08.913564+0000","last_fullsized":"2024-03-26T03:00:08.913564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:15.198398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914381+0000","last_change":"2024-03-26T02:59:00.940001+0000","last_active":"2024-03-26T03:00:08.914381+0000","last_peered":"2024-03-26T03:00:08.914381+0000","last_clean":"2024-03-26T03:00:08.914381+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:00:08.914381+0000","last_undegraded":"2024-03-26T03:00:08.914381+0000","last_fullsized":"2024-03-26T03:00:08.914381+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865553+0000","last_change":"2024-03-26T02:59:02.944368+0000","last_active":"2024-03-26T03:00:08.865553+0000","last_peered":"2024-03-26T03:00:08.865553+0000","last_clean":"2024-03-26T03:00:08.865553+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:00:08.865553+0000","last_undegraded":"2024-03-26T03:00:08.865553+0000","last_fullsized":"2024-03-26T03:00:08.865553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.999221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"64'1","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868361+0000","last_change":"2024-03-26T02:58:56.931335+0000","last_active":"2024-03-26T03:00:08.868361+0000","last_peered":"2024-03-26T03:00:08.868361+0000","last_clean":"2024-03-26T03:00:08.868361+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:00:08.868361+0000","last_undegraded":"2024-03-26T03:00:08.868361+0000","last_fullsized":"2024-03-26T03:00:08.868361+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":125,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912422+0000","last_change":"2024-03-26T03:00:08.772624+0000","last_active":"2024-03-26T03:00:08.912422+0000","last_peered":"2024-03-26T03:00:08.912422+0000","last_clean":"2024-03-26T03:00:08.912422+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:00:08.912422+0000","last_undegraded":"2024-03-26T03:00:08.912422+0000","last_fullsized":"2024-03-26T03:00:08.912422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.772486+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.103774+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865247+0000","last_change":"2024-03-26T02:59:00.944090+0000","last_active":"2024-03-26T03:00:08.865247+0000","last_peered":"2024-03-26T03:00:08.865247+0000","last_clean":"2024-03-26T03:00:08.865247+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:00:08.865247+0000","last_undegraded":"2024-03-26T03:00:08.865247+0000","last_fullsized":"2024-03-26T03:00:08.865247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.484558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913676+0000","last_change":"2024-03-26T02:59:03.208325+0000","last_active":"2024-03-26T03:00:08.913676+0000","last_peered":"2024-03-26T03:00:08.913676+0000","last_clean":"2024-03-26T03:00:08.913676+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:00:08.913676+0000","last_undegraded":"2024-03-26T03:00:08.913676+0000","last_fullsized":"2024-03-26T03:00:08.913676+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:16.189758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914769+0000","last_change":"2024-03-26T02:58:56.922909+0000","last_active":"2024-03-26T03:00:08.914769+0000","last_peered":"2024-03-26T03:00:08.914769+0000","last_clean":"2024-03-26T03:00:08.914769+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:00:08.914769+0000","last_undegraded":"2024-03-26T03:00:08.914769+0000","last_fullsized":"2024-03-26T03:00:08.914769+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.667025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":89,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.864944+0000","last_change":"2024-03-26T03:00:07.927770+0000","last_active":"2024-03-26T03:00:08.864944+0000","last_peered":"2024-03-26T03:00:08.864944+0000","last_clean":"2024-03-26T03:00:08.864944+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:00:08.864944+0000","last_undegraded":"2024-03-26T03:00:08.864944+0000","last_fullsized":"2024-03-26T03:00:08.864944+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:00:07.927648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.927648+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:12.893425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912299+0000","last_change":"2024-03-26T02:59:00.937344+0000","last_active":"2024-03-26T03:00:08.912299+0000","last_peered":"2024-03-26T03:00:08.912299+0000","last_clean":"2024-03-26T03:00:08.912299+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:00:08.912299+0000","last_undegraded":"2024-03-26T03:00:08.912299+0000","last_fullsized":"2024-03-26T03:00:08.912299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.676429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912791+0000","last_change":"2024-03-26T02:59:03.207469+0000","last_active":"2024-03-26T03:00:08.912791+0000","last_peered":"2024-03-26T03:00:08.912791+0000","last_clean":"2024-03-26T03:00:08.912791+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:00:08.912791+0000","last_undegraded":"2024-03-26T03:00:08.912791+0000","last_fullsized":"2024-03-26T03:00:08.912791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915342+0000","last_change":"2024-03-26T02:59:00.937750+0000","last_active":"2024-03-26T03:00:08.915342+0000","last_peered":"2024-03-26T03:00:08.915342+0000","last_clean":"2024-03-26T03:00:08.915342+0000","last_became_active":"2024-03-26T02:59:00.937450+0000","last_became_peered":"2024-03-26T02:59:00.937450+0000","last_unstale":"2024-03-26T03:00:08.915342+0000","last_undegraded":"2024-03-26T03:00:08.915342+0000","last_fullsized":"2024-03-26T03:00:08.915342+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.422213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912994+0000","last_change":"2024-03-26T02:58:56.926585+0000","last_active":"2024-03-26T03:00:08.912994+0000","last_peered":"2024-03-26T03:00:08.912994+0000","last_clean":"2024-03-26T03:00:08.912994+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:00:08.912994+0000","last_undegraded":"2024-03-26T03:00:08.912994+0000","last_fullsized":"2024-03-26T03:00:08.912994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866231+0000","last_change":"2024-03-26T03:00:05.879783+0000","last_active":"2024-03-26T03:00:08.866231+0000","last_peered":"2024-03-26T03:00:08.866231+0000","last_clean":"2024-03-26T03:00:08.866231+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:00:08.866231+0000","last_undegraded":"2024-03-26T03:00:08.866231+0000","last_fullsized":"2024-03-26T03:00:08.866231+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:05.879672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.879672+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.035707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.915210+0000","last_change":"2024-03-26T02:59:02.944645+0000","last_active":"2024-03-26T03:00:08.915210+0000","last_peered":"2024-03-26T03:00:08.915210+0000","last_clean":"2024-03-26T03:00:08.915210+0000","last_became_active":"2024-03-26T02:59:02.944279+0000","last_became_peered":"2024-03-26T02:59:02.944279+0000","last_unstale":"2024-03-26T03:00:08.915210+0000","last_undegraded":"2024-03-26T03:00:08.915210+0000","last_fullsized":"2024-03-26T03:00:08.915210+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.965129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865929+0000","last_change":"2024-03-26T02:59:00.934826+0000","last_active":"2024-03-26T03:00:08.865929+0000","last_peered":"2024-03-26T03:00:08.865929+0000","last_clean":"2024-03-26T03:00:08.865929+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:00:08.865929+0000","last_undegraded":"2024-03-26T03:00:08.865929+0000","last_fullsized":"2024-03-26T03:00:08.865929+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:10.681363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867704+0000","last_change":"2024-03-26T02:58:56.930599+0000","last_active":"2024-03-26T03:00:08.867704+0000","last_peered":"2024-03-26T03:00:08.867704+0000","last_clean":"2024-03-26T03:00:08.867704+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:00:08.867704+0000","last_undegraded":"2024-03-26T03:00:08.867704+0000","last_fullsized":"2024-03-26T03:00:08.867704+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.361297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868217+0000","last_change":"2024-03-26T02:58:58.931743+0000","last_active":"2024-03-26T03:00:08.868217+0000","last_peered":"2024-03-26T03:00:08.868217+0000","last_clean":"2024-03-26T03:00:08.868217+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:00:08.868217+0000","last_undegraded":"2024-03-26T03:00:08.868217+0000","last_fullsized":"2024-03-26T03:00:08.868217+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865811+0000","last_change":"2024-03-26T02:59:02.945294+0000","last_active":"2024-03-26T03:00:08.865811+0000","last_peered":"2024-03-26T03:00:08.865811+0000","last_clean":"2024-03-26T03:00:08.865811+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:00:08.865811+0000","last_undegraded":"2024-03-26T03:00:08.865811+0000","last_fullsized":"2024-03-26T03:00:08.865811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865487+0000","last_change":"2024-03-26T02:59:00.939210+0000","last_active":"2024-03-26T03:00:08.865487+0000","last_peered":"2024-03-26T03:00:08.865487+0000","last_clean":"2024-03-26T03:00:08.865487+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:00:08.865487+0000","last_undegraded":"2024-03-26T03:00:08.865487+0000","last_fullsized":"2024-03-26T03:00:08.865487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:13.723096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.867288+0000","last_change":"2024-03-26T02:58:56.930656+0000","last_active":"2024-03-26T03:00:08.867288+0000","last_peered":"2024-03-26T03:00:08.867288+0000","last_clean":"2024-03-26T03:00:08.867288+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:00:08.867288+0000","last_undegraded":"2024-03-26T03:00:08.867288+0000","last_fullsized":"2024-03-26T03:00:08.867288+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865475+0000","last_change":"2024-03-26T02:58:58.931224+0000","last_active":"2024-03-26T03:00:08.865475+0000","last_peered":"2024-03-26T03:00:08.865475+0000","last_clean":"2024-03-26T03:00:08.865475+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:00:08.865475+0000","last_undegraded":"2024-03-26T03:00:08.865475+0000","last_fullsized":"2024-03-26T03:00:08.865475+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865561+0000","last_change":"2024-03-26T02:59:02.942823+0000","last_active":"2024-03-26T03:00:08.865561+0000","last_peered":"2024-03-26T03:00:08.865561+0000","last_clean":"2024-03-26T03:00:08.865561+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:00:08.865561+0000","last_undegraded":"2024-03-26T03:00:08.865561+0000","last_fullsized":"2024-03-26T03:00:08.865561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.346480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868168+0000","last_change":"2024-03-26T02:59:00.937106+0000","last_active":"2024-03-26T03:00:08.868168+0000","last_peered":"2024-03-26T03:00:08.868168+0000","last_clean":"2024-03-26T03:00:08.868168+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:00:08.868168+0000","last_undegraded":"2024-03-26T03:00:08.868168+0000","last_fullsized":"2024-03-26T03:00:08.868168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:13.576767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913353+0000","last_change":"2024-03-26T03:00:00.526049+0000","last_active":"2024-03-26T03:00:08.913353+0000","last_peered":"2024-03-26T03:00:08.913353+0000","last_clean":"2024-03-26T03:00:08.913353+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:00:08.913353+0000","last_undegraded":"2024-03-26T03:00:08.913353+0000","last_fullsized":"2024-03-26T03:00:08.913353+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:00.525932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:00.525932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.138757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866175+0000","last_change":"2024-03-26T02:58:58.921800+0000","last_active":"2024-03-26T03:00:08.866175+0000","last_peered":"2024-03-26T03:00:08.866175+0000","last_clean":"2024-03-26T03:00:08.866175+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:00:08.866175+0000","last_undegraded":"2024-03-26T03:00:08.866175+0000","last_fullsized":"2024-03-26T03:00:08.866175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.470151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912682+0000","last_change":"2024-03-26T02:59:02.945728+0000","last_active":"2024-03-26T03:00:08.912682+0000","last_peered":"2024-03-26T03:00:08.912682+0000","last_clean":"2024-03-26T03:00:08.912682+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:00:08.912682+0000","last_undegraded":"2024-03-26T03:00:08.912682+0000","last_fullsized":"2024-03-26T03:00:08.912682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.349779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865423+0000","last_change":"2024-03-26T02:59:00.933188+0000","last_active":"2024-03-26T03:00:08.865423+0000","last_peered":"2024-03-26T03:00:08.865423+0000","last_clean":"2024-03-26T03:00:08.865423+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:00:08.865423+0000","last_undegraded":"2024-03-26T03:00:08.865423+0000","last_fullsized":"2024-03-26T03:00:08.865423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:22.441717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.866178+0000","last_change":"2024-03-26T02:59:58.637319+0000","last_active":"2024-03-26T03:00:08.866178+0000","last_peered":"2024-03-26T03:00:08.866178+0000","last_clean":"2024-03-26T03:00:08.866178+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:00:08.866178+0000","last_undegraded":"2024-03-26T03:00:08.866178+0000","last_fullsized":"2024-03-26T03:00:08.866178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:58.637207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.637207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.247095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":120,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914218+0000","last_change":"2024-03-26T02:59:30.881595+0000","last_active":"2024-03-26T03:00:08.914218+0000","last_peered":"2024-03-26T03:00:08.914218+0000","last_clean":"2024-03-26T03:00:08.914218+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:00:08.914218+0000","last_undegraded":"2024-03-26T03:00:08.914218+0000","last_fullsized":"2024-03-26T03:00:08.914218+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.707701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.912547+0000","last_change":"2024-03-26T02:59:02.948314+0000","last_active":"2024-03-26T03:00:08.912547+0000","last_peered":"2024-03-26T03:00:08.912547+0000","last_clean":"2024-03-26T03:00:08.912547+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:00:08.912547+0000","last_undegraded":"2024-03-26T03:00:08.912547+0000","last_fullsized":"2024-03-26T03:00:08.912547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.533166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865757+0000","last_change":"2024-03-26T02:59:00.941640+0000","last_active":"2024-03-26T03:00:08.865757+0000","last_peered":"2024-03-26T03:00:08.865757+0000","last_clean":"2024-03-26T03:00:08.865757+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:00:08.865757+0000","last_undegraded":"2024-03-26T03:00:08.865757+0000","last_fullsized":"2024-03-26T03:00:08.865757+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.227740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.868598+0000","last_change":"2024-03-26T02:59:57.244851+0000","last_active":"2024-03-26T03:00:08.868598+0000","last_peered":"2024-03-26T03:00:08.868598+0000","last_clean":"2024-03-26T03:00:08.868598+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:00:08.868598+0000","last_undegraded":"2024-03-26T03:00:08.868598+0000","last_fullsized":"2024-03-26T03:00:08.868598+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_clean_scrub_stamp":"2024-03-26T02:59:57.244756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.738002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"71'15","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913683+0000","last_change":"2024-03-26T02:58:58.935684+0000","last_active":"2024-03-26T03:00:08.913683+0000","last_peered":"2024-03-26T03:00:08.913683+0000","last_clean":"2024-03-26T03:00:08.913683+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:00:08.913683+0000","last_undegraded":"2024-03-26T03:00:08.913683+0000","last_fullsized":"2024-03-26T03:00:08.913683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:16.466363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913573+0000","last_change":"2024-03-26T02:59:03.206522+0000","last_active":"2024-03-26T03:00:08.913573+0000","last_peered":"2024-03-26T03:00:08.913573+0000","last_clean":"2024-03-26T03:00:08.913573+0000","last_became_active":"2024-03-26T02:59:03.206195+0000","last_became_peered":"2024-03-26T02:59:03.206195+0000","last_unstale":"2024-03-26T03:00:08.913573+0000","last_undegraded":"2024-03-26T03:00:08.913573+0000","last_fullsized":"2024-03-26T03:00:08.913573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.119564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.865781+0000","last_change":"2024-03-26T02:58:56.928192+0000","last_active":"2024-03-26T03:00:08.865781+0000","last_peered":"2024-03-26T03:00:08.865781+0000","last_clean":"2024-03-26T03:00:08.865781+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:00:08.865781+0000","last_undegraded":"2024-03-26T03:00:08.865781+0000","last_fullsized":"2024-03-26T03:00:08.865781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.248388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":97,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.913208+0000","last_change":"2024-03-26T02:58:58.945556+0000","last_active":"2024-03-26T03:00:08.913208+0000","last_peered":"2024-03-26T03:00:08.913208+0000","last_clean":"2024-03-26T03:00:08.913208+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:00:08.913208+0000","last_undegraded":"2024-03-26T03:00:08.913208+0000","last_fullsized":"2024-03-26T03:00:08.913208+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-26T03:00:08.914176+0000","last_change":"2024-03-26T02:59:00.938985+0000","last_active":"2024-03-26T03:00:08.914176+0000","last_peered":"2024-03-26T03:00:08.914176+0000","last_clean":"2024-03-26T03:00:08.914176+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:00:08.914176+0000","last_undegraded":"2024-03-26T03:00:08.914176+0000","last_fullsized":"2024-03-26T03:00:08.914176+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:13.254915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":598,"ondisk_log_size":598,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":58,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103187,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28324,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716828,"statfs":{"total":95995035648,"available":95966031872,"internally_reserved":0,"allocated":1339392,"data_stored":912918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430231,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1343488,"data_stored":918838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":323752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1347584,"data_stored":913571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":749568,"data_stored":329527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":765952,"data_stored":322361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":757760,"data_stored":322459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691869,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":741376,"data_stored":322010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:00:14.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:00:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: Updating smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:00:14.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: Deploying daemon alertmanager.a on smithi066 2024-03-26T03:00:14.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: 5.1e deep-scrub starts 2024-03-26T03:00:14.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: 5.1e deep-scrub ok 2024-03-26T03:00:14.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: 4.4 deep-scrub starts 2024-03-26T03:00:14.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: 4.4 deep-scrub ok 2024-03-26T03:00:14.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:14 smithi192 ceph-mon[33110]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T03:00:15.098 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:15.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[37920]: from='client.34622 v1:172.21.15.66:0/2030756708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[37920]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[29666]: from='client.34622 v1:172.21.15.66:0/2030756708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[29666]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:15.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:15 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:15 smithi192 ceph-mon[33110]: from='client.34622 v1:172.21.15.66:0/2030756708' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:15 smithi192 ceph-mon[33110]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-26T03:00:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:15 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:15 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:16.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[37920]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T03:00:16.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:00:16.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[37920]: 6.c deep-scrub starts 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[37920]: 6.c deep-scrub ok 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[29666]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[29666]: 6.c deep-scrub starts 2024-03-26T03:00:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:16 smithi066 ceph-mon[29666]: 6.c deep-scrub ok 2024-03-26T03:00:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:16 smithi192 ceph-mon[33110]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T03:00:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:16 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:00:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:16 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:00:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:16 smithi192 ceph-mon[33110]: 6.c deep-scrub starts 2024-03-26T03:00:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:16 smithi192 ceph-mon[33110]: 6.c deep-scrub ok 2024-03-26T03:00:17.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:17 smithi066 ceph-mon[29666]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:17.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:17 smithi066 ceph-mon[37920]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:17 smithi192 ceph-mon[33110]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:18.276 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[29666]: 2.6 deep-scrub starts 2024-03-26T03:00:18.276 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[29666]: 2.6 deep-scrub ok 2024-03-26T03:00:18.276 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[29666]: 5.13 scrub starts 2024-03-26T03:00:18.276 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[29666]: 5.13 scrub ok 2024-03-26T03:00:18.277 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[37920]: 2.6 deep-scrub starts 2024-03-26T03:00:18.277 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[37920]: 2.6 deep-scrub ok 2024-03-26T03:00:18.277 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[37920]: 5.13 scrub starts 2024-03-26T03:00:18.277 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:18 smithi066 ceph-mon[37920]: 5.13 scrub ok 2024-03-26T03:00:18.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:18 smithi192 ceph-mon[33110]: 2.6 deep-scrub starts 2024-03-26T03:00:18.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:18 smithi192 ceph-mon[33110]: 2.6 deep-scrub ok 2024-03-26T03:00:18.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:18 smithi192 ceph-mon[33110]: 5.13 scrub starts 2024-03-26T03:00:18.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:18 smithi192 ceph-mon[33110]: 5.13 scrub ok 2024-03-26T03:00:19.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-03-26T03:00:19.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[37920]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:00:19.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:00:19.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:19.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:19.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:00:19.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:00:19.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:19 smithi066 ceph-mon[29666]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T03:00:19.127 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:00:19.129 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:00:19.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:00:18 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:00:19.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:00:19.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:19 smithi192 ceph-mon[33110]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T03:00:20.327 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[29666]: 2.18 deep-scrub starts 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[29666]: 2.18 deep-scrub ok 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[29666]: from='client.34628 v1:172.21.15.66:0/2187223565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[37920]: 2.18 deep-scrub starts 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[37920]: 2.18 deep-scrub ok 2024-03-26T03:00:20.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:20 smithi066 ceph-mon[37920]: from='client.34628 v1:172.21.15.66:0/2187223565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:20 smithi192 ceph-mon[33110]: 2.18 deep-scrub starts 2024-03-26T03:00:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:20 smithi192 ceph-mon[33110]: 2.18 deep-scrub ok 2024-03-26T03:00:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:20 smithi192 ceph-mon[33110]: from='client.34628 v1:172.21.15.66:0/2187223565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:21.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:21.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:21.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[29666]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:00:21.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[29666]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T03:00:21.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:21.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:21.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[37920]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:00:21.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:21 smithi066 ceph-mon[37920]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T03:00:21.361 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:21 smithi066 systemd[1]: Starting Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:00:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:21 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:21 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:21 smithi192 ceph-mon[33110]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:00:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:21 smithi192 ceph-mon[33110]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T03:00:21.511 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-03-26T03:00:18.868128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1290,"num_read_kb":1104,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236540,"kb_used_data":7900,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724676,"statfs":{"total":767960285184,"available":767718068224,"internally_reserved":0,"allocated":8089600,"data_stored":4508872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.949742"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868687+0000","last_change":"2024-03-26T02:59:03.208828+0000","last_active":"2024-03-26T03:00:13.868687+0000","last_peered":"2024-03-26T03:00:13.868687+0000","last_clean":"2024-03-26T03:00:13.868687+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:00:13.868687+0000","last_undegraded":"2024-03-26T03:00:13.868687+0000","last_fullsized":"2024-03-26T03:00:13.868687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.769121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.866991+0000","last_change":"2024-03-26T02:59:00.943722+0000","last_active":"2024-03-26T03:00:13.866991+0000","last_peered":"2024-03-26T03:00:13.866991+0000","last_clean":"2024-03-26T03:00:13.866991+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:00:13.866991+0000","last_undegraded":"2024-03-26T03:00:13.866991+0000","last_fullsized":"2024-03-26T03:00:13.866991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.322632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912810+0000","last_change":"2024-03-26T02:58:56.920384+0000","last_active":"2024-03-26T03:00:13.912810+0000","last_peered":"2024-03-26T03:00:13.912810+0000","last_clean":"2024-03-26T03:00:13.912810+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:00:13.912810+0000","last_undegraded":"2024-03-26T03:00:13.912810+0000","last_fullsized":"2024-03-26T03:00:13.912810+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.399933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":127,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868689+0000","last_change":"2024-03-26T02:58:58.929759+0000","last_active":"2024-03-26T03:00:13.868689+0000","last_peered":"2024-03-26T03:00:13.868689+0000","last_clean":"2024-03-26T03:00:13.868689+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:00:13.868689+0000","last_undegraded":"2024-03-26T03:00:13.868689+0000","last_fullsized":"2024-03-26T03:00:13.868689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.022510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867167+0000","last_change":"2024-03-26T02:59:02.945015+0000","last_active":"2024-03-26T03:00:13.867167+0000","last_peered":"2024-03-26T03:00:13.867167+0000","last_clean":"2024-03-26T03:00:13.867167+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:00:13.867167+0000","last_undegraded":"2024-03-26T03:00:13.867167+0000","last_fullsized":"2024-03-26T03:00:13.867167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.068015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868837+0000","last_change":"2024-03-26T02:59:00.932288+0000","last_active":"2024-03-26T03:00:13.868837+0000","last_peered":"2024-03-26T03:00:13.868837+0000","last_clean":"2024-03-26T03:00:13.868837+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:00:13.868837+0000","last_undegraded":"2024-03-26T03:00:13.868837+0000","last_fullsized":"2024-03-26T03:00:13.868837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.996531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869817+0000","last_change":"2024-03-26T02:58:56.920528+0000","last_active":"2024-03-26T03:00:13.869817+0000","last_peered":"2024-03-26T03:00:13.869817+0000","last_clean":"2024-03-26T03:00:13.869817+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:00:13.869817+0000","last_undegraded":"2024-03-26T03:00:13.869817+0000","last_fullsized":"2024-03-26T03:00:13.869817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.721957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867246+0000","last_change":"2024-03-26T02:58:58.927137+0000","last_active":"2024-03-26T03:00:13.867246+0000","last_peered":"2024-03-26T03:00:13.867246+0000","last_clean":"2024-03-26T03:00:13.867246+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:00:13.867246+0000","last_undegraded":"2024-03-26T03:00:13.867246+0000","last_fullsized":"2024-03-26T03:00:13.867246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912384+0000","last_change":"2024-03-26T02:59:02.943093+0000","last_active":"2024-03-26T03:00:13.912384+0000","last_peered":"2024-03-26T03:00:13.912384+0000","last_clean":"2024-03-26T03:00:13.912384+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:00:13.912384+0000","last_undegraded":"2024-03-26T03:00:13.912384+0000","last_fullsized":"2024-03-26T03:00:13.912384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913279+0000","last_change":"2024-03-26T02:59:00.937245+0000","last_active":"2024-03-26T03:00:13.913279+0000","last_peered":"2024-03-26T03:00:13.913279+0000","last_clean":"2024-03-26T03:00:13.913279+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:00:13.913279+0000","last_undegraded":"2024-03-26T03:00:13.913279+0000","last_fullsized":"2024-03-26T03:00:13.913279+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914342+0000","last_change":"2024-03-26T02:58:56.924320+0000","last_active":"2024-03-26T03:00:13.914342+0000","last_peered":"2024-03-26T03:00:13.914342+0000","last_clean":"2024-03-26T03:00:13.914342+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:00:13.914342+0000","last_undegraded":"2024-03-26T03:00:13.914342+0000","last_fullsized":"2024-03-26T03:00:13.914342+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.644583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867446+0000","last_change":"2024-03-26T02:58:58.931435+0000","last_active":"2024-03-26T03:00:13.867446+0000","last_peered":"2024-03-26T03:00:13.867446+0000","last_clean":"2024-03-26T03:00:13.867446+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:00:13.867446+0000","last_undegraded":"2024-03-26T03:00:13.867446+0000","last_fullsized":"2024-03-26T03:00:13.867446+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.301399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912410+0000","last_change":"2024-03-26T02:59:02.947270+0000","last_active":"2024-03-26T03:00:13.912410+0000","last_peered":"2024-03-26T03:00:13.912410+0000","last_clean":"2024-03-26T03:00:13.912410+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:00:13.912410+0000","last_undegraded":"2024-03-26T03:00:13.912410+0000","last_fullsized":"2024-03-26T03:00:13.912410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.504143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912986+0000","last_change":"2024-03-26T03:00:05.805098+0000","last_active":"2024-03-26T03:00:13.912986+0000","last_peered":"2024-03-26T03:00:13.912986+0000","last_clean":"2024-03-26T03:00:13.912986+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:00:13.912986+0000","last_undegraded":"2024-03-26T03:00:13.912986+0000","last_fullsized":"2024-03-26T03:00:13.912986+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:05.804994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.804994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.354036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914334+0000","last_change":"2024-03-26T02:58:56.922748+0000","last_active":"2024-03-26T03:00:13.914334+0000","last_peered":"2024-03-26T03:00:13.914334+0000","last_clean":"2024-03-26T03:00:13.914334+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:00:13.914334+0000","last_undegraded":"2024-03-26T03:00:13.914334+0000","last_fullsized":"2024-03-26T03:00:13.914334+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867266+0000","last_change":"2024-03-26T03:00:06.927566+0000","last_active":"2024-03-26T03:00:13.867266+0000","last_peered":"2024-03-26T03:00:13.867266+0000","last_clean":"2024-03-26T03:00:13.867266+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:00:13.867266+0000","last_undegraded":"2024-03-26T03:00:13.867266+0000","last_fullsized":"2024-03-26T03:00:13.867266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:06.927439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:06.927439+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:07.212337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869905+0000","last_change":"2024-03-26T02:59:03.207222+0000","last_active":"2024-03-26T03:00:13.869905+0000","last_peered":"2024-03-26T03:00:13.869905+0000","last_clean":"2024-03-26T03:00:13.869905+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:00:13.869905+0000","last_undegraded":"2024-03-26T03:00:13.869905+0000","last_fullsized":"2024-03-26T03:00:13.869905+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.830165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867675+0000","last_change":"2024-03-26T02:59:00.943118+0000","last_active":"2024-03-26T03:00:13.867675+0000","last_peered":"2024-03-26T03:00:13.867675+0000","last_clean":"2024-03-26T03:00:13.867675+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:00:13.867675+0000","last_undegraded":"2024-03-26T03:00:13.867675+0000","last_fullsized":"2024-03-26T03:00:13.867675+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:15.636738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868969+0000","last_change":"2024-03-26T03:00:02.741809+0000","last_active":"2024-03-26T03:00:13.868969+0000","last_peered":"2024-03-26T03:00:13.868969+0000","last_clean":"2024-03-26T03:00:13.868969+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:00:13.868969+0000","last_undegraded":"2024-03-26T03:00:13.868969+0000","last_fullsized":"2024-03-26T03:00:13.868969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.741725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.741725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.926475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869268+0000","last_change":"2024-03-26T02:58:58.935294+0000","last_active":"2024-03-26T03:00:13.869268+0000","last_peered":"2024-03-26T03:00:13.869268+0000","last_clean":"2024-03-26T03:00:13.869268+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:00:13.869268+0000","last_undegraded":"2024-03-26T03:00:13.869268+0000","last_fullsized":"2024-03-26T03:00:13.869268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868083+0000","last_change":"2024-03-26T02:59:03.206538+0000","last_active":"2024-03-26T03:00:13.868083+0000","last_peered":"2024-03-26T03:00:13.868083+0000","last_clean":"2024-03-26T03:00:13.868083+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:00:13.868083+0000","last_undegraded":"2024-03-26T03:00:13.868083+0000","last_fullsized":"2024-03-26T03:00:13.868083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870936+0000","last_change":"2024-03-26T02:59:00.936674+0000","last_active":"2024-03-26T03:00:13.870936+0000","last_peered":"2024-03-26T03:00:13.870936+0000","last_clean":"2024-03-26T03:00:13.870936+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:00:13.870936+0000","last_undegraded":"2024-03-26T03:00:13.870936+0000","last_fullsized":"2024-03-26T03:00:13.870936+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913469+0000","last_change":"2024-03-26T02:58:56.922081+0000","last_active":"2024-03-26T03:00:13.913469+0000","last_peered":"2024-03-26T03:00:13.913469+0000","last_clean":"2024-03-26T03:00:13.913469+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:00:13.913469+0000","last_undegraded":"2024-03-26T03:00:13.913469+0000","last_fullsized":"2024-03-26T03:00:13.913469+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912400+0000","last_change":"2024-03-26T03:00:04.540480+0000","last_active":"2024-03-26T03:00:13.912400+0000","last_peered":"2024-03-26T03:00:13.912400+0000","last_clean":"2024-03-26T03:00:13.912400+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:00:13.912400+0000","last_undegraded":"2024-03-26T03:00:13.912400+0000","last_fullsized":"2024-03-26T03:00:13.912400+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:04.540375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.540375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.104233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870591+0000","last_change":"2024-03-26T02:59:02.938768+0000","last_active":"2024-03-26T03:00:13.870591+0000","last_peered":"2024-03-26T03:00:13.870591+0000","last_clean":"2024-03-26T03:00:13.870591+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:00:13.870591+0000","last_undegraded":"2024-03-26T03:00:13.870591+0000","last_fullsized":"2024-03-26T03:00:13.870591+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.885569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913046+0000","last_change":"2024-03-26T03:00:13.118260+0000","last_active":"2024-03-26T03:00:13.913046+0000","last_peered":"2024-03-26T03:00:13.913046+0000","last_clean":"2024-03-26T03:00:13.913046+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:00:13.913046+0000","last_undegraded":"2024-03-26T03:00:13.913046+0000","last_fullsized":"2024-03-26T03:00:13.913046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.118140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.552791+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868851+0000","last_change":"2024-03-26T02:58:56.922782+0000","last_active":"2024-03-26T03:00:13.868851+0000","last_peered":"2024-03-26T03:00:13.868851+0000","last_clean":"2024-03-26T03:00:13.868851+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:00:13.868851+0000","last_undegraded":"2024-03-26T03:00:13.868851+0000","last_fullsized":"2024-03-26T03:00:13.868851+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:22.137252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913578+0000","last_change":"2024-03-26T02:58:58.929170+0000","last_active":"2024-03-26T03:00:13.913578+0000","last_peered":"2024-03-26T03:00:13.913578+0000","last_clean":"2024-03-26T03:00:13.913578+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:00:13.913578+0000","last_undegraded":"2024-03-26T03:00:13.913578+0000","last_fullsized":"2024-03-26T03:00:13.913578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.666367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":46,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914888+0000","last_change":"2024-03-26T02:59:02.943645+0000","last_active":"2024-03-26T03:00:13.914888+0000","last_peered":"2024-03-26T03:00:13.914888+0000","last_clean":"2024-03-26T03:00:13.914888+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:00:13.914888+0000","last_undegraded":"2024-03-26T03:00:13.914888+0000","last_fullsized":"2024-03-26T03:00:13.914888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913601+0000","last_change":"2024-03-26T02:59:00.939809+0000","last_active":"2024-03-26T03:00:13.913601+0000","last_peered":"2024-03-26T03:00:13.913601+0000","last_clean":"2024-03-26T03:00:13.913601+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:00:13.913601+0000","last_undegraded":"2024-03-26T03:00:13.913601+0000","last_fullsized":"2024-03-26T03:00:13.913601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912326+0000","last_change":"2024-03-26T02:58:56.924883+0000","last_active":"2024-03-26T03:00:13.912326+0000","last_peered":"2024-03-26T03:00:13.912326+0000","last_clean":"2024-03-26T03:00:13.912326+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:00:13.912326+0000","last_undegraded":"2024-03-26T03:00:13.912326+0000","last_fullsized":"2024-03-26T03:00:13.912326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912983+0000","last_change":"2024-03-26T02:58:58.945932+0000","last_active":"2024-03-26T03:00:13.912983+0000","last_peered":"2024-03-26T03:00:13.912983+0000","last_clean":"2024-03-26T03:00:13.912983+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:00:13.912983+0000","last_undegraded":"2024-03-26T03:00:13.912983+0000","last_fullsized":"2024-03-26T03:00:13.912983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.665509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869391+0000","last_change":"2024-03-26T02:59:03.207173+0000","last_active":"2024-03-26T03:00:13.869391+0000","last_peered":"2024-03-26T03:00:13.869391+0000","last_clean":"2024-03-26T03:00:13.869391+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:00:13.869391+0000","last_undegraded":"2024-03-26T03:00:13.869391+0000","last_fullsized":"2024-03-26T03:00:13.869391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.434157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912878+0000","last_change":"2024-03-26T03:00:04.453033+0000","last_active":"2024-03-26T03:00:13.912878+0000","last_peered":"2024-03-26T03:00:13.912878+0000","last_clean":"2024-03-26T03:00:13.912878+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:00:13.912878+0000","last_undegraded":"2024-03-26T03:00:13.912878+0000","last_fullsized":"2024-03-26T03:00:13.912878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.452915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.452915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.497308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913760+0000","last_change":"2024-03-26T02:58:56.925003+0000","last_active":"2024-03-26T03:00:13.913760+0000","last_peered":"2024-03-26T03:00:13.913760+0000","last_clean":"2024-03-26T03:00:13.913760+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:00:13.913760+0000","last_undegraded":"2024-03-26T03:00:13.913760+0000","last_fullsized":"2024-03-26T03:00:13.913760+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:15.012511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"71'17","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869220+0000","last_change":"2024-03-26T02:58:58.945243+0000","last_active":"2024-03-26T03:00:13.869220+0000","last_peered":"2024-03-26T03:00:13.869220+0000","last_clean":"2024-03-26T03:00:13.869220+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:00:13.869220+0000","last_undegraded":"2024-03-26T03:00:13.869220+0000","last_fullsized":"2024-03-26T03:00:13.869220+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.347219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913182+0000","last_change":"2024-03-26T02:59:02.941127+0000","last_active":"2024-03-26T03:00:13.913182+0000","last_peered":"2024-03-26T03:00:13.913182+0000","last_clean":"2024-03-26T03:00:13.913182+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:00:13.913182+0000","last_undegraded":"2024-03-26T03:00:13.913182+0000","last_fullsized":"2024-03-26T03:00:13.913182+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912719+0000","last_change":"2024-03-26T02:59:00.940480+0000","last_active":"2024-03-26T03:00:13.912719+0000","last_peered":"2024-03-26T03:00:13.912719+0000","last_clean":"2024-03-26T03:00:13.912719+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:00:13.912719+0000","last_undegraded":"2024-03-26T03:00:13.912719+0000","last_fullsized":"2024-03-26T03:00:13.912719+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912828+0000","last_change":"2024-03-26T03:00:02.803641+0000","last_active":"2024-03-26T03:00:13.912828+0000","last_peered":"2024-03-26T03:00:13.912828+0000","last_clean":"2024-03-26T03:00:13.912828+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:00:13.912828+0000","last_undegraded":"2024-03-26T03:00:13.912828+0000","last_fullsized":"2024-03-26T03:00:13.912828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.803511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.803511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.142559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868519+0000","last_change":"2024-03-26T02:58:58.935611+0000","last_active":"2024-03-26T03:00:13.868519+0000","last_peered":"2024-03-26T03:00:13.868519+0000","last_clean":"2024-03-26T03:00:13.868519+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:00:13.868519+0000","last_undegraded":"2024-03-26T03:00:13.868519+0000","last_fullsized":"2024-03-26T03:00:13.868519+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868578+0000","last_change":"2024-03-26T02:59:02.945112+0000","last_active":"2024-03-26T03:00:13.868578+0000","last_peered":"2024-03-26T03:00:13.868578+0000","last_clean":"2024-03-26T03:00:13.868578+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:00:13.868578+0000","last_undegraded":"2024-03-26T03:00:13.868578+0000","last_fullsized":"2024-03-26T03:00:13.868578+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.165290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869618+0000","last_change":"2024-03-26T02:59:00.936834+0000","last_active":"2024-03-26T03:00:13.869618+0000","last_peered":"2024-03-26T03:00:13.869618+0000","last_clean":"2024-03-26T03:00:13.869618+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:00:13.869618+0000","last_undegraded":"2024-03-26T03:00:13.869618+0000","last_fullsized":"2024-03-26T03:00:13.869618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.104445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870703+0000","last_change":"2024-03-26T03:00:02.224750+0000","last_active":"2024-03-26T03:00:13.870703+0000","last_peered":"2024-03-26T03:00:13.870703+0000","last_clean":"2024-03-26T03:00:13.870703+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:00:13.870703+0000","last_undegraded":"2024-03-26T03:00:13.870703+0000","last_fullsized":"2024-03-26T03:00:13.870703+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.224639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.224639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.778744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867646+0000","last_change":"2024-03-26T02:58:58.925892+0000","last_active":"2024-03-26T03:00:13.867646+0000","last_peered":"2024-03-26T03:00:13.867646+0000","last_clean":"2024-03-26T03:00:13.867646+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:00:13.867646+0000","last_undegraded":"2024-03-26T03:00:13.867646+0000","last_fullsized":"2024-03-26T03:00:13.867646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:22.434368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912076+0000","last_change":"2024-03-26T02:59:02.948086+0000","last_active":"2024-03-26T03:00:13.912076+0000","last_peered":"2024-03-26T03:00:13.912076+0000","last_clean":"2024-03-26T03:00:13.912076+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:00:13.912076+0000","last_undegraded":"2024-03-26T03:00:13.912076+0000","last_fullsized":"2024-03-26T03:00:13.912076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:21.896674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869211+0000","last_change":"2024-03-26T02:59:00.933280+0000","last_active":"2024-03-26T03:00:13.869211+0000","last_peered":"2024-03-26T03:00:13.869211+0000","last_clean":"2024-03-26T03:00:13.869211+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:00:13.869211+0000","last_undegraded":"2024-03-26T03:00:13.869211+0000","last_fullsized":"2024-03-26T03:00:13.869211+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912735+0000","last_change":"2024-03-26T03:00:07.455788+0000","last_active":"2024-03-26T03:00:13.912735+0000","last_peered":"2024-03-26T03:00:13.912735+0000","last_clean":"2024-03-26T03:00:13.912735+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:00:13.912735+0000","last_undegraded":"2024-03-26T03:00:13.912735+0000","last_fullsized":"2024-03-26T03:00:13.912735+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.455691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.455691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.746322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869390+0000","last_change":"2024-03-26T02:58:58.928475+0000","last_active":"2024-03-26T03:00:13.869390+0000","last_peered":"2024-03-26T03:00:13.869390+0000","last_clean":"2024-03-26T03:00:13.869390+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:00:13.869390+0000","last_undegraded":"2024-03-26T03:00:13.869390+0000","last_fullsized":"2024-03-26T03:00:13.869390+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.922564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868128+0000","last_change":"2024-03-26T02:59:02.944972+0000","last_active":"2024-03-26T03:00:13.868128+0000","last_peered":"2024-03-26T03:00:13.868128+0000","last_clean":"2024-03-26T03:00:13.868128+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:00:13.868128+0000","last_undegraded":"2024-03-26T03:00:13.868128+0000","last_fullsized":"2024-03-26T03:00:13.868128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.212361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869652+0000","last_change":"2024-03-26T02:59:00.933699+0000","last_active":"2024-03-26T03:00:13.869652+0000","last_peered":"2024-03-26T03:00:13.869652+0000","last_clean":"2024-03-26T03:00:13.869652+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:00:13.869652+0000","last_undegraded":"2024-03-26T03:00:13.869652+0000","last_fullsized":"2024-03-26T03:00:13.869652+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.593836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912084+0000","last_change":"2024-03-26T02:58:56.914857+0000","last_active":"2024-03-26T03:00:13.912084+0000","last_peered":"2024-03-26T03:00:13.912084+0000","last_clean":"2024-03-26T03:00:13.912084+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:00:13.912084+0000","last_undegraded":"2024-03-26T03:00:13.912084+0000","last_fullsized":"2024-03-26T03:00:13.912084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.500241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914305+0000","last_change":"2024-03-26T02:58:58.945768+0000","last_active":"2024-03-26T03:00:13.914305+0000","last_peered":"2024-03-26T03:00:13.914305+0000","last_clean":"2024-03-26T03:00:13.914305+0000","last_became_active":"2024-03-26T02:58:58.945477+0000","last_became_peered":"2024-03-26T02:58:58.945477+0000","last_unstale":"2024-03-26T03:00:13.914305+0000","last_undegraded":"2024-03-26T03:00:13.914305+0000","last_fullsized":"2024-03-26T03:00:13.914305+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.280772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912673+0000","last_change":"2024-03-26T03:00:09.986685+0000","last_active":"2024-03-26T03:00:13.912673+0000","last_peered":"2024-03-26T03:00:13.912673+0000","last_clean":"2024-03-26T03:00:13.912673+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:00:13.912673+0000","last_undegraded":"2024-03-26T03:00:13.912673+0000","last_fullsized":"2024-03-26T03:00:13.912673+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:09.986591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:09.986591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.856847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914173+0000","last_change":"2024-03-26T03:00:10.855859+0000","last_active":"2024-03-26T03:00:13.914173+0000","last_peered":"2024-03-26T03:00:13.914173+0000","last_clean":"2024-03-26T03:00:13.914173+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:00:13.914173+0000","last_undegraded":"2024-03-26T03:00:13.914173+0000","last_fullsized":"2024-03-26T03:00:13.914173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:10.855698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.855698+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.308595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913984+0000","last_change":"2024-03-26T03:00:12.868763+0000","last_active":"2024-03-26T03:00:13.913984+0000","last_peered":"2024-03-26T03:00:13.913984+0000","last_clean":"2024-03-26T03:00:13.913984+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:00:13.913984+0000","last_undegraded":"2024-03-26T03:00:13.913984+0000","last_fullsized":"2024-03-26T03:00:13.913984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:12.868624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:12.868624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.398619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"71'22","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869817+0000","last_change":"2024-03-26T02:58:58.946522+0000","last_active":"2024-03-26T03:00:13.869817+0000","last_peered":"2024-03-26T03:00:13.869817+0000","last_clean":"2024-03-26T03:00:13.869817+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:00:13.869817+0000","last_undegraded":"2024-03-26T03:00:13.869817+0000","last_fullsized":"2024-03-26T03:00:13.869817+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914329+0000","last_change":"2024-03-26T02:59:03.208607+0000","last_active":"2024-03-26T03:00:13.914329+0000","last_peered":"2024-03-26T03:00:13.914329+0000","last_clean":"2024-03-26T03:00:13.914329+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:00:13.914329+0000","last_undegraded":"2024-03-26T03:00:13.914329+0000","last_fullsized":"2024-03-26T03:00:13.914329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.254105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912773+0000","last_change":"2024-03-26T02:59:00.930582+0000","last_active":"2024-03-26T03:00:13.912773+0000","last_peered":"2024-03-26T03:00:13.912773+0000","last_clean":"2024-03-26T03:00:13.912773+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:00:13.912773+0000","last_undegraded":"2024-03-26T03:00:13.912773+0000","last_fullsized":"2024-03-26T03:00:13.912773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912817+0000","last_change":"2024-03-26T03:00:04.420607+0000","last_active":"2024-03-26T03:00:13.912817+0000","last_peered":"2024-03-26T03:00:13.912817+0000","last_clean":"2024-03-26T03:00:13.912817+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:00:13.912817+0000","last_undegraded":"2024-03-26T03:00:13.912817+0000","last_fullsized":"2024-03-26T03:00:13.912817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.420504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.420504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.414672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870129+0000","last_change":"2024-03-26T03:00:05.814094+0000","last_active":"2024-03-26T03:00:13.870129+0000","last_peered":"2024-03-26T03:00:13.870129+0000","last_clean":"2024-03-26T03:00:13.870129+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:00:13.870129+0000","last_undegraded":"2024-03-26T03:00:13.870129+0000","last_fullsized":"2024-03-26T03:00:13.870129+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:00:05.813958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.813958+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.910237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911878+0000","last_change":"2024-03-26T02:59:02.942788+0000","last_active":"2024-03-26T03:00:13.911878+0000","last_peered":"2024-03-26T03:00:13.911878+0000","last_clean":"2024-03-26T03:00:13.911878+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:00:13.911878+0000","last_undegraded":"2024-03-26T03:00:13.911878+0000","last_fullsized":"2024-03-26T03:00:13.911878+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.084675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869994+0000","last_change":"2024-03-26T03:00:07.801621+0000","last_active":"2024-03-26T03:00:13.869994+0000","last_peered":"2024-03-26T03:00:13.869994+0000","last_clean":"2024-03-26T03:00:13.869994+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:00:13.869994+0000","last_undegraded":"2024-03-26T03:00:13.869994+0000","last_fullsized":"2024-03-26T03:00:13.869994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.801516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.801516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.176058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":63,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911671+0000","last_change":"2024-03-26T02:58:56.925054+0000","last_active":"2024-03-26T03:00:13.911671+0000","last_peered":"2024-03-26T03:00:13.911671+0000","last_clean":"2024-03-26T03:00:13.911671+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:00:13.911671+0000","last_undegraded":"2024-03-26T03:00:13.911671+0000","last_fullsized":"2024-03-26T03:00:13.911671+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.471451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912909+0000","last_change":"2024-03-26T02:58:58.946140+0000","last_active":"2024-03-26T03:00:13.912909+0000","last_peered":"2024-03-26T03:00:13.912909+0000","last_clean":"2024-03-26T03:00:13.912909+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:00:13.912909+0000","last_undegraded":"2024-03-26T03:00:13.912909+0000","last_fullsized":"2024-03-26T03:00:13.912909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:21.167455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869305+0000","last_change":"2024-03-26T02:59:02.942511+0000","last_active":"2024-03-26T03:00:13.869305+0000","last_peered":"2024-03-26T03:00:13.869305+0000","last_clean":"2024-03-26T03:00:13.869305+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:00:13.869305+0000","last_undegraded":"2024-03-26T03:00:13.869305+0000","last_fullsized":"2024-03-26T03:00:13.869305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.547464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867951+0000","last_change":"2024-03-26T02:58:56.919704+0000","last_active":"2024-03-26T03:00:13.867951+0000","last_peered":"2024-03-26T03:00:13.867951+0000","last_clean":"2024-03-26T03:00:13.867951+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:00:13.867951+0000","last_undegraded":"2024-03-26T03:00:13.867951+0000","last_fullsized":"2024-03-26T03:00:13.867951+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.046057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":121,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867262+0000","last_change":"2024-03-26T02:59:30.875750+0000","last_active":"2024-03-26T03:00:13.867262+0000","last_peered":"2024-03-26T03:00:13.867262+0000","last_clean":"2024-03-26T03:00:13.867262+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:00:13.867262+0000","last_undegraded":"2024-03-26T03:00:13.867262+0000","last_fullsized":"2024-03-26T03:00:13.867262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911672+0000","last_change":"2024-03-26T03:00:08.469473+0000","last_active":"2024-03-26T03:00:13.911672+0000","last_peered":"2024-03-26T03:00:13.911672+0000","last_clean":"2024-03-26T03:00:13.911672+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:00:13.911672+0000","last_undegraded":"2024-03-26T03:00:13.911672+0000","last_fullsized":"2024-03-26T03:00:13.911672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:08.469321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.469321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.714692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912138+0000","last_change":"2024-03-26T03:00:11.023432+0000","last_active":"2024-03-26T03:00:13.912138+0000","last_peered":"2024-03-26T03:00:13.912138+0000","last_clean":"2024-03-26T03:00:13.912138+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:00:13.912138+0000","last_undegraded":"2024-03-26T03:00:13.912138+0000","last_fullsized":"2024-03-26T03:00:13.912138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:11.023351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:11.023351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.872070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870348+0000","last_change":"2024-03-26T02:58:56.929860+0000","last_active":"2024-03-26T03:00:13.870348+0000","last_peered":"2024-03-26T03:00:13.870348+0000","last_clean":"2024-03-26T03:00:13.870348+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:00:13.870348+0000","last_undegraded":"2024-03-26T03:00:13.870348+0000","last_fullsized":"2024-03-26T03:00:13.870348+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"71'16","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869520+0000","last_change":"2024-03-26T02:58:58.926863+0000","last_active":"2024-03-26T03:00:13.869520+0000","last_peered":"2024-03-26T03:00:13.869520+0000","last_clean":"2024-03-26T03:00:13.869520+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:00:13.869520+0000","last_undegraded":"2024-03-26T03:00:13.869520+0000","last_fullsized":"2024-03-26T03:00:13.869520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912342+0000","last_change":"2024-03-26T03:00:08.968963+0000","last_active":"2024-03-26T03:00:13.912342+0000","last_peered":"2024-03-26T03:00:13.912342+0000","last_clean":"2024-03-26T03:00:13.912342+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:00:13.912342+0000","last_undegraded":"2024-03-26T03:00:13.912342+0000","last_fullsized":"2024-03-26T03:00:13.912342+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:08.968848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.968848+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.313221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869605+0000","last_change":"2024-03-26T02:59:02.941781+0000","last_active":"2024-03-26T03:00:13.869605+0000","last_peered":"2024-03-26T03:00:13.869605+0000","last_clean":"2024-03-26T03:00:13.869605+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:00:13.869605+0000","last_undegraded":"2024-03-26T03:00:13.869605+0000","last_fullsized":"2024-03-26T03:00:13.869605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.196474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914125+0000","last_change":"2024-03-26T02:58:56.923680+0000","last_active":"2024-03-26T03:00:13.914125+0000","last_peered":"2024-03-26T03:00:13.914125+0000","last_clean":"2024-03-26T03:00:13.914125+0000","last_became_active":"2024-03-26T02:58:56.923317+0000","last_became_peered":"2024-03-26T02:58:56.923317+0000","last_unstale":"2024-03-26T03:00:13.914125+0000","last_undegraded":"2024-03-26T03:00:13.914125+0000","last_fullsized":"2024-03-26T03:00:13.914125+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.601353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":148,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870452+0000","last_change":"2024-03-26T03:00:13.375248+0000","last_active":"2024-03-26T03:00:13.870452+0000","last_peered":"2024-03-26T03:00:13.870452+0000","last_clean":"2024-03-26T03:00:13.870452+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:00:13.870452+0000","last_undegraded":"2024-03-26T03:00:13.870452+0000","last_fullsized":"2024-03-26T03:00:13.870452+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.375080+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.080768+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912569+0000","last_change":"2024-03-26T02:59:00.940319+0000","last_active":"2024-03-26T03:00:13.912569+0000","last_peered":"2024-03-26T03:00:13.912569+0000","last_clean":"2024-03-26T03:00:13.912569+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:00:13.912569+0000","last_undegraded":"2024-03-26T03:00:13.912569+0000","last_fullsized":"2024-03-26T03:00:13.912569+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.642394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914544+0000","last_change":"2024-03-26T02:59:03.208010+0000","last_active":"2024-03-26T03:00:13.914544+0000","last_peered":"2024-03-26T03:00:13.914544+0000","last_clean":"2024-03-26T03:00:13.914544+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:00:13.914544+0000","last_undegraded":"2024-03-26T03:00:13.914544+0000","last_fullsized":"2024-03-26T03:00:13.914544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":143,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868957+0000","last_change":"2024-03-26T03:00:04.289538+0000","last_active":"2024-03-26T03:00:13.868957+0000","last_peered":"2024-03-26T03:00:13.868957+0000","last_clean":"2024-03-26T03:00:13.868957+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:00:13.868957+0000","last_undegraded":"2024-03-26T03:00:13.868957+0000","last_fullsized":"2024-03-26T03:00:13.868957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:00:04.289452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.289452+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.576447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911654+0000","last_change":"2024-03-26T02:58:56.927484+0000","last_active":"2024-03-26T03:00:13.911654+0000","last_peered":"2024-03-26T03:00:13.911654+0000","last_clean":"2024-03-26T03:00:13.911654+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:00:13.911654+0000","last_undegraded":"2024-03-26T03:00:13.911654+0000","last_fullsized":"2024-03-26T03:00:13.911654+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.094713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870797+0000","last_change":"2024-03-26T02:58:58.929597+0000","last_active":"2024-03-26T03:00:13.870797+0000","last_peered":"2024-03-26T03:00:13.870797+0000","last_clean":"2024-03-26T03:00:13.870797+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:00:13.870797+0000","last_undegraded":"2024-03-26T03:00:13.870797+0000","last_fullsized":"2024-03-26T03:00:13.870797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912546+0000","last_change":"2024-03-26T02:59:00.935309+0000","last_active":"2024-03-26T03:00:13.912546+0000","last_peered":"2024-03-26T03:00:13.912546+0000","last_clean":"2024-03-26T03:00:13.912546+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:00:13.912546+0000","last_undegraded":"2024-03-26T03:00:13.912546+0000","last_fullsized":"2024-03-26T03:00:13.912546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:16.113547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914000+0000","last_change":"2024-03-26T02:59:02.942463+0000","last_active":"2024-03-26T03:00:13.914000+0000","last_peered":"2024-03-26T03:00:13.914000+0000","last_clean":"2024-03-26T03:00:13.914000+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:00:13.914000+0000","last_undegraded":"2024-03-26T03:00:13.914000+0000","last_fullsized":"2024-03-26T03:00:13.914000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868687+0000","last_change":"2024-03-26T02:59:00.924687+0000","last_active":"2024-03-26T03:00:13.868687+0000","last_peered":"2024-03-26T03:00:13.868687+0000","last_clean":"2024-03-26T03:00:13.868687+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:00:13.868687+0000","last_undegraded":"2024-03-26T03:00:13.868687+0000","last_fullsized":"2024-03-26T03:00:13.868687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.037055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912877+0000","last_change":"2024-03-26T03:00:03.775166+0000","last_active":"2024-03-26T03:00:13.912877+0000","last_peered":"2024-03-26T03:00:13.912877+0000","last_clean":"2024-03-26T03:00:13.912877+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:00:13.912877+0000","last_undegraded":"2024-03-26T03:00:13.912877+0000","last_fullsized":"2024-03-26T03:00:13.912877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:03.775065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:03.775065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:04.410162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":304,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912564+0000","last_change":"2024-03-26T02:59:36.442802+0000","last_active":"2024-03-26T03:00:13.912564+0000","last_peered":"2024-03-26T03:00:13.912564+0000","last_clean":"2024-03-26T03:00:13.912564+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:00:13.912564+0000","last_undegraded":"2024-03-26T03:00:13.912564+0000","last_fullsized":"2024-03-26T03:00:13.912564+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-26T02:59:36.442703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T02:59:36.442703+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:44.641515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913072+0000","last_change":"2024-03-26T02:58:58.945716+0000","last_active":"2024-03-26T03:00:13.913072+0000","last_peered":"2024-03-26T03:00:13.913072+0000","last_clean":"2024-03-26T03:00:13.913072+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:00:13.913072+0000","last_undegraded":"2024-03-26T03:00:13.913072+0000","last_fullsized":"2024-03-26T03:00:13.913072+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914661+0000","last_change":"2024-03-26T02:59:00.935161+0000","last_active":"2024-03-26T03:00:13.914661+0000","last_peered":"2024-03-26T03:00:13.914661+0000","last_clean":"2024-03-26T03:00:13.914661+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:00:13.914661+0000","last_undegraded":"2024-03-26T03:00:13.914661+0000","last_fullsized":"2024-03-26T03:00:13.914661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870019+0000","last_change":"2024-03-26T02:59:02.935205+0000","last_active":"2024-03-26T03:00:13.870019+0000","last_peered":"2024-03-26T03:00:13.870019+0000","last_clean":"2024-03-26T03:00:13.870019+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:00:13.870019+0000","last_undegraded":"2024-03-26T03:00:13.870019+0000","last_fullsized":"2024-03-26T03:00:13.870019+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912203+0000","last_change":"2024-03-26T02:59:00.929356+0000","last_active":"2024-03-26T03:00:13.912203+0000","last_peered":"2024-03-26T03:00:13.912203+0000","last_clean":"2024-03-26T03:00:13.912203+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:00:13.912203+0000","last_undegraded":"2024-03-26T03:00:13.912203+0000","last_fullsized":"2024-03-26T03:00:13.912203+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.193414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912466+0000","last_change":"2024-03-26T02:58:56.926508+0000","last_active":"2024-03-26T03:00:13.912466+0000","last_peered":"2024-03-26T03:00:13.912466+0000","last_clean":"2024-03-26T03:00:13.912466+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:00:13.912466+0000","last_undegraded":"2024-03-26T03:00:13.912466+0000","last_fullsized":"2024-03-26T03:00:13.912466+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:14.271613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912775+0000","last_change":"2024-03-26T02:59:58.957331+0000","last_active":"2024-03-26T03:00:13.912775+0000","last_peered":"2024-03-26T03:00:13.912775+0000","last_clean":"2024-03-26T03:00:13.912775+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:00:13.912775+0000","last_undegraded":"2024-03-26T03:00:13.912775+0000","last_fullsized":"2024-03-26T03:00:13.912775+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T02:59:58.957239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.957239+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:59.168727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913990+0000","last_change":"2024-03-26T02:59:00.935022+0000","last_active":"2024-03-26T03:00:13.913990+0000","last_peered":"2024-03-26T03:00:13.913990+0000","last_clean":"2024-03-26T03:00:13.913990+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:00:13.913990+0000","last_undegraded":"2024-03-26T03:00:13.913990+0000","last_fullsized":"2024-03-26T03:00:13.913990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.032534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868726+0000","last_change":"2024-03-26T03:00:07.343102+0000","last_active":"2024-03-26T03:00:13.868726+0000","last_peered":"2024-03-26T03:00:13.868726+0000","last_clean":"2024-03-26T03:00:13.868726+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:00:13.868726+0000","last_undegraded":"2024-03-26T03:00:13.868726+0000","last_fullsized":"2024-03-26T03:00:13.868726+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.343020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.343020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.674172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914603+0000","last_change":"2024-03-26T02:58:56.923089+0000","last_active":"2024-03-26T03:00:13.914603+0000","last_peered":"2024-03-26T03:00:13.914603+0000","last_clean":"2024-03-26T03:00:13.914603+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:00:13.914603+0000","last_undegraded":"2024-03-26T03:00:13.914603+0000","last_fullsized":"2024-03-26T03:00:13.914603+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.419614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911978+0000","last_change":"2024-03-26T02:58:58.945770+0000","last_active":"2024-03-26T03:00:13.911978+0000","last_peered":"2024-03-26T03:00:13.911978+0000","last_clean":"2024-03-26T03:00:13.911978+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:00:13.911978+0000","last_undegraded":"2024-03-26T03:00:13.911978+0000","last_fullsized":"2024-03-26T03:00:13.911978+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:15.198398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913597+0000","last_change":"2024-03-26T03:00:10.472741+0000","last_active":"2024-03-26T03:00:13.913597+0000","last_peered":"2024-03-26T03:00:13.913597+0000","last_clean":"2024-03-26T03:00:13.913597+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:00:13.913597+0000","last_undegraded":"2024-03-26T03:00:13.913597+0000","last_fullsized":"2024-03-26T03:00:13.913597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:10.472642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.472642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.985525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867526+0000","last_change":"2024-03-26T02:59:02.944368+0000","last_active":"2024-03-26T03:00:13.867526+0000","last_peered":"2024-03-26T03:00:13.867526+0000","last_clean":"2024-03-26T03:00:13.867526+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:00:13.867526+0000","last_undegraded":"2024-03-26T03:00:13.867526+0000","last_fullsized":"2024-03-26T03:00:13.867526+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.999221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"64'1","reported_seq":58,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870994+0000","last_change":"2024-03-26T02:58:56.931335+0000","last_active":"2024-03-26T03:00:13.870994+0000","last_peered":"2024-03-26T03:00:13.870994+0000","last_clean":"2024-03-26T03:00:13.870994+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:00:13.870994+0000","last_undegraded":"2024-03-26T03:00:13.870994+0000","last_fullsized":"2024-03-26T03:00:13.870994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":129,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868404+0000","last_change":"2024-03-26T03:00:08.772624+0000","last_active":"2024-03-26T03:00:13.868404+0000","last_peered":"2024-03-26T03:00:13.868404+0000","last_clean":"2024-03-26T03:00:13.868404+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:00:13.868404+0000","last_undegraded":"2024-03-26T03:00:13.868404+0000","last_fullsized":"2024-03-26T03:00:13.868404+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.772486+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.103774+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867352+0000","last_change":"2024-03-26T02:59:00.944090+0000","last_active":"2024-03-26T03:00:13.867352+0000","last_peered":"2024-03-26T03:00:13.867352+0000","last_clean":"2024-03-26T03:00:13.867352+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:00:13.867352+0000","last_undegraded":"2024-03-26T03:00:13.867352+0000","last_fullsized":"2024-03-26T03:00:13.867352+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.484558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914520+0000","last_change":"2024-03-26T02:59:03.208325+0000","last_active":"2024-03-26T03:00:13.914520+0000","last_peered":"2024-03-26T03:00:13.914520+0000","last_clean":"2024-03-26T03:00:13.914520+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:00:13.914520+0000","last_undegraded":"2024-03-26T03:00:13.914520+0000","last_fullsized":"2024-03-26T03:00:13.914520+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:16.189758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913717+0000","last_change":"2024-03-26T02:58:56.922909+0000","last_active":"2024-03-26T03:00:13.913717+0000","last_peered":"2024-03-26T03:00:13.913717+0000","last_clean":"2024-03-26T03:00:13.913717+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:00:13.913717+0000","last_undegraded":"2024-03-26T03:00:13.913717+0000","last_fullsized":"2024-03-26T03:00:13.913717+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:17.667025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867106+0000","last_change":"2024-03-26T03:00:07.927770+0000","last_active":"2024-03-26T03:00:13.867106+0000","last_peered":"2024-03-26T03:00:13.867106+0000","last_clean":"2024-03-26T03:00:13.867106+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:00:13.867106+0000","last_undegraded":"2024-03-26T03:00:13.867106+0000","last_fullsized":"2024-03-26T03:00:13.867106+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:00:07.927648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.927648+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:12.893425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868343+0000","last_change":"2024-03-26T02:59:00.937344+0000","last_active":"2024-03-26T03:00:13.868343+0000","last_peered":"2024-03-26T03:00:13.868343+0000","last_clean":"2024-03-26T03:00:13.868343+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:00:13.868343+0000","last_undegraded":"2024-03-26T03:00:13.868343+0000","last_fullsized":"2024-03-26T03:00:13.868343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:19.676429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869443+0000","last_change":"2024-03-26T02:59:03.207469+0000","last_active":"2024-03-26T03:00:13.869443+0000","last_peered":"2024-03-26T03:00:13.869443+0000","last_clean":"2024-03-26T03:00:13.869443+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:00:13.869443+0000","last_undegraded":"2024-03-26T03:00:13.869443+0000","last_fullsized":"2024-03-26T03:00:13.869443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913847+0000","last_change":"2024-03-26T02:59:00.937750+0000","last_active":"2024-03-26T03:00:13.913847+0000","last_peered":"2024-03-26T03:00:13.913847+0000","last_clean":"2024-03-26T03:00:13.913847+0000","last_became_active":"2024-03-26T02:59:00.937450+0000","last_became_peered":"2024-03-26T02:59:00.937450+0000","last_unstale":"2024-03-26T03:00:13.913847+0000","last_undegraded":"2024-03-26T03:00:13.913847+0000","last_fullsized":"2024-03-26T03:00:13.913847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.422213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912668+0000","last_change":"2024-03-26T02:58:56.926585+0000","last_active":"2024-03-26T03:00:13.912668+0000","last_peered":"2024-03-26T03:00:13.912668+0000","last_clean":"2024-03-26T03:00:13.912668+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:00:13.912668+0000","last_undegraded":"2024-03-26T03:00:13.912668+0000","last_fullsized":"2024-03-26T03:00:13.912668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868151+0000","last_change":"2024-03-26T03:00:05.879783+0000","last_active":"2024-03-26T03:00:13.868151+0000","last_peered":"2024-03-26T03:00:13.868151+0000","last_clean":"2024-03-26T03:00:13.868151+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:00:13.868151+0000","last_undegraded":"2024-03-26T03:00:13.868151+0000","last_fullsized":"2024-03-26T03:00:13.868151+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:05.879672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.879672+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.035707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914101+0000","last_change":"2024-03-26T02:59:02.944645+0000","last_active":"2024-03-26T03:00:13.914101+0000","last_peered":"2024-03-26T03:00:13.914101+0000","last_clean":"2024-03-26T03:00:13.914101+0000","last_became_active":"2024-03-26T02:59:02.944279+0000","last_became_peered":"2024-03-26T02:59:02.944279+0000","last_unstale":"2024-03-26T03:00:13.914101+0000","last_undegraded":"2024-03-26T03:00:13.914101+0000","last_fullsized":"2024-03-26T03:00:13.914101+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.965129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867892+0000","last_change":"2024-03-26T02:59:00.934826+0000","last_active":"2024-03-26T03:00:13.867892+0000","last_peered":"2024-03-26T03:00:13.867892+0000","last_clean":"2024-03-26T03:00:13.867892+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:00:13.867892+0000","last_undegraded":"2024-03-26T03:00:13.867892+0000","last_fullsized":"2024-03-26T03:00:13.867892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868944+0000","last_change":"2024-03-26T02:58:56.930599+0000","last_active":"2024-03-26T03:00:13.868944+0000","last_peered":"2024-03-26T03:00:13.868944+0000","last_clean":"2024-03-26T03:00:13.868944+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:00:13.868944+0000","last_undegraded":"2024-03-26T03:00:13.868944+0000","last_fullsized":"2024-03-26T03:00:13.868944+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.361297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870946+0000","last_change":"2024-03-26T02:58:58.931743+0000","last_active":"2024-03-26T03:00:13.870946+0000","last_peered":"2024-03-26T03:00:13.870946+0000","last_clean":"2024-03-26T03:00:13.870946+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:00:13.870946+0000","last_undegraded":"2024-03-26T03:00:13.870946+0000","last_fullsized":"2024-03-26T03:00:13.870946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867842+0000","last_change":"2024-03-26T02:59:02.945294+0000","last_active":"2024-03-26T03:00:13.867842+0000","last_peered":"2024-03-26T03:00:13.867842+0000","last_clean":"2024-03-26T03:00:13.867842+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:00:13.867842+0000","last_undegraded":"2024-03-26T03:00:13.867842+0000","last_fullsized":"2024-03-26T03:00:13.867842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"71'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912307+0000","last_change":"2024-03-26T02:59:00.939210+0000","last_active":"2024-03-26T03:00:13.912307+0000","last_peered":"2024-03-26T03:00:13.912307+0000","last_clean":"2024-03-26T03:00:13.912307+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:00:13.912307+0000","last_undegraded":"2024-03-26T03:00:13.912307+0000","last_fullsized":"2024-03-26T03:00:13.912307+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.869291+0000","last_change":"2024-03-26T02:58:56.930656+0000","last_active":"2024-03-26T03:00:13.869291+0000","last_peered":"2024-03-26T03:00:13.869291+0000","last_clean":"2024-03-26T03:00:13.869291+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:00:13.869291+0000","last_undegraded":"2024-03-26T03:00:13.869291+0000","last_fullsized":"2024-03-26T03:00:13.869291+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.805791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.867536+0000","last_change":"2024-03-26T03:00:08.975032+0000","last_active":"2024-03-26T03:00:13.867536+0000","last_peered":"2024-03-26T03:00:13.867536+0000","last_clean":"2024-03-26T03:00:13.867536+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:00:13.867536+0000","last_undegraded":"2024-03-26T03:00:13.867536+0000","last_fullsized":"2024-03-26T03:00:13.867536+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:00:08.974874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.974874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.309201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912604+0000","last_change":"2024-03-26T02:59:02.942823+0000","last_active":"2024-03-26T03:00:13.912604+0000","last_peered":"2024-03-26T03:00:13.912604+0000","last_clean":"2024-03-26T03:00:13.912604+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:00:13.912604+0000","last_undegraded":"2024-03-26T03:00:13.912604+0000","last_fullsized":"2024-03-26T03:00:13.912604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.346480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870115+0000","last_change":"2024-03-26T02:59:00.937106+0000","last_active":"2024-03-26T03:00:13.870115+0000","last_peered":"2024-03-26T03:00:13.870115+0000","last_clean":"2024-03-26T03:00:13.870115+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:00:13.870115+0000","last_undegraded":"2024-03-26T03:00:13.870115+0000","last_fullsized":"2024-03-26T03:00:13.870115+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.914154+0000","last_change":"2024-03-26T03:00:00.526049+0000","last_active":"2024-03-26T03:00:13.914154+0000","last_peered":"2024-03-26T03:00:13.914154+0000","last_clean":"2024-03-26T03:00:13.914154+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:00:13.914154+0000","last_undegraded":"2024-03-26T03:00:13.914154+0000","last_fullsized":"2024-03-26T03:00:13.914154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:00.525932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:00.525932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.138757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.868006+0000","last_change":"2024-03-26T02:58:58.921800+0000","last_active":"2024-03-26T03:00:13.868006+0000","last_peered":"2024-03-26T03:00:13.868006+0000","last_clean":"2024-03-26T03:00:13.868006+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:00:13.868006+0000","last_undegraded":"2024-03-26T03:00:13.868006+0000","last_fullsized":"2024-03-26T03:00:13.868006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:20.470151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912351+0000","last_change":"2024-03-26T02:59:02.945728+0000","last_active":"2024-03-26T03:00:13.912351+0000","last_peered":"2024-03-26T03:00:13.912351+0000","last_clean":"2024-03-26T03:00:13.912351+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:00:13.912351+0000","last_undegraded":"2024-03-26T03:00:13.912351+0000","last_fullsized":"2024-03-26T03:00:13.912351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.349779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911905+0000","last_change":"2024-03-26T02:59:00.933188+0000","last_active":"2024-03-26T03:00:13.911905+0000","last_peered":"2024-03-26T03:00:13.911905+0000","last_clean":"2024-03-26T03:00:13.911905+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:00:13.911905+0000","last_undegraded":"2024-03-26T03:00:13.911905+0000","last_fullsized":"2024-03-26T03:00:13.911905+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:22.441717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912180+0000","last_change":"2024-03-26T02:59:58.637319+0000","last_active":"2024-03-26T03:00:13.912180+0000","last_peered":"2024-03-26T03:00:13.912180+0000","last_clean":"2024-03-26T03:00:13.912180+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:00:13.912180+0000","last_undegraded":"2024-03-26T03:00:13.912180+0000","last_fullsized":"2024-03-26T03:00:13.912180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:58.637207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.637207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.247095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":124,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913374+0000","last_change":"2024-03-26T02:59:30.881595+0000","last_active":"2024-03-26T03:00:13.913374+0000","last_peered":"2024-03-26T03:00:13.913374+0000","last_clean":"2024-03-26T03:00:13.913374+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:00:13.913374+0000","last_undegraded":"2024-03-26T03:00:13.913374+0000","last_fullsized":"2024-03-26T03:00:13.913374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:18.707701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.911823+0000","last_change":"2024-03-26T02:59:02.948314+0000","last_active":"2024-03-26T03:00:13.911823+0000","last_peered":"2024-03-26T03:00:13.911823+0000","last_clean":"2024-03-26T03:00:13.911823+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:00:13.911823+0000","last_undegraded":"2024-03-26T03:00:13.911823+0000","last_fullsized":"2024-03-26T03:00:13.911823+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.533166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912004+0000","last_change":"2024-03-26T02:59:00.941640+0000","last_active":"2024-03-26T03:00:13.912004+0000","last_peered":"2024-03-26T03:00:13.912004+0000","last_clean":"2024-03-26T03:00:13.912004+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:00:13.912004+0000","last_undegraded":"2024-03-26T03:00:13.912004+0000","last_fullsized":"2024-03-26T03:00:13.912004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.227740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.870797+0000","last_change":"2024-03-26T02:59:57.244851+0000","last_active":"2024-03-26T03:00:13.870797+0000","last_peered":"2024-03-26T03:00:13.870797+0000","last_clean":"2024-03-26T03:00:13.870797+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:00:13.870797+0000","last_undegraded":"2024-03-26T03:00:13.870797+0000","last_fullsized":"2024-03-26T03:00:13.870797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_clean_scrub_stamp":"2024-03-26T02:59:57.244756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.738002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"71'15","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912167+0000","last_change":"2024-03-26T02:58:58.935684+0000","last_active":"2024-03-26T03:00:13.912167+0000","last_peered":"2024-03-26T03:00:13.912167+0000","last_clean":"2024-03-26T03:00:13.912167+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:00:13.912167+0000","last_undegraded":"2024-03-26T03:00:13.912167+0000","last_fullsized":"2024-03-26T03:00:13.912167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:16.466363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913850+0000","last_change":"2024-03-26T02:59:03.206522+0000","last_active":"2024-03-26T03:00:13.913850+0000","last_peered":"2024-03-26T03:00:13.913850+0000","last_clean":"2024-03-26T03:00:13.913850+0000","last_became_active":"2024-03-26T02:59:03.206195+0000","last_became_peered":"2024-03-26T02:59:03.206195+0000","last_unstale":"2024-03-26T03:00:13.913850+0000","last_undegraded":"2024-03-26T03:00:13.913850+0000","last_fullsized":"2024-03-26T03:00:13.913850+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.119564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":57,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912702+0000","last_change":"2024-03-26T02:58:56.928192+0000","last_active":"2024-03-26T03:00:13.912702+0000","last_peered":"2024-03-26T03:00:13.912702+0000","last_clean":"2024-03-26T03:00:13.912702+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:00:13.912702+0000","last_undegraded":"2024-03-26T03:00:13.912702+0000","last_fullsized":"2024-03-26T03:00:13.912702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:23.248388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.912453+0000","last_change":"2024-03-26T02:58:58.945556+0000","last_active":"2024-03-26T03:00:13.912453+0000","last_peered":"2024-03-26T03:00:13.912453+0000","last_clean":"2024-03-26T03:00:13.912453+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:00:13.912453+0000","last_undegraded":"2024-03-26T03:00:13.912453+0000","last_fullsized":"2024-03-26T03:00:13.912453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:00:13.913185+0000","last_change":"2024-03-26T02:59:00.938985+0000","last_active":"2024-03-26T03:00:13.913185+0000","last_peered":"2024-03-26T03:00:13.913185+0000","last_clean":"2024-03-26T03:00:13.913185+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:00:13.913185+0000","last_undegraded":"2024-03-26T03:00:13.913185+0000","last_fullsized":"2024-03-26T03:00:13.913185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":63,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103188,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1376256,"data_stored":932482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430232,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1368064,"data_stored":931864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":811008,"data_stored":343316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1384448,"data_stored":933135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":774144,"data_stored":342553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":802816,"data_stored":341925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":342023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691870,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":778240,"data_stored":341574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:00:21.519 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T03:00:21.521 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T03:00:21.521 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T03:00:21.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T03:00:21.769 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:21 smithi066 podman[72787]: 2024-03-26 03:00:21.578567431 +0000 UTC m=+0.019591341 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T03:00:21.769 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:21 smithi066 podman[72787]: 2024-03-26 03:00:21.676407673 +0000 UTC m=+0.117431573 volume create bd0cd39abbd83ca0fdf134a1acbb1baa264aa91b64373704dcc210590b52970c 2024-03-26T03:00:22.063 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:21 smithi066 podman[72787]: 2024-03-26 03:00:21.768765623 +0000 UTC m=+0.209789564 container create 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:00:22.063 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 podman[72787]: 2024-03-26 03:00:22.062779604 +0000 UTC m=+0.503803504 container init 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:00:22.195 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:22.320 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[37920]: 5.d deep-scrub starts 2024-03-26T03:00:22.320 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[37920]: 5.d deep-scrub ok 2024-03-26T03:00:22.320 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[29666]: 5.d deep-scrub starts 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[29666]: 5.d deep-scrub ok 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:00:22.321 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 podman[72787]: 2024-03-26 03:00:22.066169265 +0000 UTC m=+0.507193170 container start 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.079Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.079Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.080Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.66 port=9094 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.081Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.104Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:00:22.322 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.105Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:00:22.323 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.108Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T03:00:22.323 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:22.108Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T03:00:22.323 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 bash[72787]: 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 2024-03-26T03:00:22.323 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:22 smithi066 systemd[1]: Started Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:00:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:22 smithi192 ceph-mon[33110]: 5.d deep-scrub starts 2024-03-26T03:00:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:22 smithi192 ceph-mon[33110]: 5.d deep-scrub ok 2024-03-26T03:00:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:22 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:00:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:22 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:00:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:22 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:00:22.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:22 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:00:22.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:22] ENGINE Bus STOPPING 2024-03-26T03:00:22.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:00:22.609 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:22] ENGINE Bus STOPPED 2024-03-26T03:00:22.609 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:22] ENGINE Bus STARTING 2024-03-26T03:00:22.609 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:22] ENGINE Serving on http://:::9283 2024-03-26T03:00:22.609 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:22 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:22] ENGINE Bus STARTED 2024-03-26T03:00:23.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:00:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:00:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:00:23.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:23.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:23.919 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:00:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:00:24.175 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:24.081Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000082716s 2024-03-26T03:00:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: Deploying daemon grafana.a on smithi192 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: 4.10 scrub starts 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: 4.10 scrub ok 2024-03-26T03:00:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:24 smithi192 ceph-mon[33110]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T03:00:24.546 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:00:24.546 INFO:teuthology.orchestra.run.smithi066.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: Deploying daemon grafana.a on smithi192 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: 4.10 scrub starts 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: 4.10 scrub ok 2024-03-26T03:00:24.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[29666]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: Deploying daemon grafana.a on smithi192 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: 4.10 scrub starts 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: 4.10 scrub ok 2024-03-26T03:00:24.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:24 smithi066 ceph-mon[37920]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T03:00:25.140 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T03:00:25.141 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T03:00:25.141 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-26T03:00:25.153 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T03:00:25.153 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-adk-testing-2024-03-25-1628'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'OSDMAP_FLAGS', 'PG_DEGRADED', 'OSD_DOWN', 'POOL_APP_NOT_ENABLED', 'CACHE_POOL_NO_HIT_SET', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-03-25-1628', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}} 2024-03-26T03:00:25.153 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T03:00:25.154 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-26T03:00:25.409 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:25 smithi192 ceph-mon[33110]: 2.1 scrub starts 2024-03-26T03:00:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:25 smithi192 ceph-mon[33110]: 2.1 scrub ok 2024-03-26T03:00:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:25 smithi192 ceph-mon[33110]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:00:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:25 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:00:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:25 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:00:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3565047774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T03:00:25.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[29666]: 2.1 scrub starts 2024-03-26T03:00:25.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[29666]: 2.1 scrub ok 2024-03-26T03:00:25.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[29666]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:00:25.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:00:25.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:00:25.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3565047774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T03:00:25.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[37920]: 2.1 scrub starts 2024-03-26T03:00:25.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[37920]: 2.1 scrub ok 2024-03-26T03:00:25.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[37920]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:00:25.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:00:25.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:00:25.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3565047774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T03:00:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 4.9 scrub starts 2024-03-26T03:00:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 4.9 scrub ok 2024-03-26T03:00:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:00:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:00:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:00:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:00:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 4.1d scrub starts 2024-03-26T03:00:26.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:26 smithi192 ceph-mon[33110]: 4.1d scrub ok 2024-03-26T03:00:26.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 4.9 scrub starts 2024-03-26T03:00:26.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 4.9 scrub ok 2024-03-26T03:00:26.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:00:26.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:00:26.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:00:26.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:00:26.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 4.1d scrub starts 2024-03-26T03:00:26.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[29666]: 4.1d scrub ok 2024-03-26T03:00:26.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 4.9 scrub starts 2024-03-26T03:00:26.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 4.9 scrub ok 2024-03-26T03:00:26.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:00:26.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:00:26.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:00:26.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:00:26.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 4.1d scrub starts 2024-03-26T03:00:26.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:26 smithi066 ceph-mon[37920]: 4.1d scrub ok 2024-03-26T03:00:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:00:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:00:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:00:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:00:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:00:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:00:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:00:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 4.14 scrub starts 2024-03-26T03:00:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:27 smithi192 ceph-mon[33110]: 4.14 scrub ok 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:00:27.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 4.14 scrub starts 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[29666]: 4.14 scrub ok 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 4.14 scrub starts 2024-03-26T03:00:27.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-mon[37920]: 4.14 scrub ok 2024-03-26T03:00:27.570 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:27] ENGINE Bus STOPPING 2024-03-26T03:00:27.570 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:00:27.570 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:27] ENGINE Bus STOPPED 2024-03-26T03:00:27.570 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:27] ENGINE Bus STARTING 2024-03-26T03:00:27.570 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:27] ENGINE Serving on http://:::9283 2024-03-26T03:00:27.570 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:27 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:00:27] ENGINE Bus STARTED 2024-03-26T03:00:28.202 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-26T03:00:28.203 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T03:00:28.203 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:00:28.465 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:28.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 5.b deep-scrub starts 2024-03-26T03:00:28.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 5.b deep-scrub ok 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 4.b deep-scrub starts 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 4.b deep-scrub ok 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: from='client.34640 ' entity='client.admin' 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 5.b deep-scrub starts 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 5.b deep-scrub ok 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:00:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 4.b deep-scrub starts 2024-03-26T03:00:28.589 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 4.b deep-scrub ok 2024-03-26T03:00:28.589 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: from='client.34640 ' entity='client.admin' 2024-03-26T03:00:28.589 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:00:28.589 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:28 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:00:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 5.b deep-scrub starts 2024-03-26T03:00:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 5.b deep-scrub ok 2024-03-26T03:00:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:00:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:00:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 4.b deep-scrub starts 2024-03-26T03:00:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 4.b deep-scrub ok 2024-03-26T03:00:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: from='client.34640 ' entity='client.admin' 2024-03-26T03:00:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:00:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:00:29.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:00:28 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:00:29.593 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:29 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:00:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:00:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:00:29.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:00:29.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:29 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:00:30.410 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:00:30.412 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:00:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:30 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:00:30.590 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:30 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:00:30.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:30 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:00:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:30 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:00:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:30 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:00:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:30 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:00:31.077 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-26T03:00:29.922448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1305,"num_read_kb":1119,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":823,"ondisk_log_size":823,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236988,"kb_used_data":8380,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724228,"statfs":{"total":767960285184,"available":767717609472,"internally_reserved":0,"allocated":8581120,"data_stored":4769592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.860792"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869732+0000","last_change":"2024-03-26T02:59:03.208828+0000","last_active":"2024-03-26T03:00:23.869732+0000","last_peered":"2024-03-26T03:00:23.869732+0000","last_clean":"2024-03-26T03:00:23.869732+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:00:23.869732+0000","last_undegraded":"2024-03-26T03:00:23.869732+0000","last_fullsized":"2024-03-26T03:00:23.869732+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.769121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058108+0000","last_change":"2024-03-26T02:59:00.943722+0000","last_active":"2024-03-26T03:00:24.058108+0000","last_peered":"2024-03-26T03:00:24.058108+0000","last_clean":"2024-03-26T03:00:24.058108+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:00:24.058108+0000","last_undegraded":"2024-03-26T03:00:24.058108+0000","last_fullsized":"2024-03-26T03:00:24.058108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.322632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871730+0000","last_change":"2024-03-26T03:00:22.240686+0000","last_active":"2024-03-26T03:00:23.871730+0000","last_peered":"2024-03-26T03:00:23.871730+0000","last_clean":"2024-03-26T03:00:23.871730+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:00:23.871730+0000","last_undegraded":"2024-03-26T03:00:23.871730+0000","last_fullsized":"2024-03-26T03:00:23.871730+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:22.240595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:22.240595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.476581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869622+0000","last_change":"2024-03-26T02:58:58.929759+0000","last_active":"2024-03-26T03:00:23.869622+0000","last_peered":"2024-03-26T03:00:23.869622+0000","last_clean":"2024-03-26T03:00:23.869622+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:00:23.869622+0000","last_undegraded":"2024-03-26T03:00:23.869622+0000","last_fullsized":"2024-03-26T03:00:23.869622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058296+0000","last_change":"2024-03-26T02:59:02.945015+0000","last_active":"2024-03-26T03:00:24.058296+0000","last_peered":"2024-03-26T03:00:24.058296+0000","last_clean":"2024-03-26T03:00:24.058296+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:00:24.058296+0000","last_undegraded":"2024-03-26T03:00:24.058296+0000","last_fullsized":"2024-03-26T03:00:24.058296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977196+0000","last_change":"2024-03-26T02:59:00.932288+0000","last_active":"2024-03-26T03:00:24.977196+0000","last_peered":"2024-03-26T03:00:24.977196+0000","last_clean":"2024-03-26T03:00:24.977196+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:00:24.977196+0000","last_undegraded":"2024-03-26T03:00:24.977196+0000","last_fullsized":"2024-03-26T03:00:24.977196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.996531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870222+0000","last_change":"2024-03-26T02:58:56.920528+0000","last_active":"2024-03-26T03:00:23.870222+0000","last_peered":"2024-03-26T03:00:23.870222+0000","last_clean":"2024-03-26T03:00:23.870222+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:00:23.870222+0000","last_undegraded":"2024-03-26T03:00:23.870222+0000","last_fullsized":"2024-03-26T03:00:23.870222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058324+0000","last_change":"2024-03-26T02:58:58.927137+0000","last_active":"2024-03-26T03:00:24.058324+0000","last_peered":"2024-03-26T03:00:24.058324+0000","last_clean":"2024-03-26T03:00:24.058324+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:00:24.058324+0000","last_undegraded":"2024-03-26T03:00:24.058324+0000","last_fullsized":"2024-03-26T03:00:24.058324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.999396+0000","last_change":"2024-03-26T03:00:25.999429+0000","last_active":"2024-03-26T03:00:25.999396+0000","last_peered":"2024-03-26T03:00:25.999396+0000","last_clean":"2024-03-26T03:00:25.999396+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:00:25.999396+0000","last_undegraded":"2024-03-26T03:00:25.999396+0000","last_fullsized":"2024-03-26T03:00:25.999396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.999320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.999320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:43.465119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870420+0000","last_change":"2024-03-26T03:00:18.269455+0000","last_active":"2024-03-26T03:00:23.870420+0000","last_peered":"2024-03-26T03:00:23.870420+0000","last_clean":"2024-03-26T03:00:23.870420+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:00:23.870420+0000","last_undegraded":"2024-03-26T03:00:23.870420+0000","last_fullsized":"2024-03-26T03:00:23.870420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.269349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.269349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:35.865636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870664+0000","last_change":"2024-03-26T02:58:56.924320+0000","last_active":"2024-03-26T03:00:23.870664+0000","last_peered":"2024-03-26T03:00:23.870664+0000","last_clean":"2024-03-26T03:00:23.870664+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:00:23.870664+0000","last_undegraded":"2024-03-26T03:00:23.870664+0000","last_fullsized":"2024-03-26T03:00:23.870664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058577+0000","last_change":"2024-03-26T02:58:58.931435+0000","last_active":"2024-03-26T03:00:24.058577+0000","last_peered":"2024-03-26T03:00:24.058577+0000","last_clean":"2024-03-26T03:00:24.058577+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:00:24.058577+0000","last_undegraded":"2024-03-26T03:00:24.058577+0000","last_fullsized":"2024-03-26T03:00:24.058577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.301399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871514+0000","last_change":"2024-03-26T02:59:02.947270+0000","last_active":"2024-03-26T03:00:23.871514+0000","last_peered":"2024-03-26T03:00:23.871514+0000","last_clean":"2024-03-26T03:00:23.871514+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:00:23.871514+0000","last_undegraded":"2024-03-26T03:00:23.871514+0000","last_fullsized":"2024-03-26T03:00:23.871514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057315+0000","last_change":"2024-03-26T03:00:05.805098+0000","last_active":"2024-03-26T03:00:24.057315+0000","last_peered":"2024-03-26T03:00:24.057315+0000","last_clean":"2024-03-26T03:00:24.057315+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:00:24.057315+0000","last_undegraded":"2024-03-26T03:00:24.057315+0000","last_fullsized":"2024-03-26T03:00:24.057315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:05.804994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.804994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.354036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870536+0000","last_change":"2024-03-26T02:58:56.922748+0000","last_active":"2024-03-26T03:00:23.870536+0000","last_peered":"2024-03-26T03:00:23.870536+0000","last_clean":"2024-03-26T03:00:23.870536+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:00:23.870536+0000","last_undegraded":"2024-03-26T03:00:23.870536+0000","last_fullsized":"2024-03-26T03:00:23.870536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058544+0000","last_change":"2024-03-26T03:00:06.927566+0000","last_active":"2024-03-26T03:00:24.058544+0000","last_peered":"2024-03-26T03:00:24.058544+0000","last_clean":"2024-03-26T03:00:24.058544+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:00:24.058544+0000","last_undegraded":"2024-03-26T03:00:24.058544+0000","last_fullsized":"2024-03-26T03:00:24.058544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:06.927439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:06.927439+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:07.212337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870295+0000","last_change":"2024-03-26T02:59:03.207222+0000","last_active":"2024-03-26T03:00:23.870295+0000","last_peered":"2024-03-26T03:00:23.870295+0000","last_clean":"2024-03-26T03:00:23.870295+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:00:23.870295+0000","last_undegraded":"2024-03-26T03:00:23.870295+0000","last_fullsized":"2024-03-26T03:00:23.870295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.830165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058594+0000","last_change":"2024-03-26T02:59:00.943118+0000","last_active":"2024-03-26T03:00:24.058594+0000","last_peered":"2024-03-26T03:00:24.058594+0000","last_clean":"2024-03-26T03:00:24.058594+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:00:24.058594+0000","last_undegraded":"2024-03-26T03:00:24.058594+0000","last_fullsized":"2024-03-26T03:00:24.058594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869957+0000","last_change":"2024-03-26T03:00:02.741809+0000","last_active":"2024-03-26T03:00:23.869957+0000","last_peered":"2024-03-26T03:00:23.869957+0000","last_clean":"2024-03-26T03:00:23.869957+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:00:23.869957+0000","last_undegraded":"2024-03-26T03:00:23.869957+0000","last_fullsized":"2024-03-26T03:00:23.869957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.741725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.741725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.926475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.578207+0000","last_change":"2024-03-26T03:00:25.578239+0000","last_active":"2024-03-26T03:00:25.578207+0000","last_peered":"2024-03-26T03:00:25.578207+0000","last_clean":"2024-03-26T03:00:25.578207+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:00:25.578207+0000","last_undegraded":"2024-03-26T03:00:25.578207+0000","last_fullsized":"2024-03-26T03:00:25.578207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:00:25.578139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.578139+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.013114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.177502+0000","last_change":"2024-03-26T03:00:27.177539+0000","last_active":"2024-03-26T03:00:27.177502+0000","last_peered":"2024-03-26T03:00:27.177502+0000","last_clean":"2024-03-26T03:00:27.177502+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:00:27.177502+0000","last_undegraded":"2024-03-26T03:00:27.177502+0000","last_fullsized":"2024-03-26T03:00:27.177502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.177404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.177404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:28.371450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977912+0000","last_change":"2024-03-26T03:00:18.300021+0000","last_active":"2024-03-26T03:00:24.977912+0000","last_peered":"2024-03-26T03:00:24.977912+0000","last_clean":"2024-03-26T03:00:24.977912+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:00:24.977912+0000","last_undegraded":"2024-03-26T03:00:24.977912+0000","last_fullsized":"2024-03-26T03:00:24.977912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.299882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.299882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.440696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080359+0000","last_change":"2024-03-26T03:00:18.484389+0000","last_active":"2024-03-26T03:00:24.080359+0000","last_peered":"2024-03-26T03:00:24.080359+0000","last_clean":"2024-03-26T03:00:24.080359+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:00:24.080359+0000","last_undegraded":"2024-03-26T03:00:24.080359+0000","last_fullsized":"2024-03-26T03:00:24.080359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.484287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.484287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.739970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.458079+0000","last_change":"2024-03-26T03:00:04.540480+0000","last_active":"2024-03-26T03:00:24.458079+0000","last_peered":"2024-03-26T03:00:24.458079+0000","last_clean":"2024-03-26T03:00:24.458079+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:00:24.458079+0000","last_undegraded":"2024-03-26T03:00:24.458079+0000","last_fullsized":"2024-03-26T03:00:24.458079+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:04.540375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.540375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.104233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977691+0000","last_change":"2024-03-26T02:59:02.938768+0000","last_active":"2024-03-26T03:00:24.977691+0000","last_peered":"2024-03-26T03:00:24.977691+0000","last_clean":"2024-03-26T03:00:24.977691+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:00:24.977691+0000","last_undegraded":"2024-03-26T03:00:24.977691+0000","last_fullsized":"2024-03-26T03:00:24.977691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.885569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872297+0000","last_change":"2024-03-26T03:00:13.118260+0000","last_active":"2024-03-26T03:00:23.872297+0000","last_peered":"2024-03-26T03:00:23.872297+0000","last_clean":"2024-03-26T03:00:23.872297+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:00:23.872297+0000","last_undegraded":"2024-03-26T03:00:23.872297+0000","last_fullsized":"2024-03-26T03:00:23.872297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.118140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.552791+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869795+0000","last_change":"2024-03-26T02:58:56.922782+0000","last_active":"2024-03-26T03:00:23.869795+0000","last_peered":"2024-03-26T03:00:23.869795+0000","last_clean":"2024-03-26T03:00:23.869795+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:00:23.869795+0000","last_undegraded":"2024-03-26T03:00:23.869795+0000","last_fullsized":"2024-03-26T03:00:23.869795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080898+0000","last_change":"2024-03-26T02:58:58.929170+0000","last_active":"2024-03-26T03:00:24.080898+0000","last_peered":"2024-03-26T03:00:24.080898+0000","last_clean":"2024-03-26T03:00:24.080898+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:00:24.080898+0000","last_undegraded":"2024-03-26T03:00:24.080898+0000","last_fullsized":"2024-03-26T03:00:24.080898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.402094+0000","last_change":"2024-03-26T03:00:25.402119+0000","last_active":"2024-03-26T03:00:25.402094+0000","last_peered":"2024-03-26T03:00:25.402094+0000","last_clean":"2024-03-26T03:00:25.402094+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:00:25.402094+0000","last_undegraded":"2024-03-26T03:00:25.402094+0000","last_fullsized":"2024-03-26T03:00:25.402094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:00:25.402032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.402032+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:40.441905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.586479+0000","last_change":"2024-03-26T03:00:25.586479+0000","last_active":"2024-03-26T03:00:25.586479+0000","last_peered":"2024-03-26T03:00:25.586479+0000","last_clean":"2024-03-26T03:00:25.586479+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:00:25.586479+0000","last_undegraded":"2024-03-26T03:00:25.586479+0000","last_fullsized":"2024-03-26T03:00:25.586479+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:25.586362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.586362+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.794553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.056989+0000","last_change":"2024-03-26T03:00:18.957459+0000","last_active":"2024-03-26T03:00:24.056989+0000","last_peered":"2024-03-26T03:00:24.056989+0000","last_clean":"2024-03-26T03:00:24.056989+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:00:24.056989+0000","last_undegraded":"2024-03-26T03:00:24.056989+0000","last_fullsized":"2024-03-26T03:00:24.056989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.957349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:36.020872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872241+0000","last_change":"2024-03-26T02:58:58.945932+0000","last_active":"2024-03-26T03:00:23.872241+0000","last_peered":"2024-03-26T03:00:23.872241+0000","last_clean":"2024-03-26T03:00:23.872241+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:00:23.872241+0000","last_undegraded":"2024-03-26T03:00:23.872241+0000","last_fullsized":"2024-03-26T03:00:23.872241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870307+0000","last_change":"2024-03-26T02:59:03.207173+0000","last_active":"2024-03-26T03:00:23.870307+0000","last_peered":"2024-03-26T03:00:23.870307+0000","last_clean":"2024-03-26T03:00:23.870307+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:00:23.870307+0000","last_undegraded":"2024-03-26T03:00:23.870307+0000","last_fullsized":"2024-03-26T03:00:23.870307+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.434157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870423+0000","last_change":"2024-03-26T03:00:04.453033+0000","last_active":"2024-03-26T03:00:23.870423+0000","last_peered":"2024-03-26T03:00:23.870423+0000","last_clean":"2024-03-26T03:00:23.870423+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:00:23.870423+0000","last_undegraded":"2024-03-26T03:00:23.870423+0000","last_fullsized":"2024-03-26T03:00:23.870423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.452915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.452915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.497308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.081019+0000","last_change":"2024-03-26T02:58:56.925003+0000","last_active":"2024-03-26T03:00:24.081019+0000","last_peered":"2024-03-26T03:00:24.081019+0000","last_clean":"2024-03-26T03:00:24.081019+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:00:24.081019+0000","last_undegraded":"2024-03-26T03:00:24.081019+0000","last_fullsized":"2024-03-26T03:00:24.081019+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"71'17","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870485+0000","last_change":"2024-03-26T02:58:58.945243+0000","last_active":"2024-03-26T03:00:23.870485+0000","last_peered":"2024-03-26T03:00:23.870485+0000","last_clean":"2024-03-26T03:00:23.870485+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:00:23.870485+0000","last_undegraded":"2024-03-26T03:00:23.870485+0000","last_fullsized":"2024-03-26T03:00:23.870485+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870199+0000","last_change":"2024-03-26T02:59:02.941127+0000","last_active":"2024-03-26T03:00:23.870199+0000","last_peered":"2024-03-26T03:00:23.870199+0000","last_clean":"2024-03-26T03:00:23.870199+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:00:23.870199+0000","last_undegraded":"2024-03-26T03:00:23.870199+0000","last_fullsized":"2024-03-26T03:00:23.870199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080061+0000","last_change":"2024-03-26T03:00:21.571415+0000","last_active":"2024-03-26T03:00:24.080061+0000","last_peered":"2024-03-26T03:00:24.080061+0000","last_clean":"2024-03-26T03:00:24.080061+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:00:24.080061+0000","last_undegraded":"2024-03-26T03:00:24.080061+0000","last_fullsized":"2024-03-26T03:00:24.080061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:21.571273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:21.571273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.284803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057744+0000","last_change":"2024-03-26T03:00:02.803641+0000","last_active":"2024-03-26T03:00:24.057744+0000","last_peered":"2024-03-26T03:00:24.057744+0000","last_clean":"2024-03-26T03:00:24.057744+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:00:24.057744+0000","last_undegraded":"2024-03-26T03:00:24.057744+0000","last_fullsized":"2024-03-26T03:00:24.057744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.803511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.803511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.142559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869615+0000","last_change":"2024-03-26T03:00:23.647837+0000","last_active":"2024-03-26T03:00:23.869615+0000","last_peered":"2024-03-26T03:00:23.869615+0000","last_clean":"2024-03-26T03:00:23.869615+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:00:23.869615+0000","last_undegraded":"2024-03-26T03:00:23.869615+0000","last_fullsized":"2024-03-26T03:00:23.869615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:00:23.647732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.647732+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.169412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869543+0000","last_change":"2024-03-26T02:59:02.945112+0000","last_active":"2024-03-26T03:00:23.869543+0000","last_peered":"2024-03-26T03:00:23.869543+0000","last_clean":"2024-03-26T03:00:23.869543+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:00:23.869543+0000","last_undegraded":"2024-03-26T03:00:23.869543+0000","last_fullsized":"2024-03-26T03:00:23.869543+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977410+0000","last_change":"2024-03-26T02:59:00.936834+0000","last_active":"2024-03-26T03:00:24.977410+0000","last_peered":"2024-03-26T03:00:24.977410+0000","last_clean":"2024-03-26T03:00:24.977410+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:00:24.977410+0000","last_undegraded":"2024-03-26T03:00:24.977410+0000","last_fullsized":"2024-03-26T03:00:24.977410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.104445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976980+0000","last_change":"2024-03-26T03:00:02.224750+0000","last_active":"2024-03-26T03:00:24.976980+0000","last_peered":"2024-03-26T03:00:24.976980+0000","last_clean":"2024-03-26T03:00:24.976980+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:00:24.976980+0000","last_undegraded":"2024-03-26T03:00:24.976980+0000","last_fullsized":"2024-03-26T03:00:24.976980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.224639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.224639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.778744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058733+0000","last_change":"2024-03-26T02:58:58.925892+0000","last_active":"2024-03-26T03:00:24.058733+0000","last_peered":"2024-03-26T03:00:24.058733+0000","last_clean":"2024-03-26T03:00:24.058733+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:00:24.058733+0000","last_undegraded":"2024-03-26T03:00:24.058733+0000","last_fullsized":"2024-03-26T03:00:24.058733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871093+0000","last_change":"2024-03-26T02:59:02.948086+0000","last_active":"2024-03-26T03:00:23.871093+0000","last_peered":"2024-03-26T03:00:23.871093+0000","last_clean":"2024-03-26T03:00:23.871093+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:00:23.871093+0000","last_undegraded":"2024-03-26T03:00:23.871093+0000","last_fullsized":"2024-03-26T03:00:23.871093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870163+0000","last_change":"2024-03-26T03:00:17.616155+0000","last_active":"2024-03-26T03:00:23.870163+0000","last_peered":"2024-03-26T03:00:23.870163+0000","last_clean":"2024-03-26T03:00:23.870163+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:00:23.870163+0000","last_undegraded":"2024-03-26T03:00:23.870163+0000","last_fullsized":"2024-03-26T03:00:23.870163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:17.616013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.616013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.986671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080509+0000","last_change":"2024-03-26T03:00:07.455788+0000","last_active":"2024-03-26T03:00:24.080509+0000","last_peered":"2024-03-26T03:00:24.080509+0000","last_clean":"2024-03-26T03:00:24.080509+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:00:24.080509+0000","last_undegraded":"2024-03-26T03:00:24.080509+0000","last_fullsized":"2024-03-26T03:00:24.080509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.455691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.455691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.746322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977255+0000","last_change":"2024-03-26T02:58:58.928475+0000","last_active":"2024-03-26T03:00:24.977255+0000","last_peered":"2024-03-26T03:00:24.977255+0000","last_clean":"2024-03-26T03:00:24.977255+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:00:24.977255+0000","last_undegraded":"2024-03-26T03:00:24.977255+0000","last_fullsized":"2024-03-26T03:00:24.977255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059183+0000","last_change":"2024-03-26T02:59:02.944972+0000","last_active":"2024-03-26T03:00:24.059183+0000","last_peered":"2024-03-26T03:00:24.059183+0000","last_clean":"2024-03-26T03:00:24.059183+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:00:24.059183+0000","last_undegraded":"2024-03-26T03:00:24.059183+0000","last_fullsized":"2024-03-26T03:00:24.059183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.212361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870128+0000","last_change":"2024-03-26T02:59:00.933699+0000","last_active":"2024-03-26T03:00:23.870128+0000","last_peered":"2024-03-26T03:00:23.870128+0000","last_clean":"2024-03-26T03:00:23.870128+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:00:23.870128+0000","last_undegraded":"2024-03-26T03:00:23.870128+0000","last_fullsized":"2024-03-26T03:00:23.870128+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871925+0000","last_change":"2024-03-26T02:58:56.914857+0000","last_active":"2024-03-26T03:00:23.871925+0000","last_peered":"2024-03-26T03:00:23.871925+0000","last_clean":"2024-03-26T03:00:23.871925+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:00:23.871925+0000","last_undegraded":"2024-03-26T03:00:23.871925+0000","last_fullsized":"2024-03-26T03:00:23.871925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.500241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057582+0000","last_change":"2024-03-26T02:58:58.945768+0000","last_active":"2024-03-26T03:00:24.057582+0000","last_peered":"2024-03-26T03:00:24.057582+0000","last_clean":"2024-03-26T03:00:24.057582+0000","last_became_active":"2024-03-26T02:58:58.945477+0000","last_became_peered":"2024-03-26T02:58:58.945477+0000","last_unstale":"2024-03-26T03:00:24.057582+0000","last_undegraded":"2024-03-26T03:00:24.057582+0000","last_fullsized":"2024-03-26T03:00:24.057582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871918+0000","last_change":"2024-03-26T03:00:09.986685+0000","last_active":"2024-03-26T03:00:23.871918+0000","last_peered":"2024-03-26T03:00:23.871918+0000","last_clean":"2024-03-26T03:00:23.871918+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:00:23.871918+0000","last_undegraded":"2024-03-26T03:00:23.871918+0000","last_fullsized":"2024-03-26T03:00:23.871918+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:09.986591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:09.986591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.856847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057559+0000","last_change":"2024-03-26T03:00:10.855859+0000","last_active":"2024-03-26T03:00:24.057559+0000","last_peered":"2024-03-26T03:00:24.057559+0000","last_clean":"2024-03-26T03:00:24.057559+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:00:24.057559+0000","last_undegraded":"2024-03-26T03:00:24.057559+0000","last_fullsized":"2024-03-26T03:00:24.057559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:10.855698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.855698+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.308595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057206+0000","last_change":"2024-03-26T03:00:12.868763+0000","last_active":"2024-03-26T03:00:24.057206+0000","last_peered":"2024-03-26T03:00:24.057206+0000","last_clean":"2024-03-26T03:00:24.057206+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:00:24.057206+0000","last_undegraded":"2024-03-26T03:00:24.057206+0000","last_fullsized":"2024-03-26T03:00:24.057206+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:12.868624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:12.868624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.398619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"71'22","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:26.545271+0000","last_change":"2024-03-26T03:00:26.545331+0000","last_active":"2024-03-26T03:00:26.545271+0000","last_peered":"2024-03-26T03:00:26.545271+0000","last_clean":"2024-03-26T03:00:26.545271+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:00:26.545271+0000","last_undegraded":"2024-03-26T03:00:26.545271+0000","last_fullsized":"2024-03-26T03:00:26.545271+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'22","last_scrub_stamp":"2024-03-26T03:00:26.545179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.545179+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.405420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870590+0000","last_change":"2024-03-26T02:59:03.208607+0000","last_active":"2024-03-26T03:00:23.870590+0000","last_peered":"2024-03-26T03:00:23.870590+0000","last_clean":"2024-03-26T03:00:23.870590+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:00:23.870590+0000","last_undegraded":"2024-03-26T03:00:23.870590+0000","last_fullsized":"2024-03-26T03:00:23.870590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057707+0000","last_change":"2024-03-26T03:00:20.999242+0000","last_active":"2024-03-26T03:00:24.057707+0000","last_peered":"2024-03-26T03:00:24.057707+0000","last_clean":"2024-03-26T03:00:24.057707+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:00:24.057707+0000","last_undegraded":"2024-03-26T03:00:24.057707+0000","last_fullsized":"2024-03-26T03:00:24.057707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:20.999149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.999149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.966830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080165+0000","last_change":"2024-03-26T03:00:04.420607+0000","last_active":"2024-03-26T03:00:24.080165+0000","last_peered":"2024-03-26T03:00:24.080165+0000","last_clean":"2024-03-26T03:00:24.080165+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:00:24.080165+0000","last_undegraded":"2024-03-26T03:00:24.080165+0000","last_fullsized":"2024-03-26T03:00:24.080165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.420504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.420504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.414672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870569+0000","last_change":"2024-03-26T03:00:05.814094+0000","last_active":"2024-03-26T03:00:23.870569+0000","last_peered":"2024-03-26T03:00:23.870569+0000","last_clean":"2024-03-26T03:00:23.870569+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:00:23.870569+0000","last_undegraded":"2024-03-26T03:00:23.870569+0000","last_fullsized":"2024-03-26T03:00:23.870569+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:00:05.813958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.813958+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.910237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457817+0000","last_change":"2024-03-26T02:59:02.942788+0000","last_active":"2024-03-26T03:00:24.457817+0000","last_peered":"2024-03-26T03:00:24.457817+0000","last_clean":"2024-03-26T03:00:24.457817+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:00:24.457817+0000","last_undegraded":"2024-03-26T03:00:24.457817+0000","last_fullsized":"2024-03-26T03:00:24.457817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.084675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870395+0000","last_change":"2024-03-26T03:00:07.801621+0000","last_active":"2024-03-26T03:00:23.870395+0000","last_peered":"2024-03-26T03:00:23.870395+0000","last_clean":"2024-03-26T03:00:23.870395+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:00:23.870395+0000","last_undegraded":"2024-03-26T03:00:23.870395+0000","last_fullsized":"2024-03-26T03:00:23.870395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.801516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.801516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.176058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457497+0000","last_change":"2024-03-26T02:58:56.925054+0000","last_active":"2024-03-26T03:00:24.457497+0000","last_peered":"2024-03-26T03:00:24.457497+0000","last_clean":"2024-03-26T03:00:24.457497+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:00:24.457497+0000","last_undegraded":"2024-03-26T03:00:24.457497+0000","last_fullsized":"2024-03-26T03:00:24.457497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.471451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872132+0000","last_change":"2024-03-26T02:58:58.946140+0000","last_active":"2024-03-26T03:00:23.872132+0000","last_peered":"2024-03-26T03:00:23.872132+0000","last_clean":"2024-03-26T03:00:23.872132+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:00:23.872132+0000","last_undegraded":"2024-03-26T03:00:23.872132+0000","last_fullsized":"2024-03-26T03:00:23.872132+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870275+0000","last_change":"2024-03-26T02:59:02.942511+0000","last_active":"2024-03-26T03:00:23.870275+0000","last_peered":"2024-03-26T03:00:23.870275+0000","last_clean":"2024-03-26T03:00:23.870275+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:00:23.870275+0000","last_undegraded":"2024-03-26T03:00:23.870275+0000","last_fullsized":"2024-03-26T03:00:23.870275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059059+0000","last_change":"2024-03-26T02:58:56.919704+0000","last_active":"2024-03-26T03:00:24.059059+0000","last_peered":"2024-03-26T03:00:24.059059+0000","last_clean":"2024-03-26T03:00:24.059059+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:00:24.059059+0000","last_undegraded":"2024-03-26T03:00:24.059059+0000","last_fullsized":"2024-03-26T03:00:24.059059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.046057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":137,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.142417+0000","last_change":"2024-03-26T03:00:25.142453+0000","last_active":"2024-03-26T03:00:25.142417+0000","last_peered":"2024-03-26T03:00:25.142417+0000","last_clean":"2024-03-26T03:00:25.142417+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:00:25.142417+0000","last_undegraded":"2024-03-26T03:00:25.142417+0000","last_fullsized":"2024-03-26T03:00:25.142417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:00:25.142334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.142334+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:37.994864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457445+0000","last_change":"2024-03-26T03:00:08.469473+0000","last_active":"2024-03-26T03:00:24.457445+0000","last_peered":"2024-03-26T03:00:24.457445+0000","last_clean":"2024-03-26T03:00:24.457445+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:00:24.457445+0000","last_undegraded":"2024-03-26T03:00:24.457445+0000","last_fullsized":"2024-03-26T03:00:24.457445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:08.469321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.469321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.714692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871178+0000","last_change":"2024-03-26T03:00:11.023432+0000","last_active":"2024-03-26T03:00:23.871178+0000","last_peered":"2024-03-26T03:00:23.871178+0000","last_clean":"2024-03-26T03:00:23.871178+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:00:23.871178+0000","last_undegraded":"2024-03-26T03:00:23.871178+0000","last_fullsized":"2024-03-26T03:00:23.871178+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:11.023351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:11.023351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.872070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.327191+0000","last_change":"2024-03-26T03:00:27.327214+0000","last_active":"2024-03-26T03:00:27.327191+0000","last_peered":"2024-03-26T03:00:27.327191+0000","last_clean":"2024-03-26T03:00:27.327191+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:00:27.327191+0000","last_undegraded":"2024-03-26T03:00:27.327191+0000","last_fullsized":"2024-03-26T03:00:27.327191+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.327116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.327116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.083798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"71'16","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977357+0000","last_change":"2024-03-26T02:58:58.926863+0000","last_active":"2024-03-26T03:00:24.977357+0000","last_peered":"2024-03-26T03:00:24.977357+0000","last_clean":"2024-03-26T03:00:24.977357+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:00:24.977357+0000","last_undegraded":"2024-03-26T03:00:24.977357+0000","last_fullsized":"2024-03-26T03:00:24.977357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872123+0000","last_change":"2024-03-26T03:00:08.968963+0000","last_active":"2024-03-26T03:00:23.872123+0000","last_peered":"2024-03-26T03:00:23.872123+0000","last_clean":"2024-03-26T03:00:23.872123+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:00:23.872123+0000","last_undegraded":"2024-03-26T03:00:23.872123+0000","last_fullsized":"2024-03-26T03:00:23.872123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:08.968848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.968848+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.313221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869815+0000","last_change":"2024-03-26T02:59:02.941781+0000","last_active":"2024-03-26T03:00:23.869815+0000","last_peered":"2024-03-26T03:00:23.869815+0000","last_clean":"2024-03-26T03:00:23.869815+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:00:23.869815+0000","last_undegraded":"2024-03-26T03:00:23.869815+0000","last_fullsized":"2024-03-26T03:00:23.869815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.196474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057273+0000","last_change":"2024-03-26T02:58:56.923680+0000","last_active":"2024-03-26T03:00:24.057273+0000","last_peered":"2024-03-26T03:00:24.057273+0000","last_clean":"2024-03-26T03:00:24.057273+0000","last_became_active":"2024-03-26T02:58:56.923317+0000","last_became_peered":"2024-03-26T02:58:56.923317+0000","last_unstale":"2024-03-26T03:00:24.057273+0000","last_undegraded":"2024-03-26T03:00:24.057273+0000","last_fullsized":"2024-03-26T03:00:24.057273+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.601353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":159,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977769+0000","last_change":"2024-03-26T03:00:13.375248+0000","last_active":"2024-03-26T03:00:24.977769+0000","last_peered":"2024-03-26T03:00:24.977769+0000","last_clean":"2024-03-26T03:00:24.977769+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:00:24.977769+0000","last_undegraded":"2024-03-26T03:00:24.977769+0000","last_fullsized":"2024-03-26T03:00:24.977769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.375080+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.080768+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870705+0000","last_change":"2024-03-26T02:59:00.940319+0000","last_active":"2024-03-26T03:00:23.870705+0000","last_peered":"2024-03-26T03:00:23.870705+0000","last_clean":"2024-03-26T03:00:23.870705+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:00:23.870705+0000","last_undegraded":"2024-03-26T03:00:23.870705+0000","last_fullsized":"2024-03-26T03:00:23.870705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870774+0000","last_change":"2024-03-26T02:59:03.208010+0000","last_active":"2024-03-26T03:00:23.870774+0000","last_peered":"2024-03-26T03:00:23.870774+0000","last_clean":"2024-03-26T03:00:23.870774+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:00:23.870774+0000","last_undegraded":"2024-03-26T03:00:23.870774+0000","last_fullsized":"2024-03-26T03:00:23.870774+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":166,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:26.972354+0000","last_change":"2024-03-26T03:00:04.289538+0000","last_active":"2024-03-26T03:00:26.972354+0000","last_peered":"2024-03-26T03:00:26.972354+0000","last_clean":"2024-03-26T03:00:26.972354+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:00:26.972354+0000","last_undegraded":"2024-03-26T03:00:26.972354+0000","last_fullsized":"2024-03-26T03:00:26.972354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:00:04.289452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.289452+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.576447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457495+0000","last_change":"2024-03-26T03:00:23.413750+0000","last_active":"2024-03-26T03:00:24.457495+0000","last_peered":"2024-03-26T03:00:24.457495+0000","last_clean":"2024-03-26T03:00:24.457495+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:00:24.457495+0000","last_undegraded":"2024-03-26T03:00:24.457495+0000","last_fullsized":"2024-03-26T03:00:24.457495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:23.413656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.413656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:37.034913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977897+0000","last_change":"2024-03-26T03:00:23.305418+0000","last_active":"2024-03-26T03:00:24.977897+0000","last_peered":"2024-03-26T03:00:24.977897+0000","last_clean":"2024-03-26T03:00:24.977897+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:00:24.977897+0000","last_undegraded":"2024-03-26T03:00:24.977897+0000","last_fullsized":"2024-03-26T03:00:24.977897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:00:23.305307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.305307+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:52.977903+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.404936+0000","last_change":"2024-03-26T03:00:25.404962+0000","last_active":"2024-03-26T03:00:25.404936+0000","last_peered":"2024-03-26T03:00:25.404936+0000","last_clean":"2024-03-26T03:00:25.404936+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:00:25.404936+0000","last_undegraded":"2024-03-26T03:00:25.404936+0000","last_fullsized":"2024-03-26T03:00:25.404936+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.404851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.404851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.989593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058495+0000","last_change":"2024-03-26T03:00:24.058533+0000","last_active":"2024-03-26T03:00:24.058495+0000","last_peered":"2024-03-26T03:00:24.058495+0000","last_clean":"2024-03-26T03:00:24.058495+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:00:24.058495+0000","last_undegraded":"2024-03-26T03:00:24.058495+0000","last_fullsized":"2024-03-26T03:00:24.058495+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:24.058394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:24.058394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.713360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869726+0000","last_change":"2024-03-26T02:59:00.924687+0000","last_active":"2024-03-26T03:00:23.869726+0000","last_peered":"2024-03-26T03:00:23.869726+0000","last_clean":"2024-03-26T03:00:23.869726+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:00:23.869726+0000","last_undegraded":"2024-03-26T03:00:23.869726+0000","last_fullsized":"2024-03-26T03:00:23.869726+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057237+0000","last_change":"2024-03-26T03:00:03.775166+0000","last_active":"2024-03-26T03:00:24.057237+0000","last_peered":"2024-03-26T03:00:24.057237+0000","last_clean":"2024-03-26T03:00:24.057237+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:00:24.057237+0000","last_undegraded":"2024-03-26T03:00:24.057237+0000","last_fullsized":"2024-03-26T03:00:24.057237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:03.775065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:03.775065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:04.410162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":312,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870189+0000","last_change":"2024-03-26T02:59:36.442802+0000","last_active":"2024-03-26T03:00:23.870189+0000","last_peered":"2024-03-26T03:00:23.870189+0000","last_clean":"2024-03-26T03:00:23.870189+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:00:23.870189+0000","last_undegraded":"2024-03-26T03:00:23.870189+0000","last_fullsized":"2024-03-26T03:00:23.870189+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-26T02:59:36.442703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T02:59:36.442703+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:44.641515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:28.559224+0000","last_change":"2024-03-26T03:00:28.559278+0000","last_active":"2024-03-26T03:00:28.559224+0000","last_peered":"2024-03-26T03:00:28.559224+0000","last_clean":"2024-03-26T03:00:28.559224+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:00:28.559224+0000","last_undegraded":"2024-03-26T03:00:28.559224+0000","last_fullsized":"2024-03-26T03:00:28.559224+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:00:28.559147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.559147+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:36.143298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870798+0000","last_change":"2024-03-26T03:00:15.331397+0000","last_active":"2024-03-26T03:00:23.870798+0000","last_peered":"2024-03-26T03:00:23.870798+0000","last_clean":"2024-03-26T03:00:23.870798+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:00:23.870798+0000","last_undegraded":"2024-03-26T03:00:23.870798+0000","last_fullsized":"2024-03-26T03:00:23.870798+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.331210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.331210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.836621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977491+0000","last_change":"2024-03-26T02:59:02.935205+0000","last_active":"2024-03-26T03:00:24.977491+0000","last_peered":"2024-03-26T03:00:24.977491+0000","last_clean":"2024-03-26T03:00:24.977491+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:00:24.977491+0000","last_undegraded":"2024-03-26T03:00:24.977491+0000","last_fullsized":"2024-03-26T03:00:24.977491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871844+0000","last_change":"2024-03-26T02:59:00.929356+0000","last_active":"2024-03-26T03:00:23.871844+0000","last_peered":"2024-03-26T03:00:23.871844+0000","last_clean":"2024-03-26T03:00:23.871844+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:00:23.871844+0000","last_undegraded":"2024-03-26T03:00:23.871844+0000","last_fullsized":"2024-03-26T03:00:23.871844+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870137+0000","last_change":"2024-03-26T02:58:56.926508+0000","last_active":"2024-03-26T03:00:23.870137+0000","last_peered":"2024-03-26T03:00:23.870137+0000","last_clean":"2024-03-26T03:00:23.870137+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:00:23.870137+0000","last_undegraded":"2024-03-26T03:00:23.870137+0000","last_fullsized":"2024-03-26T03:00:23.870137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872024+0000","last_change":"2024-03-26T02:59:58.957331+0000","last_active":"2024-03-26T03:00:23.872024+0000","last_peered":"2024-03-26T03:00:23.872024+0000","last_clean":"2024-03-26T03:00:23.872024+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:00:23.872024+0000","last_undegraded":"2024-03-26T03:00:23.872024+0000","last_fullsized":"2024-03-26T03:00:23.872024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T02:59:58.957239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.957239+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:59.168727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057996+0000","last_change":"2024-03-26T02:59:00.935022+0000","last_active":"2024-03-26T03:00:24.057996+0000","last_peered":"2024-03-26T03:00:24.057996+0000","last_clean":"2024-03-26T03:00:24.057996+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:00:24.057996+0000","last_undegraded":"2024-03-26T03:00:24.057996+0000","last_fullsized":"2024-03-26T03:00:24.057996+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.032534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977083+0000","last_change":"2024-03-26T03:00:07.343102+0000","last_active":"2024-03-26T03:00:24.977083+0000","last_peered":"2024-03-26T03:00:24.977083+0000","last_clean":"2024-03-26T03:00:24.977083+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:00:24.977083+0000","last_undegraded":"2024-03-26T03:00:24.977083+0000","last_fullsized":"2024-03-26T03:00:24.977083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.343020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.343020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.674172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870779+0000","last_change":"2024-03-26T02:58:56.923089+0000","last_active":"2024-03-26T03:00:23.870779+0000","last_peered":"2024-03-26T03:00:23.870779+0000","last_clean":"2024-03-26T03:00:23.870779+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:00:23.870779+0000","last_undegraded":"2024-03-26T03:00:23.870779+0000","last_fullsized":"2024-03-26T03:00:23.870779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.226053+0000","last_change":"2024-03-26T03:00:25.226087+0000","last_active":"2024-03-26T03:00:25.226053+0000","last_peered":"2024-03-26T03:00:25.226053+0000","last_clean":"2024-03-26T03:00:25.226053+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:00:25.226053+0000","last_undegraded":"2024-03-26T03:00:25.226053+0000","last_fullsized":"2024-03-26T03:00:25.226053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:00:25.225991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.225991+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.215639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080446+0000","last_change":"2024-03-26T03:00:10.472741+0000","last_active":"2024-03-26T03:00:24.080446+0000","last_peered":"2024-03-26T03:00:24.080446+0000","last_clean":"2024-03-26T03:00:24.080446+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:00:24.080446+0000","last_undegraded":"2024-03-26T03:00:24.080446+0000","last_fullsized":"2024-03-26T03:00:24.080446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:10.472642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.472642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.985525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058713+0000","last_change":"2024-03-26T02:59:02.944368+0000","last_active":"2024-03-26T03:00:24.058713+0000","last_peered":"2024-03-26T03:00:24.058713+0000","last_clean":"2024-03-26T03:00:24.058713+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:00:24.058713+0000","last_undegraded":"2024-03-26T03:00:24.058713+0000","last_fullsized":"2024-03-26T03:00:24.058713+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.999221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976932+0000","last_change":"2024-03-26T03:00:17.286899+0000","last_active":"2024-03-26T03:00:24.976932+0000","last_peered":"2024-03-26T03:00:24.976932+0000","last_clean":"2024-03-26T03:00:24.976932+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:00:24.976932+0000","last_undegraded":"2024-03-26T03:00:24.976932+0000","last_fullsized":"2024-03-26T03:00:24.976932+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.286796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.424244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":137,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869503+0000","last_change":"2024-03-26T03:00:08.772624+0000","last_active":"2024-03-26T03:00:23.869503+0000","last_peered":"2024-03-26T03:00:23.869503+0000","last_clean":"2024-03-26T03:00:23.869503+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:00:23.869503+0000","last_undegraded":"2024-03-26T03:00:23.869503+0000","last_fullsized":"2024-03-26T03:00:23.869503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.772486+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.103774+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058396+0000","last_change":"2024-03-26T02:59:00.944090+0000","last_active":"2024-03-26T03:00:24.058396+0000","last_peered":"2024-03-26T03:00:24.058396+0000","last_clean":"2024-03-26T03:00:24.058396+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:00:24.058396+0000","last_undegraded":"2024-03-26T03:00:24.058396+0000","last_fullsized":"2024-03-26T03:00:24.058396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870697+0000","last_change":"2024-03-26T02:59:03.208325+0000","last_active":"2024-03-26T03:00:23.870697+0000","last_peered":"2024-03-26T03:00:23.870697+0000","last_clean":"2024-03-26T03:00:23.870697+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:00:23.870697+0000","last_undegraded":"2024-03-26T03:00:23.870697+0000","last_fullsized":"2024-03-26T03:00:23.870697+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.081018+0000","last_change":"2024-03-26T02:58:56.922909+0000","last_active":"2024-03-26T03:00:24.081018+0000","last_peered":"2024-03-26T03:00:24.081018+0000","last_clean":"2024-03-26T03:00:24.081018+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:00:24.081018+0000","last_undegraded":"2024-03-26T03:00:24.081018+0000","last_fullsized":"2024-03-26T03:00:24.081018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058262+0000","last_change":"2024-03-26T03:00:07.927770+0000","last_active":"2024-03-26T03:00:24.058262+0000","last_peered":"2024-03-26T03:00:24.058262+0000","last_clean":"2024-03-26T03:00:24.058262+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:00:24.058262+0000","last_undegraded":"2024-03-26T03:00:24.058262+0000","last_fullsized":"2024-03-26T03:00:24.058262+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:00:07.927648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.927648+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:12.893425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869486+0000","last_change":"2024-03-26T02:59:00.937344+0000","last_active":"2024-03-26T03:00:23.869486+0000","last_peered":"2024-03-26T03:00:23.869486+0000","last_clean":"2024-03-26T03:00:23.869486+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:00:23.869486+0000","last_undegraded":"2024-03-26T03:00:23.869486+0000","last_fullsized":"2024-03-26T03:00:23.869486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869822+0000","last_change":"2024-03-26T03:00:15.672174+0000","last_active":"2024-03-26T03:00:23.869822+0000","last_peered":"2024-03-26T03:00:23.869822+0000","last_clean":"2024-03-26T03:00:23.869822+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:00:23.869822+0000","last_undegraded":"2024-03-26T03:00:23.869822+0000","last_fullsized":"2024-03-26T03:00:23.869822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.672090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.667763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.990790+0000","last_change":"2024-03-26T03:00:27.990812+0000","last_active":"2024-03-26T03:00:27.990790+0000","last_peered":"2024-03-26T03:00:27.990790+0000","last_clean":"2024-03-26T03:00:27.990790+0000","last_became_active":"2024-03-26T02:59:00.937450+0000","last_became_peered":"2024-03-26T02:59:00.937450+0000","last_unstale":"2024-03-26T03:00:27.990790+0000","last_undegraded":"2024-03-26T03:00:27.990790+0000","last_fullsized":"2024-03-26T03:00:27.990790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.990730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.990730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.775418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870231+0000","last_change":"2024-03-26T02:58:56.926585+0000","last_active":"2024-03-26T03:00:23.870231+0000","last_peered":"2024-03-26T03:00:23.870231+0000","last_clean":"2024-03-26T03:00:23.870231+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:00:23.870231+0000","last_undegraded":"2024-03-26T03:00:23.870231+0000","last_fullsized":"2024-03-26T03:00:23.870231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059147+0000","last_change":"2024-03-26T03:00:05.879783+0000","last_active":"2024-03-26T03:00:24.059147+0000","last_peered":"2024-03-26T03:00:24.059147+0000","last_clean":"2024-03-26T03:00:24.059147+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:00:24.059147+0000","last_undegraded":"2024-03-26T03:00:24.059147+0000","last_fullsized":"2024-03-26T03:00:24.059147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:05.879672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.879672+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.035707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057412+0000","last_change":"2024-03-26T02:59:02.944645+0000","last_active":"2024-03-26T03:00:24.057412+0000","last_peered":"2024-03-26T03:00:24.057412+0000","last_clean":"2024-03-26T03:00:24.057412+0000","last_became_active":"2024-03-26T02:59:02.944279+0000","last_became_peered":"2024-03-26T02:59:02.944279+0000","last_unstale":"2024-03-26T03:00:24.057412+0000","last_undegraded":"2024-03-26T03:00:24.057412+0000","last_fullsized":"2024-03-26T03:00:24.057412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.965129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059066+0000","last_change":"2024-03-26T02:59:00.934826+0000","last_active":"2024-03-26T03:00:24.059066+0000","last_peered":"2024-03-26T03:00:24.059066+0000","last_clean":"2024-03-26T03:00:24.059066+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:00:24.059066+0000","last_undegraded":"2024-03-26T03:00:24.059066+0000","last_fullsized":"2024-03-26T03:00:24.059066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976774+0000","last_change":"2024-03-26T02:58:56.930599+0000","last_active":"2024-03-26T03:00:24.976774+0000","last_peered":"2024-03-26T03:00:24.976774+0000","last_clean":"2024-03-26T03:00:24.976774+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:00:24.976774+0000","last_undegraded":"2024-03-26T03:00:24.976774+0000","last_fullsized":"2024-03-26T03:00:24.976774+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:28.370997+0000","last_change":"2024-03-26T03:00:28.371034+0000","last_active":"2024-03-26T03:00:28.370997+0000","last_peered":"2024-03-26T03:00:28.370997+0000","last_clean":"2024-03-26T03:00:28.370997+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:00:28.370997+0000","last_undegraded":"2024-03-26T03:00:28.370997+0000","last_fullsized":"2024-03-26T03:00:28.370997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:28.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.370925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:35.020056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058973+0000","last_change":"2024-03-26T02:59:02.945294+0000","last_active":"2024-03-26T03:00:24.058973+0000","last_peered":"2024-03-26T03:00:24.058973+0000","last_clean":"2024-03-26T03:00:24.058973+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:00:24.058973+0000","last_undegraded":"2024-03-26T03:00:24.058973+0000","last_fullsized":"2024-03-26T03:00:24.058973+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457739+0000","last_change":"2024-03-26T03:00:20.452771+0000","last_active":"2024-03-26T03:00:24.457739+0000","last_peered":"2024-03-26T03:00:24.457739+0000","last_clean":"2024-03-26T03:00:24.457739+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:00:24.457739+0000","last_undegraded":"2024-03-26T03:00:24.457739+0000","last_fullsized":"2024-03-26T03:00:24.457739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.452648+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.711803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976779+0000","last_change":"2024-03-26T02:58:56.930656+0000","last_active":"2024-03-26T03:00:24.976779+0000","last_peered":"2024-03-26T03:00:24.976779+0000","last_clean":"2024-03-26T03:00:24.976779+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:00:24.976779+0000","last_undegraded":"2024-03-26T03:00:24.976779+0000","last_fullsized":"2024-03-26T03:00:24.976779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058449+0000","last_change":"2024-03-26T03:00:08.975032+0000","last_active":"2024-03-26T03:00:24.058449+0000","last_peered":"2024-03-26T03:00:24.058449+0000","last_clean":"2024-03-26T03:00:24.058449+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:00:24.058449+0000","last_undegraded":"2024-03-26T03:00:24.058449+0000","last_fullsized":"2024-03-26T03:00:24.058449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:00:08.974874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.974874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.309201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457956+0000","last_change":"2024-03-26T02:59:02.942823+0000","last_active":"2024-03-26T03:00:24.457956+0000","last_peered":"2024-03-26T03:00:24.457956+0000","last_clean":"2024-03-26T03:00:24.457956+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:00:24.457956+0000","last_undegraded":"2024-03-26T03:00:24.457956+0000","last_fullsized":"2024-03-26T03:00:24.457956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.346480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977585+0000","last_change":"2024-03-26T02:59:00.937106+0000","last_active":"2024-03-26T03:00:24.977585+0000","last_peered":"2024-03-26T03:00:24.977585+0000","last_clean":"2024-03-26T03:00:24.977585+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:00:24.977585+0000","last_undegraded":"2024-03-26T03:00:24.977585+0000","last_fullsized":"2024-03-26T03:00:24.977585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870518+0000","last_change":"2024-03-26T03:00:00.526049+0000","last_active":"2024-03-26T03:00:23.870518+0000","last_peered":"2024-03-26T03:00:23.870518+0000","last_clean":"2024-03-26T03:00:23.870518+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:00:23.870518+0000","last_undegraded":"2024-03-26T03:00:23.870518+0000","last_fullsized":"2024-03-26T03:00:23.870518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:00.525932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:00.525932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.138757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058920+0000","last_change":"2024-03-26T02:58:58.921800+0000","last_active":"2024-03-26T03:00:24.058920+0000","last_peered":"2024-03-26T03:00:24.058920+0000","last_clean":"2024-03-26T03:00:24.058920+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:00:24.058920+0000","last_undegraded":"2024-03-26T03:00:24.058920+0000","last_fullsized":"2024-03-26T03:00:24.058920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870069+0000","last_change":"2024-03-26T02:59:02.945728+0000","last_active":"2024-03-26T03:00:23.870069+0000","last_peered":"2024-03-26T03:00:23.870069+0000","last_clean":"2024-03-26T03:00:23.870069+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:00:23.870069+0000","last_undegraded":"2024-03-26T03:00:23.870069+0000","last_fullsized":"2024-03-26T03:00:23.870069+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.349779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:26.426188+0000","last_change":"2024-03-26T03:00:26.426212+0000","last_active":"2024-03-26T03:00:26.426188+0000","last_peered":"2024-03-26T03:00:26.426188+0000","last_clean":"2024-03-26T03:00:26.426188+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:00:26.426188+0000","last_undegraded":"2024-03-26T03:00:26.426188+0000","last_fullsized":"2024-03-26T03:00:26.426188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.426122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.618075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457964+0000","last_change":"2024-03-26T02:59:58.637319+0000","last_active":"2024-03-26T03:00:24.457964+0000","last_peered":"2024-03-26T03:00:24.457964+0000","last_clean":"2024-03-26T03:00:24.457964+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:00:24.457964+0000","last_undegraded":"2024-03-26T03:00:24.457964+0000","last_fullsized":"2024-03-26T03:00:24.457964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:58.637207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.637207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.247095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":132,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080607+0000","last_change":"2024-03-26T02:59:30.881595+0000","last_active":"2024-03-26T03:00:24.080607+0000","last_peered":"2024-03-26T03:00:24.080607+0000","last_clean":"2024-03-26T03:00:24.080607+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:00:24.080607+0000","last_undegraded":"2024-03-26T03:00:24.080607+0000","last_fullsized":"2024-03-26T03:00:24.080607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871610+0000","last_change":"2024-03-26T02:59:02.948314+0000","last_active":"2024-03-26T03:00:23.871610+0000","last_peered":"2024-03-26T03:00:23.871610+0000","last_clean":"2024-03-26T03:00:23.871610+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:00:23.871610+0000","last_undegraded":"2024-03-26T03:00:23.871610+0000","last_fullsized":"2024-03-26T03:00:23.871610+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.533166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457747+0000","last_change":"2024-03-26T02:59:00.941640+0000","last_active":"2024-03-26T03:00:24.457747+0000","last_peered":"2024-03-26T03:00:24.457747+0000","last_clean":"2024-03-26T03:00:24.457747+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:00:24.457747+0000","last_undegraded":"2024-03-26T03:00:24.457747+0000","last_fullsized":"2024-03-26T03:00:24.457747+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.227740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976778+0000","last_change":"2024-03-26T02:59:57.244851+0000","last_active":"2024-03-26T03:00:24.976778+0000","last_peered":"2024-03-26T03:00:24.976778+0000","last_clean":"2024-03-26T03:00:24.976778+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:00:24.976778+0000","last_undegraded":"2024-03-26T03:00:24.976778+0000","last_fullsized":"2024-03-26T03:00:24.976778+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_clean_scrub_stamp":"2024-03-26T02:59:57.244756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.738002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.238941+0000","last_change":"2024-03-26T03:00:27.238978+0000","last_active":"2024-03-26T03:00:27.238941+0000","last_peered":"2024-03-26T03:00:27.238941+0000","last_clean":"2024-03-26T03:00:27.238941+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:00:27.238941+0000","last_undegraded":"2024-03-26T03:00:27.238941+0000","last_fullsized":"2024-03-26T03:00:27.238941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.238859+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:48.977374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057150+0000","last_change":"2024-03-26T02:59:03.206522+0000","last_active":"2024-03-26T03:00:24.057150+0000","last_peered":"2024-03-26T03:00:24.057150+0000","last_clean":"2024-03-26T03:00:24.057150+0000","last_became_active":"2024-03-26T02:59:03.206195+0000","last_became_peered":"2024-03-26T02:59:03.206195+0000","last_unstale":"2024-03-26T03:00:24.057150+0000","last_undegraded":"2024-03-26T03:00:24.057150+0000","last_fullsized":"2024-03-26T03:00:24.057150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.119564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.458006+0000","last_change":"2024-03-26T02:58:56.928192+0000","last_active":"2024-03-26T03:00:24.458006+0000","last_peered":"2024-03-26T03:00:24.458006+0000","last_clean":"2024-03-26T03:00:24.458006+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:00:24.458006+0000","last_undegraded":"2024-03-26T03:00:24.458006+0000","last_fullsized":"2024-03-26T03:00:24.458006+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057028+0000","last_change":"2024-03-26T03:00:23.030834+0000","last_active":"2024-03-26T03:00:24.057028+0000","last_peered":"2024-03-26T03:00:24.057028+0000","last_clean":"2024-03-26T03:00:24.057028+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:00:24.057028+0000","last_undegraded":"2024-03-26T03:00:24.057028+0000","last_fullsized":"2024-03-26T03:00:24.057028+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:00:23.030734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.030734+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.236468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870326+0000","last_change":"2024-03-26T02:59:00.938985+0000","last_active":"2024-03-26T03:00:23.870326+0000","last_peered":"2024-03-26T03:00:23.870326+0000","last_clean":"2024-03-26T03:00:23.870326+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:00:23.870326+0000","last_undegraded":"2024-03-26T03:00:23.870326+0000","last_fullsized":"2024-03-26T03:00:23.870326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103191,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":965072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430235,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28412,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716740,"statfs":{"total":95995035648,"available":95965941760,"internally_reserved":0,"allocated":1429504,"data_stored":964454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":872448,"data_stored":375906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28428,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716724,"statfs":{"total":95995035648,"available":95965925376,"internally_reserved":0,"allocated":1445888,"data_stored":965725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":375143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":374515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":856064,"data_stored":374613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":374164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:00:31.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:00:31.337 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:00:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:00:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[29666]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:00:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:00:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:00:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:00:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:00:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[37920]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:00:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:00:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:31 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:00:31.697 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:31 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:00:31.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:31 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:00:31.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:31 smithi192 ceph-mon[33110]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:00:31.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:31 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:00:31.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:31 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:00:31.987 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:31 smithi192 systemd[1]: Starting Ceph grafana.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:00:32.310 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:00:32.082Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001287078s 2024-03-26T03:00:32.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: 2.e scrub starts 2024-03-26T03:00:32.410 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: 2.e scrub ok 2024-03-26T03:00:32.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: from='client.34646 v1:172.21.15.66:0/3325085077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:32.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:00:32.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:00:32.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: 4.15 scrub starts 2024-03-26T03:00:32.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-mon[33110]: 4.15 scrub ok 2024-03-26T03:00:32.411 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 podman[63935]: 2024-03-26 03:00:32.001114499 +0000 UTC m=+0.018393489 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-26T03:00:32.411 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 podman[63935]: 2024-03-26 03:00:32.0994744 +0000 UTC m=+0.116753392 container create d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 (image=quay.io/ceph/grafana:9.4.12, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:00:32.411 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 podman[63935]: 2024-03-26 03:00:32.293407027 +0000 UTC m=+0.310686017 container init d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 (image=quay.io/ceph/grafana:9.4.12, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:00:32.411 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 podman[63935]: 2024-03-26 03:00:32.301309349 +0000 UTC m=+0.318588339 container start d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 (image=quay.io/ceph/grafana:9.4.12, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:00:32.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: 2.e scrub starts 2024-03-26T03:00:32.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: 2.e scrub ok 2024-03-26T03:00:32.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: from='client.34646 v1:172.21.15.66:0/3325085077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:32.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: 4.15 scrub starts 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[29666]: 4.15 scrub ok 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: 2.e scrub starts 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: 2.e scrub ok 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: from='client.34646 v1:172.21.15.66:0/3325085077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: 4.15 scrub starts 2024-03-26T03:00:32.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:32 smithi066 ceph-mon[37920]: 4.15 scrub ok 2024-03-26T03:00:32.662 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 bash[63935]: d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 2024-03-26T03:00:32.662 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 systemd[1]: Started Ceph grafana.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:00:32.662 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.457851802Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T03:00:32.662 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458192318Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458210348Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.45821724Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458222684Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458228198Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.45823315Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458238578Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458243743Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458249209Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458254205Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458259148Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458264014Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458268973Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458273704Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458278787Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T03:00:32.663 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458283189Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458287985Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=settings t=2024-03-26T03:00:32.458293047Z level=info msg="App mode production" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=sqlstore t=2024-03-26T03:00:32.458355311Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=sqlstore t=2024-03-26T03:00:32.458375992Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.459714318Z level=info msg="Starting DB migrations" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.459834324Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.460614299Z level=info msg="Executing migration" id="create user table" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.461257005Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.461839818Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.464482705Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.467165299Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T03:00:32.665 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.469868174Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.472503943Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.4751353Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.477939589Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.480569121Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.483137553Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.483669622Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.486546466Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.489355188Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.491737877Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.492112018Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T03:00:32.666 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.49474798Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.497393928Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.499884599Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.501262592Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.503836766Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.506505661Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.509045983Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.511617675Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.514235082Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.516709138Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.519212299Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.52174722Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.524208825Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.52677615Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.529371882Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T03:00:32.667 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.53198632Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.53264337Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.533407999Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.533835002Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.534260505Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.535073794Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.537264717Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.540150495Z level=info msg="Executing migration" id="create star table" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.542508987Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.542976742Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.54574698Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T03:00:32.668 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.548426041Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T03:00:32.669 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.548873822Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T03:00:32.669 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.551580072Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T03:00:32.669 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.554295187Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.556955612Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.557309239Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.560101767Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.560505225Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.562495445Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.565229115Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.565761113Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T03:00:32.670 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.568345712Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.570960643Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.573748049Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.574822804Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.577380608Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.580226039Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.58097736Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.581451491Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.582237141Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.582536509Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T03:00:32.671 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.583024886Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.583509606Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.583957718Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.584374056Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.584834843Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.585249729Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.585640796Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.586463553Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.586967742Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.587540458Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.588046271Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.588599002Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.589115248Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.590031545Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T03:00:32.672 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.590947879Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.591489512Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.592362067Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.593028657Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.593441109Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.594570258Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.59501962Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.595995944Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.596690271Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.597566103Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.598075851Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.598708788Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.599519794Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.600404669Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.60131812Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T03:00:32.673 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.601992702Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.602830643Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.605606225Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.608080108Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.608541226Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.611091113Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.611538221Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.613863737Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.616466951Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.616932745Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.619792435Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.622511208Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.62527967Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.625913141Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.628728196Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T03:00:32.674 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.631439339Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.634133279Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.634475914Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.637107615Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.637704547Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.639953115Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.640414596Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T03:00:32.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.642707795Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T03:00:32.676 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.643164877Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T03:00:32.676 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.645896706Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T03:00:32.676 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.648669615Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T03:00:32.676 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.651350881Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T03:00:32.677 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.653903021Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T03:00:32.677 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.656365259Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T03:00:32.677 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.659198988Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T03:00:32.677 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.661719295Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T03:00:32.677 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.664227155Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T03:00:32.913 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.666918155Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.669477631Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.672118936Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.674710345Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.6773706Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.67816735Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.678934845Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.679788812Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.680548356Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.681138817Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.681742462Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T03:00:32.914 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.682709741Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.683437935Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.684337896Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.685216191Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.686050282Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.686785099Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.687686308Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.688192329Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.688771553Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.689554473Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.690516669Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.691240861Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.692197301Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.693073678Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T03:00:32.915 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.693817494Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.69478204Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.695571314Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.69641346Z level=info msg="Executing migration" id="create session table" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.697301783Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.698226045Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.699090712Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.699840626Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.700767618Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.701482528Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.702422329Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.703328871Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T03:00:32.916 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.704085901Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T03:00:32.917 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.705047052Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.705783751Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.708649942Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.711142604Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.714041602Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.716661592Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.719288263Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.721959417Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T03:00:32.918 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.724699642Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.727197626Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.729740819Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.732424482Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.735136159Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.737772542Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.740551667Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.741048169Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.742777945Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.743590111Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.74435442Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.745237136Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T03:00:32.919 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.745977247Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.74679311Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.7475377Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.748322286Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.749153407Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.749932253Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.750888711Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.751571028Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.752339448Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.752776203Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.753198773Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.754002256Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.754823606Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.755722375Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.756592605Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T03:00:32.920 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.757382265Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.758268424Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.758941711Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.759734211Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.760558219Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.761362095Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.7622322Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.762914846Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.763872656Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.764599563Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.767329023Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.770157819Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.77282661Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.775532212Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T03:00:32.921 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.778390129Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.780725726Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.783586369Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.786359931Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.789084317Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.79153935Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.794246384Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.797015146Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.799772745Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.802361898Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.805119028Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.807994185Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.810641745Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.812993712Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T03:00:32.922 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.815557771Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.817940539Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.820652259Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.823082934Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.82557114Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.827998351Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.830528457Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.833029156Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.835505276Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.83788373Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.840520929Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.842877519Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.845458974Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.847859731Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T03:00:32.923 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.850221069Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.852740439Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.855199904Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.85589458Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.856651839Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.857543998Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.858294416Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.859043147Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.859830198Z level=info msg="Executing migration" id="create team table" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.860623488Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.861413199Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.862201066Z level=info msg="Executing migration" id="create team member table" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.863115995Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.863830467Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T03:00:32.924 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.864583071Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.865394256Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.866241856Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.867067473Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.867908679Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.868741017Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.869613741Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.870290312Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.871215628Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.871902644Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.872825454Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.873476834Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.873877647Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.874681847Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.875601306Z level=info msg="Executing migration" id="create tag table" 2024-03-26T03:00:32.925 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.876268038Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.876728941Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.87753842Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.878317297Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.879091514Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.881482146Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.882361158Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.883272896Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.883899018Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.884254504Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.887161211Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.889612766Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.892254643Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.893172618Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T03:00:32.926 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.895741788Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.898401005Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.901244818Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.901678359Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.904156985Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.906947832Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.910015404Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.910472398Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.913216282Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T03:00:32.927 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.915823328Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T03:00:33.159 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:00:33.162 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:00:33.163 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.918411368Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.919498392Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.921907291Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.924631784Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.927118235Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.927590888Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.930168253Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.930500833Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.932680265Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.935103985Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.935570169Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.938044324Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.938515478Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.940456471Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T03:00:33.164 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.941000411Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T03:00:33.165 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.943216227Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T03:00:33.165 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.9441839Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T03:00:33.165 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.946858106Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.949645646Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.952452845Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.953032953Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.95583961Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.958358689Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.961165881Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.961639096Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.964217419Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.964713219Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T03:00:33.166 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.967108658Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.969628135Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.970078765Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.974543512Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.978887956Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.981500852Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.981956847Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.984668818Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.987509252Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.990227605Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.990770614Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.993422009Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.995941487Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:32 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:32.998780639Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T03:00:33.167 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.001259148Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.004004138Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.006457636Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.006921584Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.009496419Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.012278734Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.01306355Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.014004668Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.014914631Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.01570922Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.01664427Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.017152033Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T03:00:33.168 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.017988646Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.018739068Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.019738135Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.020691764Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.021669726Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.022656959Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.02364059Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T03:00:33.169 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.023960961Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.024895796Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.025984283Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.026780721Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.027819244Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.02880901Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.029869182Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.030749295Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.031255157Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.032285929Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T03:00:33.170 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.033139542Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.034151017Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.034747814Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.035605966Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.036552347Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.037448982Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.038043203Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.038125149Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.038999787Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.039731413Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.040684945Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.040974274Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T03:00:33.171 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.041094963Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.041506604Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.042527088Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.045128806Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.047901587Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.048374653Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.05121144Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.053896093Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.05651078Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.056970018Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.059638011Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.065161274Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.067940291Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.070757102Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.076568167Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T03:00:33.172 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.082217438Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.084859511Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.085321251Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.088028304Z level=info msg="Executing migration" id="create permission table" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.090774546Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.093551405Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.094059981Z level=info msg="Executing migration" id="create role table" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.096770728Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.099325093Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.101900725Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T03:00:33.173 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.102368712Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.105070323Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.107773121Z level=info msg="Executing migration" id="create team role table" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.110199725Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.110782695Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.113450815Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.116061442Z level=info msg="Executing migration" id="create user role table" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.118773358Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.119246451Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.12187291Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.124622336Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.127207558Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.127706084Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T03:00:33.174 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.128481415Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.129658621Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.130488971Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.130939521Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.131774015Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.132711022Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.133442558Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.133930472Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T03:00:33.175 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.135111774Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.13594625Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.136814354Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.13758631Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.138426189Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.139217957Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.140034084Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.140371622Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.141164948Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.141615686Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.142077717Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.143449081Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.143774216Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T03:00:33.176 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.144647975Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.145494864Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.146368591Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.147709503Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.148453649Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.149272271Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.149389986Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.150251364Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.150350149Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.150643374Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.153193774Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.153893625Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.15574105Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.158352445Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.158813616Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T03:00:33.177 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.161192666Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T03:00:33.178 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.163646481Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T03:00:33.178 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.166165163Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T03:00:33.178 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.1666486Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T03:00:33.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:00:33.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:00:33.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:00:33.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 3.1 deep-scrub starts 2024-03-26T03:00:33.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 3.1 deep-scrub ok 2024-03-26T03:00:33.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 5.14 scrub starts 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: 5.14 scrub ok 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.169080742Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.171494273Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.175254398Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.176424604Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.179147884Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.179755512Z level=info msg="Executing migration" id="add share column" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.18092092Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.181394135Z level=info msg="Executing migration" id="create file table" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.182108707Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.182887083Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.183638364Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T03:00:33.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.184054133Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.18482179Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.185913394Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.186366518Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.186667885Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.187077239Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.188266957Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.188576426Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.18941468Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.19017271Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.190492Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.191250385Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.192446714Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.193628532Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T03:00:33.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.194528738Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.207315746Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.208239489Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.208879863Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.213269441Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.213585144Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.214433284Z level=info msg="Executing migration" id="create folder table" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.215085918Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.215561894Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.21633488Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.216636262Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=migrator t=2024-03-26T03:00:33.217430434Z level=info msg="migrations completed" performed=485 skipped=0 duration=757.623703ms 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=sqlstore t=2024-03-26T03:00:33.21782187Z level=info msg="Created default organization" 2024-03-26T03:00:33.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=plugin.finder t=2024-03-26T03:00:33.251016536Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-26T03:00:33.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=secrets t=2024-03-26T03:00:33.251122352Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T03:00:33.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=query_data t=2024-03-26T03:00:33.253225046Z level=info msg="Query Service initialization" 2024-03-26T03:00:33.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=live.push_http t=2024-03-26T03:00:33.255676376Z level=info msg="Live Push Gateway initialization" 2024-03-26T03:00:33.551 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:33 smithi066 systemd[1]: Starting Ceph node-exporter.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:00:33.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 3.1 deep-scrub starts 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 3.1 deep-scrub ok 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 5.14 scrub starts 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: 5.14 scrub ok 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 3.1 deep-scrub starts 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 3.1 deep-scrub ok 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 5.14 scrub starts 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: 5.14 scrub ok 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:33.993 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-26T03:00:31.923151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1305,"num_read_kb":1119,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":823,"ondisk_log_size":823,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236988,"kb_used_data":8380,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724228,"statfs":{"total":767960285184,"available":767717609472,"internally_reserved":0,"allocated":8581120,"data_stored":4769592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002506"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869732+0000","last_change":"2024-03-26T02:59:03.208828+0000","last_active":"2024-03-26T03:00:23.869732+0000","last_peered":"2024-03-26T03:00:23.869732+0000","last_clean":"2024-03-26T03:00:23.869732+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:00:23.869732+0000","last_undegraded":"2024-03-26T03:00:23.869732+0000","last_fullsized":"2024-03-26T03:00:23.869732+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.769121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058108+0000","last_change":"2024-03-26T02:59:00.943722+0000","last_active":"2024-03-26T03:00:24.058108+0000","last_peered":"2024-03-26T03:00:24.058108+0000","last_clean":"2024-03-26T03:00:24.058108+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:00:24.058108+0000","last_undegraded":"2024-03-26T03:00:24.058108+0000","last_fullsized":"2024-03-26T03:00:24.058108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.322632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871730+0000","last_change":"2024-03-26T03:00:22.240686+0000","last_active":"2024-03-26T03:00:23.871730+0000","last_peered":"2024-03-26T03:00:23.871730+0000","last_clean":"2024-03-26T03:00:23.871730+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:00:23.871730+0000","last_undegraded":"2024-03-26T03:00:23.871730+0000","last_fullsized":"2024-03-26T03:00:23.871730+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:22.240595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:22.240595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.476581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":135,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869622+0000","last_change":"2024-03-26T02:58:58.929759+0000","last_active":"2024-03-26T03:00:23.869622+0000","last_peered":"2024-03-26T03:00:23.869622+0000","last_clean":"2024-03-26T03:00:23.869622+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:00:23.869622+0000","last_undegraded":"2024-03-26T03:00:23.869622+0000","last_fullsized":"2024-03-26T03:00:23.869622+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058296+0000","last_change":"2024-03-26T02:59:02.945015+0000","last_active":"2024-03-26T03:00:24.058296+0000","last_peered":"2024-03-26T03:00:24.058296+0000","last_clean":"2024-03-26T03:00:24.058296+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:00:24.058296+0000","last_undegraded":"2024-03-26T03:00:24.058296+0000","last_fullsized":"2024-03-26T03:00:24.058296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977196+0000","last_change":"2024-03-26T02:59:00.932288+0000","last_active":"2024-03-26T03:00:24.977196+0000","last_peered":"2024-03-26T03:00:24.977196+0000","last_clean":"2024-03-26T03:00:24.977196+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:00:24.977196+0000","last_undegraded":"2024-03-26T03:00:24.977196+0000","last_fullsized":"2024-03-26T03:00:24.977196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.996531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870222+0000","last_change":"2024-03-26T02:58:56.920528+0000","last_active":"2024-03-26T03:00:23.870222+0000","last_peered":"2024-03-26T03:00:23.870222+0000","last_clean":"2024-03-26T03:00:23.870222+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:00:23.870222+0000","last_undegraded":"2024-03-26T03:00:23.870222+0000","last_fullsized":"2024-03-26T03:00:23.870222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058324+0000","last_change":"2024-03-26T02:58:58.927137+0000","last_active":"2024-03-26T03:00:24.058324+0000","last_peered":"2024-03-26T03:00:24.058324+0000","last_clean":"2024-03-26T03:00:24.058324+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:00:24.058324+0000","last_undegraded":"2024-03-26T03:00:24.058324+0000","last_fullsized":"2024-03-26T03:00:24.058324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.999396+0000","last_change":"2024-03-26T03:00:25.999429+0000","last_active":"2024-03-26T03:00:25.999396+0000","last_peered":"2024-03-26T03:00:25.999396+0000","last_clean":"2024-03-26T03:00:25.999396+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:00:25.999396+0000","last_undegraded":"2024-03-26T03:00:25.999396+0000","last_fullsized":"2024-03-26T03:00:25.999396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.999320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.999320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:43.465119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870420+0000","last_change":"2024-03-26T03:00:18.269455+0000","last_active":"2024-03-26T03:00:23.870420+0000","last_peered":"2024-03-26T03:00:23.870420+0000","last_clean":"2024-03-26T03:00:23.870420+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:00:23.870420+0000","last_undegraded":"2024-03-26T03:00:23.870420+0000","last_fullsized":"2024-03-26T03:00:23.870420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.269349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.269349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:35.865636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870664+0000","last_change":"2024-03-26T02:58:56.924320+0000","last_active":"2024-03-26T03:00:23.870664+0000","last_peered":"2024-03-26T03:00:23.870664+0000","last_clean":"2024-03-26T03:00:23.870664+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:00:23.870664+0000","last_undegraded":"2024-03-26T03:00:23.870664+0000","last_fullsized":"2024-03-26T03:00:23.870664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058577+0000","last_change":"2024-03-26T02:58:58.931435+0000","last_active":"2024-03-26T03:00:24.058577+0000","last_peered":"2024-03-26T03:00:24.058577+0000","last_clean":"2024-03-26T03:00:24.058577+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:00:24.058577+0000","last_undegraded":"2024-03-26T03:00:24.058577+0000","last_fullsized":"2024-03-26T03:00:24.058577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.301399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871514+0000","last_change":"2024-03-26T02:59:02.947270+0000","last_active":"2024-03-26T03:00:23.871514+0000","last_peered":"2024-03-26T03:00:23.871514+0000","last_clean":"2024-03-26T03:00:23.871514+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:00:23.871514+0000","last_undegraded":"2024-03-26T03:00:23.871514+0000","last_fullsized":"2024-03-26T03:00:23.871514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057315+0000","last_change":"2024-03-26T03:00:05.805098+0000","last_active":"2024-03-26T03:00:24.057315+0000","last_peered":"2024-03-26T03:00:24.057315+0000","last_clean":"2024-03-26T03:00:24.057315+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:00:24.057315+0000","last_undegraded":"2024-03-26T03:00:24.057315+0000","last_fullsized":"2024-03-26T03:00:24.057315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:05.804994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.804994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.354036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870536+0000","last_change":"2024-03-26T02:58:56.922748+0000","last_active":"2024-03-26T03:00:23.870536+0000","last_peered":"2024-03-26T03:00:23.870536+0000","last_clean":"2024-03-26T03:00:23.870536+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:00:23.870536+0000","last_undegraded":"2024-03-26T03:00:23.870536+0000","last_fullsized":"2024-03-26T03:00:23.870536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058544+0000","last_change":"2024-03-26T03:00:06.927566+0000","last_active":"2024-03-26T03:00:24.058544+0000","last_peered":"2024-03-26T03:00:24.058544+0000","last_clean":"2024-03-26T03:00:24.058544+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:00:24.058544+0000","last_undegraded":"2024-03-26T03:00:24.058544+0000","last_fullsized":"2024-03-26T03:00:24.058544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:06.927439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:06.927439+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:07.212337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870295+0000","last_change":"2024-03-26T02:59:03.207222+0000","last_active":"2024-03-26T03:00:23.870295+0000","last_peered":"2024-03-26T03:00:23.870295+0000","last_clean":"2024-03-26T03:00:23.870295+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:00:23.870295+0000","last_undegraded":"2024-03-26T03:00:23.870295+0000","last_fullsized":"2024-03-26T03:00:23.870295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.830165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058594+0000","last_change":"2024-03-26T02:59:00.943118+0000","last_active":"2024-03-26T03:00:24.058594+0000","last_peered":"2024-03-26T03:00:24.058594+0000","last_clean":"2024-03-26T03:00:24.058594+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:00:24.058594+0000","last_undegraded":"2024-03-26T03:00:24.058594+0000","last_fullsized":"2024-03-26T03:00:24.058594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869957+0000","last_change":"2024-03-26T03:00:02.741809+0000","last_active":"2024-03-26T03:00:23.869957+0000","last_peered":"2024-03-26T03:00:23.869957+0000","last_clean":"2024-03-26T03:00:23.869957+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:00:23.869957+0000","last_undegraded":"2024-03-26T03:00:23.869957+0000","last_fullsized":"2024-03-26T03:00:23.869957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.741725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.741725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.926475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.578207+0000","last_change":"2024-03-26T03:00:25.578239+0000","last_active":"2024-03-26T03:00:25.578207+0000","last_peered":"2024-03-26T03:00:25.578207+0000","last_clean":"2024-03-26T03:00:25.578207+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:00:25.578207+0000","last_undegraded":"2024-03-26T03:00:25.578207+0000","last_fullsized":"2024-03-26T03:00:25.578207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:00:25.578139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.578139+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.013114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.177502+0000","last_change":"2024-03-26T03:00:27.177539+0000","last_active":"2024-03-26T03:00:27.177502+0000","last_peered":"2024-03-26T03:00:27.177502+0000","last_clean":"2024-03-26T03:00:27.177502+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:00:27.177502+0000","last_undegraded":"2024-03-26T03:00:27.177502+0000","last_fullsized":"2024-03-26T03:00:27.177502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.177404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.177404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:28.371450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977912+0000","last_change":"2024-03-26T03:00:18.300021+0000","last_active":"2024-03-26T03:00:24.977912+0000","last_peered":"2024-03-26T03:00:24.977912+0000","last_clean":"2024-03-26T03:00:24.977912+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:00:24.977912+0000","last_undegraded":"2024-03-26T03:00:24.977912+0000","last_fullsized":"2024-03-26T03:00:24.977912+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.299882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.299882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.440696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080359+0000","last_change":"2024-03-26T03:00:18.484389+0000","last_active":"2024-03-26T03:00:24.080359+0000","last_peered":"2024-03-26T03:00:24.080359+0000","last_clean":"2024-03-26T03:00:24.080359+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:00:24.080359+0000","last_undegraded":"2024-03-26T03:00:24.080359+0000","last_fullsized":"2024-03-26T03:00:24.080359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.484287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.484287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.739970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.458079+0000","last_change":"2024-03-26T03:00:04.540480+0000","last_active":"2024-03-26T03:00:24.458079+0000","last_peered":"2024-03-26T03:00:24.458079+0000","last_clean":"2024-03-26T03:00:24.458079+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:00:24.458079+0000","last_undegraded":"2024-03-26T03:00:24.458079+0000","last_fullsized":"2024-03-26T03:00:24.458079+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:04.540375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.540375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.104233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977691+0000","last_change":"2024-03-26T02:59:02.938768+0000","last_active":"2024-03-26T03:00:24.977691+0000","last_peered":"2024-03-26T03:00:24.977691+0000","last_clean":"2024-03-26T03:00:24.977691+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:00:24.977691+0000","last_undegraded":"2024-03-26T03:00:24.977691+0000","last_fullsized":"2024-03-26T03:00:24.977691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.885569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872297+0000","last_change":"2024-03-26T03:00:13.118260+0000","last_active":"2024-03-26T03:00:23.872297+0000","last_peered":"2024-03-26T03:00:23.872297+0000","last_clean":"2024-03-26T03:00:23.872297+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:00:23.872297+0000","last_undegraded":"2024-03-26T03:00:23.872297+0000","last_fullsized":"2024-03-26T03:00:23.872297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.118140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.552791+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869795+0000","last_change":"2024-03-26T02:58:56.922782+0000","last_active":"2024-03-26T03:00:23.869795+0000","last_peered":"2024-03-26T03:00:23.869795+0000","last_clean":"2024-03-26T03:00:23.869795+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:00:23.869795+0000","last_undegraded":"2024-03-26T03:00:23.869795+0000","last_fullsized":"2024-03-26T03:00:23.869795+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080898+0000","last_change":"2024-03-26T02:58:58.929170+0000","last_active":"2024-03-26T03:00:24.080898+0000","last_peered":"2024-03-26T03:00:24.080898+0000","last_clean":"2024-03-26T03:00:24.080898+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:00:24.080898+0000","last_undegraded":"2024-03-26T03:00:24.080898+0000","last_fullsized":"2024-03-26T03:00:24.080898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.402094+0000","last_change":"2024-03-26T03:00:25.402119+0000","last_active":"2024-03-26T03:00:25.402094+0000","last_peered":"2024-03-26T03:00:25.402094+0000","last_clean":"2024-03-26T03:00:25.402094+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:00:25.402094+0000","last_undegraded":"2024-03-26T03:00:25.402094+0000","last_fullsized":"2024-03-26T03:00:25.402094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:00:25.402032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.402032+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:40.441905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.586479+0000","last_change":"2024-03-26T03:00:25.586479+0000","last_active":"2024-03-26T03:00:25.586479+0000","last_peered":"2024-03-26T03:00:25.586479+0000","last_clean":"2024-03-26T03:00:25.586479+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:00:25.586479+0000","last_undegraded":"2024-03-26T03:00:25.586479+0000","last_fullsized":"2024-03-26T03:00:25.586479+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:25.586362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.586362+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.794553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.056989+0000","last_change":"2024-03-26T03:00:18.957459+0000","last_active":"2024-03-26T03:00:24.056989+0000","last_peered":"2024-03-26T03:00:24.056989+0000","last_clean":"2024-03-26T03:00:24.056989+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:00:24.056989+0000","last_undegraded":"2024-03-26T03:00:24.056989+0000","last_fullsized":"2024-03-26T03:00:24.056989+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.957349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:36.020872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872241+0000","last_change":"2024-03-26T02:58:58.945932+0000","last_active":"2024-03-26T03:00:23.872241+0000","last_peered":"2024-03-26T03:00:23.872241+0000","last_clean":"2024-03-26T03:00:23.872241+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:00:23.872241+0000","last_undegraded":"2024-03-26T03:00:23.872241+0000","last_fullsized":"2024-03-26T03:00:23.872241+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870307+0000","last_change":"2024-03-26T02:59:03.207173+0000","last_active":"2024-03-26T03:00:23.870307+0000","last_peered":"2024-03-26T03:00:23.870307+0000","last_clean":"2024-03-26T03:00:23.870307+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:00:23.870307+0000","last_undegraded":"2024-03-26T03:00:23.870307+0000","last_fullsized":"2024-03-26T03:00:23.870307+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.434157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870423+0000","last_change":"2024-03-26T03:00:04.453033+0000","last_active":"2024-03-26T03:00:23.870423+0000","last_peered":"2024-03-26T03:00:23.870423+0000","last_clean":"2024-03-26T03:00:23.870423+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:00:23.870423+0000","last_undegraded":"2024-03-26T03:00:23.870423+0000","last_fullsized":"2024-03-26T03:00:23.870423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.452915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.452915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.497308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.081019+0000","last_change":"2024-03-26T02:58:56.925003+0000","last_active":"2024-03-26T03:00:24.081019+0000","last_peered":"2024-03-26T03:00:24.081019+0000","last_clean":"2024-03-26T03:00:24.081019+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:00:24.081019+0000","last_undegraded":"2024-03-26T03:00:24.081019+0000","last_fullsized":"2024-03-26T03:00:24.081019+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"71'17","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870485+0000","last_change":"2024-03-26T02:58:58.945243+0000","last_active":"2024-03-26T03:00:23.870485+0000","last_peered":"2024-03-26T03:00:23.870485+0000","last_clean":"2024-03-26T03:00:23.870485+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:00:23.870485+0000","last_undegraded":"2024-03-26T03:00:23.870485+0000","last_fullsized":"2024-03-26T03:00:23.870485+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870199+0000","last_change":"2024-03-26T02:59:02.941127+0000","last_active":"2024-03-26T03:00:23.870199+0000","last_peered":"2024-03-26T03:00:23.870199+0000","last_clean":"2024-03-26T03:00:23.870199+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:00:23.870199+0000","last_undegraded":"2024-03-26T03:00:23.870199+0000","last_fullsized":"2024-03-26T03:00:23.870199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080061+0000","last_change":"2024-03-26T03:00:21.571415+0000","last_active":"2024-03-26T03:00:24.080061+0000","last_peered":"2024-03-26T03:00:24.080061+0000","last_clean":"2024-03-26T03:00:24.080061+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:00:24.080061+0000","last_undegraded":"2024-03-26T03:00:24.080061+0000","last_fullsized":"2024-03-26T03:00:24.080061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:21.571273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:21.571273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.284803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057744+0000","last_change":"2024-03-26T03:00:02.803641+0000","last_active":"2024-03-26T03:00:24.057744+0000","last_peered":"2024-03-26T03:00:24.057744+0000","last_clean":"2024-03-26T03:00:24.057744+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:00:24.057744+0000","last_undegraded":"2024-03-26T03:00:24.057744+0000","last_fullsized":"2024-03-26T03:00:24.057744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.803511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.803511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.142559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869615+0000","last_change":"2024-03-26T03:00:23.647837+0000","last_active":"2024-03-26T03:00:23.869615+0000","last_peered":"2024-03-26T03:00:23.869615+0000","last_clean":"2024-03-26T03:00:23.869615+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:00:23.869615+0000","last_undegraded":"2024-03-26T03:00:23.869615+0000","last_fullsized":"2024-03-26T03:00:23.869615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:00:23.647732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.647732+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.169412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869543+0000","last_change":"2024-03-26T02:59:02.945112+0000","last_active":"2024-03-26T03:00:23.869543+0000","last_peered":"2024-03-26T03:00:23.869543+0000","last_clean":"2024-03-26T03:00:23.869543+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:00:23.869543+0000","last_undegraded":"2024-03-26T03:00:23.869543+0000","last_fullsized":"2024-03-26T03:00:23.869543+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977410+0000","last_change":"2024-03-26T02:59:00.936834+0000","last_active":"2024-03-26T03:00:24.977410+0000","last_peered":"2024-03-26T03:00:24.977410+0000","last_clean":"2024-03-26T03:00:24.977410+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:00:24.977410+0000","last_undegraded":"2024-03-26T03:00:24.977410+0000","last_fullsized":"2024-03-26T03:00:24.977410+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.104445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976980+0000","last_change":"2024-03-26T03:00:02.224750+0000","last_active":"2024-03-26T03:00:24.976980+0000","last_peered":"2024-03-26T03:00:24.976980+0000","last_clean":"2024-03-26T03:00:24.976980+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:00:24.976980+0000","last_undegraded":"2024-03-26T03:00:24.976980+0000","last_fullsized":"2024-03-26T03:00:24.976980+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.224639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.224639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.778744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058733+0000","last_change":"2024-03-26T02:58:58.925892+0000","last_active":"2024-03-26T03:00:24.058733+0000","last_peered":"2024-03-26T03:00:24.058733+0000","last_clean":"2024-03-26T03:00:24.058733+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:00:24.058733+0000","last_undegraded":"2024-03-26T03:00:24.058733+0000","last_fullsized":"2024-03-26T03:00:24.058733+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871093+0000","last_change":"2024-03-26T02:59:02.948086+0000","last_active":"2024-03-26T03:00:23.871093+0000","last_peered":"2024-03-26T03:00:23.871093+0000","last_clean":"2024-03-26T03:00:23.871093+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:00:23.871093+0000","last_undegraded":"2024-03-26T03:00:23.871093+0000","last_fullsized":"2024-03-26T03:00:23.871093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870163+0000","last_change":"2024-03-26T03:00:17.616155+0000","last_active":"2024-03-26T03:00:23.870163+0000","last_peered":"2024-03-26T03:00:23.870163+0000","last_clean":"2024-03-26T03:00:23.870163+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:00:23.870163+0000","last_undegraded":"2024-03-26T03:00:23.870163+0000","last_fullsized":"2024-03-26T03:00:23.870163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:17.616013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.616013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.986671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080509+0000","last_change":"2024-03-26T03:00:07.455788+0000","last_active":"2024-03-26T03:00:24.080509+0000","last_peered":"2024-03-26T03:00:24.080509+0000","last_clean":"2024-03-26T03:00:24.080509+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:00:24.080509+0000","last_undegraded":"2024-03-26T03:00:24.080509+0000","last_fullsized":"2024-03-26T03:00:24.080509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.455691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.455691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.746322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977255+0000","last_change":"2024-03-26T02:58:58.928475+0000","last_active":"2024-03-26T03:00:24.977255+0000","last_peered":"2024-03-26T03:00:24.977255+0000","last_clean":"2024-03-26T03:00:24.977255+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:00:24.977255+0000","last_undegraded":"2024-03-26T03:00:24.977255+0000","last_fullsized":"2024-03-26T03:00:24.977255+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059183+0000","last_change":"2024-03-26T02:59:02.944972+0000","last_active":"2024-03-26T03:00:24.059183+0000","last_peered":"2024-03-26T03:00:24.059183+0000","last_clean":"2024-03-26T03:00:24.059183+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:00:24.059183+0000","last_undegraded":"2024-03-26T03:00:24.059183+0000","last_fullsized":"2024-03-26T03:00:24.059183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.212361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870128+0000","last_change":"2024-03-26T02:59:00.933699+0000","last_active":"2024-03-26T03:00:23.870128+0000","last_peered":"2024-03-26T03:00:23.870128+0000","last_clean":"2024-03-26T03:00:23.870128+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:00:23.870128+0000","last_undegraded":"2024-03-26T03:00:23.870128+0000","last_fullsized":"2024-03-26T03:00:23.870128+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871925+0000","last_change":"2024-03-26T02:58:56.914857+0000","last_active":"2024-03-26T03:00:23.871925+0000","last_peered":"2024-03-26T03:00:23.871925+0000","last_clean":"2024-03-26T03:00:23.871925+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:00:23.871925+0000","last_undegraded":"2024-03-26T03:00:23.871925+0000","last_fullsized":"2024-03-26T03:00:23.871925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:24.500241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057582+0000","last_change":"2024-03-26T02:58:58.945768+0000","last_active":"2024-03-26T03:00:24.057582+0000","last_peered":"2024-03-26T03:00:24.057582+0000","last_clean":"2024-03-26T03:00:24.057582+0000","last_became_active":"2024-03-26T02:58:58.945477+0000","last_became_peered":"2024-03-26T02:58:58.945477+0000","last_unstale":"2024-03-26T03:00:24.057582+0000","last_undegraded":"2024-03-26T03:00:24.057582+0000","last_fullsized":"2024-03-26T03:00:24.057582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871918+0000","last_change":"2024-03-26T03:00:09.986685+0000","last_active":"2024-03-26T03:00:23.871918+0000","last_peered":"2024-03-26T03:00:23.871918+0000","last_clean":"2024-03-26T03:00:23.871918+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:00:23.871918+0000","last_undegraded":"2024-03-26T03:00:23.871918+0000","last_fullsized":"2024-03-26T03:00:23.871918+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:09.986591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:09.986591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.856847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057559+0000","last_change":"2024-03-26T03:00:10.855859+0000","last_active":"2024-03-26T03:00:24.057559+0000","last_peered":"2024-03-26T03:00:24.057559+0000","last_clean":"2024-03-26T03:00:24.057559+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:00:24.057559+0000","last_undegraded":"2024-03-26T03:00:24.057559+0000","last_fullsized":"2024-03-26T03:00:24.057559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:10.855698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.855698+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.308595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057206+0000","last_change":"2024-03-26T03:00:12.868763+0000","last_active":"2024-03-26T03:00:24.057206+0000","last_peered":"2024-03-26T03:00:24.057206+0000","last_clean":"2024-03-26T03:00:24.057206+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:00:24.057206+0000","last_undegraded":"2024-03-26T03:00:24.057206+0000","last_fullsized":"2024-03-26T03:00:24.057206+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:12.868624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:12.868624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.398619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"71'22","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:26.545271+0000","last_change":"2024-03-26T03:00:26.545331+0000","last_active":"2024-03-26T03:00:26.545271+0000","last_peered":"2024-03-26T03:00:26.545271+0000","last_clean":"2024-03-26T03:00:26.545271+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:00:26.545271+0000","last_undegraded":"2024-03-26T03:00:26.545271+0000","last_fullsized":"2024-03-26T03:00:26.545271+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'22","last_scrub_stamp":"2024-03-26T03:00:26.545179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.545179+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.405420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870590+0000","last_change":"2024-03-26T02:59:03.208607+0000","last_active":"2024-03-26T03:00:23.870590+0000","last_peered":"2024-03-26T03:00:23.870590+0000","last_clean":"2024-03-26T03:00:23.870590+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:00:23.870590+0000","last_undegraded":"2024-03-26T03:00:23.870590+0000","last_fullsized":"2024-03-26T03:00:23.870590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057707+0000","last_change":"2024-03-26T03:00:20.999242+0000","last_active":"2024-03-26T03:00:24.057707+0000","last_peered":"2024-03-26T03:00:24.057707+0000","last_clean":"2024-03-26T03:00:24.057707+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:00:24.057707+0000","last_undegraded":"2024-03-26T03:00:24.057707+0000","last_fullsized":"2024-03-26T03:00:24.057707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:20.999149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.999149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.966830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080165+0000","last_change":"2024-03-26T03:00:04.420607+0000","last_active":"2024-03-26T03:00:24.080165+0000","last_peered":"2024-03-26T03:00:24.080165+0000","last_clean":"2024-03-26T03:00:24.080165+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:00:24.080165+0000","last_undegraded":"2024-03-26T03:00:24.080165+0000","last_fullsized":"2024-03-26T03:00:24.080165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.420504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.420504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.414672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870569+0000","last_change":"2024-03-26T03:00:05.814094+0000","last_active":"2024-03-26T03:00:23.870569+0000","last_peered":"2024-03-26T03:00:23.870569+0000","last_clean":"2024-03-26T03:00:23.870569+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:00:23.870569+0000","last_undegraded":"2024-03-26T03:00:23.870569+0000","last_fullsized":"2024-03-26T03:00:23.870569+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:00:05.813958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.813958+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.910237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457817+0000","last_change":"2024-03-26T02:59:02.942788+0000","last_active":"2024-03-26T03:00:24.457817+0000","last_peered":"2024-03-26T03:00:24.457817+0000","last_clean":"2024-03-26T03:00:24.457817+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:00:24.457817+0000","last_undegraded":"2024-03-26T03:00:24.457817+0000","last_fullsized":"2024-03-26T03:00:24.457817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.084675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870395+0000","last_change":"2024-03-26T03:00:07.801621+0000","last_active":"2024-03-26T03:00:23.870395+0000","last_peered":"2024-03-26T03:00:23.870395+0000","last_clean":"2024-03-26T03:00:23.870395+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:00:23.870395+0000","last_undegraded":"2024-03-26T03:00:23.870395+0000","last_fullsized":"2024-03-26T03:00:23.870395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.801516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.801516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.176058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457497+0000","last_change":"2024-03-26T02:58:56.925054+0000","last_active":"2024-03-26T03:00:24.457497+0000","last_peered":"2024-03-26T03:00:24.457497+0000","last_clean":"2024-03-26T03:00:24.457497+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:00:24.457497+0000","last_undegraded":"2024-03-26T03:00:24.457497+0000","last_fullsized":"2024-03-26T03:00:24.457497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.471451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872132+0000","last_change":"2024-03-26T02:58:58.946140+0000","last_active":"2024-03-26T03:00:23.872132+0000","last_peered":"2024-03-26T03:00:23.872132+0000","last_clean":"2024-03-26T03:00:23.872132+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:00:23.872132+0000","last_undegraded":"2024-03-26T03:00:23.872132+0000","last_fullsized":"2024-03-26T03:00:23.872132+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870275+0000","last_change":"2024-03-26T02:59:02.942511+0000","last_active":"2024-03-26T03:00:23.870275+0000","last_peered":"2024-03-26T03:00:23.870275+0000","last_clean":"2024-03-26T03:00:23.870275+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:00:23.870275+0000","last_undegraded":"2024-03-26T03:00:23.870275+0000","last_fullsized":"2024-03-26T03:00:23.870275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059059+0000","last_change":"2024-03-26T02:58:56.919704+0000","last_active":"2024-03-26T03:00:24.059059+0000","last_peered":"2024-03-26T03:00:24.059059+0000","last_clean":"2024-03-26T03:00:24.059059+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:00:24.059059+0000","last_undegraded":"2024-03-26T03:00:24.059059+0000","last_fullsized":"2024-03-26T03:00:24.059059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.046057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":137,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.142417+0000","last_change":"2024-03-26T03:00:25.142453+0000","last_active":"2024-03-26T03:00:25.142417+0000","last_peered":"2024-03-26T03:00:25.142417+0000","last_clean":"2024-03-26T03:00:25.142417+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:00:25.142417+0000","last_undegraded":"2024-03-26T03:00:25.142417+0000","last_fullsized":"2024-03-26T03:00:25.142417+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:00:25.142334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.142334+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:37.994864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457445+0000","last_change":"2024-03-26T03:00:08.469473+0000","last_active":"2024-03-26T03:00:24.457445+0000","last_peered":"2024-03-26T03:00:24.457445+0000","last_clean":"2024-03-26T03:00:24.457445+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:00:24.457445+0000","last_undegraded":"2024-03-26T03:00:24.457445+0000","last_fullsized":"2024-03-26T03:00:24.457445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:08.469321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.469321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.714692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871178+0000","last_change":"2024-03-26T03:00:11.023432+0000","last_active":"2024-03-26T03:00:23.871178+0000","last_peered":"2024-03-26T03:00:23.871178+0000","last_clean":"2024-03-26T03:00:23.871178+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:00:23.871178+0000","last_undegraded":"2024-03-26T03:00:23.871178+0000","last_fullsized":"2024-03-26T03:00:23.871178+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:11.023351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:11.023351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.872070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.327191+0000","last_change":"2024-03-26T03:00:27.327214+0000","last_active":"2024-03-26T03:00:27.327191+0000","last_peered":"2024-03-26T03:00:27.327191+0000","last_clean":"2024-03-26T03:00:27.327191+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:00:27.327191+0000","last_undegraded":"2024-03-26T03:00:27.327191+0000","last_fullsized":"2024-03-26T03:00:27.327191+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.327116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.327116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.083798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"71'16","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977357+0000","last_change":"2024-03-26T02:58:58.926863+0000","last_active":"2024-03-26T03:00:24.977357+0000","last_peered":"2024-03-26T03:00:24.977357+0000","last_clean":"2024-03-26T03:00:24.977357+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:00:24.977357+0000","last_undegraded":"2024-03-26T03:00:24.977357+0000","last_fullsized":"2024-03-26T03:00:24.977357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872123+0000","last_change":"2024-03-26T03:00:08.968963+0000","last_active":"2024-03-26T03:00:23.872123+0000","last_peered":"2024-03-26T03:00:23.872123+0000","last_clean":"2024-03-26T03:00:23.872123+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:00:23.872123+0000","last_undegraded":"2024-03-26T03:00:23.872123+0000","last_fullsized":"2024-03-26T03:00:23.872123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:08.968848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.968848+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.313221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869815+0000","last_change":"2024-03-26T02:59:02.941781+0000","last_active":"2024-03-26T03:00:23.869815+0000","last_peered":"2024-03-26T03:00:23.869815+0000","last_clean":"2024-03-26T03:00:23.869815+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:00:23.869815+0000","last_undegraded":"2024-03-26T03:00:23.869815+0000","last_fullsized":"2024-03-26T03:00:23.869815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.196474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057273+0000","last_change":"2024-03-26T02:58:56.923680+0000","last_active":"2024-03-26T03:00:24.057273+0000","last_peered":"2024-03-26T03:00:24.057273+0000","last_clean":"2024-03-26T03:00:24.057273+0000","last_became_active":"2024-03-26T02:58:56.923317+0000","last_became_peered":"2024-03-26T02:58:56.923317+0000","last_unstale":"2024-03-26T03:00:24.057273+0000","last_undegraded":"2024-03-26T03:00:24.057273+0000","last_fullsized":"2024-03-26T03:00:24.057273+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:25.601353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":159,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977769+0000","last_change":"2024-03-26T03:00:13.375248+0000","last_active":"2024-03-26T03:00:24.977769+0000","last_peered":"2024-03-26T03:00:24.977769+0000","last_clean":"2024-03-26T03:00:24.977769+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:00:24.977769+0000","last_undegraded":"2024-03-26T03:00:24.977769+0000","last_fullsized":"2024-03-26T03:00:24.977769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.375080+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.080768+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870705+0000","last_change":"2024-03-26T02:59:00.940319+0000","last_active":"2024-03-26T03:00:23.870705+0000","last_peered":"2024-03-26T03:00:23.870705+0000","last_clean":"2024-03-26T03:00:23.870705+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:00:23.870705+0000","last_undegraded":"2024-03-26T03:00:23.870705+0000","last_fullsized":"2024-03-26T03:00:23.870705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870774+0000","last_change":"2024-03-26T02:59:03.208010+0000","last_active":"2024-03-26T03:00:23.870774+0000","last_peered":"2024-03-26T03:00:23.870774+0000","last_clean":"2024-03-26T03:00:23.870774+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:00:23.870774+0000","last_undegraded":"2024-03-26T03:00:23.870774+0000","last_fullsized":"2024-03-26T03:00:23.870774+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":166,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:26.972354+0000","last_change":"2024-03-26T03:00:04.289538+0000","last_active":"2024-03-26T03:00:26.972354+0000","last_peered":"2024-03-26T03:00:26.972354+0000","last_clean":"2024-03-26T03:00:26.972354+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:00:26.972354+0000","last_undegraded":"2024-03-26T03:00:26.972354+0000","last_fullsized":"2024-03-26T03:00:26.972354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:00:04.289452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.289452+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:13.576447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457495+0000","last_change":"2024-03-26T03:00:23.413750+0000","last_active":"2024-03-26T03:00:24.457495+0000","last_peered":"2024-03-26T03:00:24.457495+0000","last_clean":"2024-03-26T03:00:24.457495+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:00:24.457495+0000","last_undegraded":"2024-03-26T03:00:24.457495+0000","last_fullsized":"2024-03-26T03:00:24.457495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:23.413656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.413656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:37.034913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977897+0000","last_change":"2024-03-26T03:00:23.305418+0000","last_active":"2024-03-26T03:00:24.977897+0000","last_peered":"2024-03-26T03:00:24.977897+0000","last_clean":"2024-03-26T03:00:24.977897+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:00:24.977897+0000","last_undegraded":"2024-03-26T03:00:24.977897+0000","last_fullsized":"2024-03-26T03:00:24.977897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:00:23.305307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.305307+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:52.977903+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.404936+0000","last_change":"2024-03-26T03:00:25.404962+0000","last_active":"2024-03-26T03:00:25.404936+0000","last_peered":"2024-03-26T03:00:25.404936+0000","last_clean":"2024-03-26T03:00:25.404936+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:00:25.404936+0000","last_undegraded":"2024-03-26T03:00:25.404936+0000","last_fullsized":"2024-03-26T03:00:25.404936+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.404851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.404851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.989593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058495+0000","last_change":"2024-03-26T03:00:24.058533+0000","last_active":"2024-03-26T03:00:24.058495+0000","last_peered":"2024-03-26T03:00:24.058495+0000","last_clean":"2024-03-26T03:00:24.058495+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:00:24.058495+0000","last_undegraded":"2024-03-26T03:00:24.058495+0000","last_fullsized":"2024-03-26T03:00:24.058495+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:24.058394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:24.058394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.713360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869726+0000","last_change":"2024-03-26T02:59:00.924687+0000","last_active":"2024-03-26T03:00:23.869726+0000","last_peered":"2024-03-26T03:00:23.869726+0000","last_clean":"2024-03-26T03:00:23.869726+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:00:23.869726+0000","last_undegraded":"2024-03-26T03:00:23.869726+0000","last_fullsized":"2024-03-26T03:00:23.869726+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057237+0000","last_change":"2024-03-26T03:00:03.775166+0000","last_active":"2024-03-26T03:00:24.057237+0000","last_peered":"2024-03-26T03:00:24.057237+0000","last_clean":"2024-03-26T03:00:24.057237+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:00:24.057237+0000","last_undegraded":"2024-03-26T03:00:24.057237+0000","last_fullsized":"2024-03-26T03:00:24.057237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:03.775065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:03.775065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:04.410162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":312,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870189+0000","last_change":"2024-03-26T02:59:36.442802+0000","last_active":"2024-03-26T03:00:23.870189+0000","last_peered":"2024-03-26T03:00:23.870189+0000","last_clean":"2024-03-26T03:00:23.870189+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:00:23.870189+0000","last_undegraded":"2024-03-26T03:00:23.870189+0000","last_fullsized":"2024-03-26T03:00:23.870189+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-26T02:59:36.442703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T02:59:36.442703+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:44.641515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:28.559224+0000","last_change":"2024-03-26T03:00:28.559278+0000","last_active":"2024-03-26T03:00:28.559224+0000","last_peered":"2024-03-26T03:00:28.559224+0000","last_clean":"2024-03-26T03:00:28.559224+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:00:28.559224+0000","last_undegraded":"2024-03-26T03:00:28.559224+0000","last_fullsized":"2024-03-26T03:00:28.559224+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:00:28.559147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.559147+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:36.143298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870798+0000","last_change":"2024-03-26T03:00:15.331397+0000","last_active":"2024-03-26T03:00:23.870798+0000","last_peered":"2024-03-26T03:00:23.870798+0000","last_clean":"2024-03-26T03:00:23.870798+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:00:23.870798+0000","last_undegraded":"2024-03-26T03:00:23.870798+0000","last_fullsized":"2024-03-26T03:00:23.870798+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.331210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.331210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.836621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977491+0000","last_change":"2024-03-26T02:59:02.935205+0000","last_active":"2024-03-26T03:00:24.977491+0000","last_peered":"2024-03-26T03:00:24.977491+0000","last_clean":"2024-03-26T03:00:24.977491+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:00:24.977491+0000","last_undegraded":"2024-03-26T03:00:24.977491+0000","last_fullsized":"2024-03-26T03:00:24.977491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871844+0000","last_change":"2024-03-26T02:59:00.929356+0000","last_active":"2024-03-26T03:00:23.871844+0000","last_peered":"2024-03-26T03:00:23.871844+0000","last_clean":"2024-03-26T03:00:23.871844+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:00:23.871844+0000","last_undegraded":"2024-03-26T03:00:23.871844+0000","last_fullsized":"2024-03-26T03:00:23.871844+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870137+0000","last_change":"2024-03-26T02:58:56.926508+0000","last_active":"2024-03-26T03:00:23.870137+0000","last_peered":"2024-03-26T03:00:23.870137+0000","last_clean":"2024-03-26T03:00:23.870137+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:00:23.870137+0000","last_undegraded":"2024-03-26T03:00:23.870137+0000","last_fullsized":"2024-03-26T03:00:23.870137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.872024+0000","last_change":"2024-03-26T02:59:58.957331+0000","last_active":"2024-03-26T03:00:23.872024+0000","last_peered":"2024-03-26T03:00:23.872024+0000","last_clean":"2024-03-26T03:00:23.872024+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:00:23.872024+0000","last_undegraded":"2024-03-26T03:00:23.872024+0000","last_fullsized":"2024-03-26T03:00:23.872024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T02:59:58.957239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.957239+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:59.168727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057996+0000","last_change":"2024-03-26T02:59:00.935022+0000","last_active":"2024-03-26T03:00:24.057996+0000","last_peered":"2024-03-26T03:00:24.057996+0000","last_clean":"2024-03-26T03:00:24.057996+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:00:24.057996+0000","last_undegraded":"2024-03-26T03:00:24.057996+0000","last_fullsized":"2024-03-26T03:00:24.057996+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.032534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977083+0000","last_change":"2024-03-26T03:00:07.343102+0000","last_active":"2024-03-26T03:00:24.977083+0000","last_peered":"2024-03-26T03:00:24.977083+0000","last_clean":"2024-03-26T03:00:24.977083+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:00:24.977083+0000","last_undegraded":"2024-03-26T03:00:24.977083+0000","last_fullsized":"2024-03-26T03:00:24.977083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.343020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.343020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.674172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870779+0000","last_change":"2024-03-26T02:58:56.923089+0000","last_active":"2024-03-26T03:00:23.870779+0000","last_peered":"2024-03-26T03:00:23.870779+0000","last_clean":"2024-03-26T03:00:23.870779+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:00:23.870779+0000","last_undegraded":"2024-03-26T03:00:23.870779+0000","last_fullsized":"2024-03-26T03:00:23.870779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:25.226053+0000","last_change":"2024-03-26T03:00:25.226087+0000","last_active":"2024-03-26T03:00:25.226053+0000","last_peered":"2024-03-26T03:00:25.226053+0000","last_clean":"2024-03-26T03:00:25.226053+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:00:25.226053+0000","last_undegraded":"2024-03-26T03:00:25.226053+0000","last_fullsized":"2024-03-26T03:00:25.226053+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:00:25.225991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.225991+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.215639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080446+0000","last_change":"2024-03-26T03:00:10.472741+0000","last_active":"2024-03-26T03:00:24.080446+0000","last_peered":"2024-03-26T03:00:24.080446+0000","last_clean":"2024-03-26T03:00:24.080446+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:00:24.080446+0000","last_undegraded":"2024-03-26T03:00:24.080446+0000","last_fullsized":"2024-03-26T03:00:24.080446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:10.472642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.472642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.985525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058713+0000","last_change":"2024-03-26T02:59:02.944368+0000","last_active":"2024-03-26T03:00:24.058713+0000","last_peered":"2024-03-26T03:00:24.058713+0000","last_clean":"2024-03-26T03:00:24.058713+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:00:24.058713+0000","last_undegraded":"2024-03-26T03:00:24.058713+0000","last_fullsized":"2024-03-26T03:00:24.058713+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.999221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976932+0000","last_change":"2024-03-26T03:00:17.286899+0000","last_active":"2024-03-26T03:00:24.976932+0000","last_peered":"2024-03-26T03:00:24.976932+0000","last_clean":"2024-03-26T03:00:24.976932+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:00:24.976932+0000","last_undegraded":"2024-03-26T03:00:24.976932+0000","last_fullsized":"2024-03-26T03:00:24.976932+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.286796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.424244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":137,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869503+0000","last_change":"2024-03-26T03:00:08.772624+0000","last_active":"2024-03-26T03:00:23.869503+0000","last_peered":"2024-03-26T03:00:23.869503+0000","last_clean":"2024-03-26T03:00:23.869503+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:00:23.869503+0000","last_undegraded":"2024-03-26T03:00:23.869503+0000","last_fullsized":"2024-03-26T03:00:23.869503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.772486+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.103774+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058396+0000","last_change":"2024-03-26T02:59:00.944090+0000","last_active":"2024-03-26T03:00:24.058396+0000","last_peered":"2024-03-26T03:00:24.058396+0000","last_clean":"2024-03-26T03:00:24.058396+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:00:24.058396+0000","last_undegraded":"2024-03-26T03:00:24.058396+0000","last_fullsized":"2024-03-26T03:00:24.058396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870697+0000","last_change":"2024-03-26T02:59:03.208325+0000","last_active":"2024-03-26T03:00:23.870697+0000","last_peered":"2024-03-26T03:00:23.870697+0000","last_clean":"2024-03-26T03:00:23.870697+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:00:23.870697+0000","last_undegraded":"2024-03-26T03:00:23.870697+0000","last_fullsized":"2024-03-26T03:00:23.870697+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.081018+0000","last_change":"2024-03-26T02:58:56.922909+0000","last_active":"2024-03-26T03:00:24.081018+0000","last_peered":"2024-03-26T03:00:24.081018+0000","last_clean":"2024-03-26T03:00:24.081018+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:00:24.081018+0000","last_undegraded":"2024-03-26T03:00:24.081018+0000","last_fullsized":"2024-03-26T03:00:24.081018+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058262+0000","last_change":"2024-03-26T03:00:07.927770+0000","last_active":"2024-03-26T03:00:24.058262+0000","last_peered":"2024-03-26T03:00:24.058262+0000","last_clean":"2024-03-26T03:00:24.058262+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:00:24.058262+0000","last_undegraded":"2024-03-26T03:00:24.058262+0000","last_fullsized":"2024-03-26T03:00:24.058262+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:00:07.927648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.927648+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:12.893425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"71'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869486+0000","last_change":"2024-03-26T02:59:00.937344+0000","last_active":"2024-03-26T03:00:23.869486+0000","last_peered":"2024-03-26T03:00:23.869486+0000","last_clean":"2024-03-26T03:00:23.869486+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:00:23.869486+0000","last_undegraded":"2024-03-26T03:00:23.869486+0000","last_fullsized":"2024-03-26T03:00:23.869486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.869822+0000","last_change":"2024-03-26T03:00:15.672174+0000","last_active":"2024-03-26T03:00:23.869822+0000","last_peered":"2024-03-26T03:00:23.869822+0000","last_clean":"2024-03-26T03:00:23.869822+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:00:23.869822+0000","last_undegraded":"2024-03-26T03:00:23.869822+0000","last_fullsized":"2024-03-26T03:00:23.869822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.672090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.667763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.990790+0000","last_change":"2024-03-26T03:00:27.990812+0000","last_active":"2024-03-26T03:00:27.990790+0000","last_peered":"2024-03-26T03:00:27.990790+0000","last_clean":"2024-03-26T03:00:27.990790+0000","last_became_active":"2024-03-26T02:59:00.937450+0000","last_became_peered":"2024-03-26T02:59:00.937450+0000","last_unstale":"2024-03-26T03:00:27.990790+0000","last_undegraded":"2024-03-26T03:00:27.990790+0000","last_fullsized":"2024-03-26T03:00:27.990790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.990730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.990730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.775418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870231+0000","last_change":"2024-03-26T02:58:56.926585+0000","last_active":"2024-03-26T03:00:23.870231+0000","last_peered":"2024-03-26T03:00:23.870231+0000","last_clean":"2024-03-26T03:00:23.870231+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:00:23.870231+0000","last_undegraded":"2024-03-26T03:00:23.870231+0000","last_fullsized":"2024-03-26T03:00:23.870231+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059147+0000","last_change":"2024-03-26T03:00:05.879783+0000","last_active":"2024-03-26T03:00:24.059147+0000","last_peered":"2024-03-26T03:00:24.059147+0000","last_clean":"2024-03-26T03:00:24.059147+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:00:24.059147+0000","last_undegraded":"2024-03-26T03:00:24.059147+0000","last_fullsized":"2024-03-26T03:00:24.059147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:05.879672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.879672+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.035707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057412+0000","last_change":"2024-03-26T02:59:02.944645+0000","last_active":"2024-03-26T03:00:24.057412+0000","last_peered":"2024-03-26T03:00:24.057412+0000","last_clean":"2024-03-26T03:00:24.057412+0000","last_became_active":"2024-03-26T02:59:02.944279+0000","last_became_peered":"2024-03-26T02:59:02.944279+0000","last_unstale":"2024-03-26T03:00:24.057412+0000","last_undegraded":"2024-03-26T03:00:24.057412+0000","last_fullsized":"2024-03-26T03:00:24.057412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.965129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.059066+0000","last_change":"2024-03-26T02:59:00.934826+0000","last_active":"2024-03-26T03:00:24.059066+0000","last_peered":"2024-03-26T03:00:24.059066+0000","last_clean":"2024-03-26T03:00:24.059066+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:00:24.059066+0000","last_undegraded":"2024-03-26T03:00:24.059066+0000","last_fullsized":"2024-03-26T03:00:24.059066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976774+0000","last_change":"2024-03-26T02:58:56.930599+0000","last_active":"2024-03-26T03:00:24.976774+0000","last_peered":"2024-03-26T03:00:24.976774+0000","last_clean":"2024-03-26T03:00:24.976774+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:00:24.976774+0000","last_undegraded":"2024-03-26T03:00:24.976774+0000","last_fullsized":"2024-03-26T03:00:24.976774+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:28.370997+0000","last_change":"2024-03-26T03:00:28.371034+0000","last_active":"2024-03-26T03:00:28.370997+0000","last_peered":"2024-03-26T03:00:28.370997+0000","last_clean":"2024-03-26T03:00:28.370997+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:00:28.370997+0000","last_undegraded":"2024-03-26T03:00:28.370997+0000","last_fullsized":"2024-03-26T03:00:28.370997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:28.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.370925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:35.020056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058973+0000","last_change":"2024-03-26T02:59:02.945294+0000","last_active":"2024-03-26T03:00:24.058973+0000","last_peered":"2024-03-26T03:00:24.058973+0000","last_clean":"2024-03-26T03:00:24.058973+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:00:24.058973+0000","last_undegraded":"2024-03-26T03:00:24.058973+0000","last_fullsized":"2024-03-26T03:00:24.058973+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:30.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457739+0000","last_change":"2024-03-26T03:00:20.452771+0000","last_active":"2024-03-26T03:00:24.457739+0000","last_peered":"2024-03-26T03:00:24.457739+0000","last_clean":"2024-03-26T03:00:24.457739+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:00:24.457739+0000","last_undegraded":"2024-03-26T03:00:24.457739+0000","last_fullsized":"2024-03-26T03:00:24.457739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.452648+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.711803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976779+0000","last_change":"2024-03-26T02:58:56.930656+0000","last_active":"2024-03-26T03:00:24.976779+0000","last_peered":"2024-03-26T03:00:24.976779+0000","last_clean":"2024-03-26T03:00:24.976779+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:00:24.976779+0000","last_undegraded":"2024-03-26T03:00:24.976779+0000","last_fullsized":"2024-03-26T03:00:24.976779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058449+0000","last_change":"2024-03-26T03:00:08.975032+0000","last_active":"2024-03-26T03:00:24.058449+0000","last_peered":"2024-03-26T03:00:24.058449+0000","last_clean":"2024-03-26T03:00:24.058449+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:00:24.058449+0000","last_undegraded":"2024-03-26T03:00:24.058449+0000","last_fullsized":"2024-03-26T03:00:24.058449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:00:08.974874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.974874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.309201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457956+0000","last_change":"2024-03-26T02:59:02.942823+0000","last_active":"2024-03-26T03:00:24.457956+0000","last_peered":"2024-03-26T03:00:24.457956+0000","last_clean":"2024-03-26T03:00:24.457956+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:00:24.457956+0000","last_undegraded":"2024-03-26T03:00:24.457956+0000","last_fullsized":"2024-03-26T03:00:24.457956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.346480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.977585+0000","last_change":"2024-03-26T02:59:00.937106+0000","last_active":"2024-03-26T03:00:24.977585+0000","last_peered":"2024-03-26T03:00:24.977585+0000","last_clean":"2024-03-26T03:00:24.977585+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:00:24.977585+0000","last_undegraded":"2024-03-26T03:00:24.977585+0000","last_fullsized":"2024-03-26T03:00:24.977585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870518+0000","last_change":"2024-03-26T03:00:00.526049+0000","last_active":"2024-03-26T03:00:23.870518+0000","last_peered":"2024-03-26T03:00:23.870518+0000","last_clean":"2024-03-26T03:00:23.870518+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:00:23.870518+0000","last_undegraded":"2024-03-26T03:00:23.870518+0000","last_fullsized":"2024-03-26T03:00:23.870518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:00.525932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:00.525932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.138757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.058920+0000","last_change":"2024-03-26T02:58:58.921800+0000","last_active":"2024-03-26T03:00:24.058920+0000","last_peered":"2024-03-26T03:00:24.058920+0000","last_clean":"2024-03-26T03:00:24.058920+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:00:24.058920+0000","last_undegraded":"2024-03-26T03:00:24.058920+0000","last_fullsized":"2024-03-26T03:00:24.058920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870069+0000","last_change":"2024-03-26T02:59:02.945728+0000","last_active":"2024-03-26T03:00:23.870069+0000","last_peered":"2024-03-26T03:00:23.870069+0000","last_clean":"2024-03-26T03:00:23.870069+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:00:23.870069+0000","last_undegraded":"2024-03-26T03:00:23.870069+0000","last_fullsized":"2024-03-26T03:00:23.870069+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.349779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:26.426188+0000","last_change":"2024-03-26T03:00:26.426212+0000","last_active":"2024-03-26T03:00:26.426188+0000","last_peered":"2024-03-26T03:00:26.426188+0000","last_clean":"2024-03-26T03:00:26.426188+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:00:26.426188+0000","last_undegraded":"2024-03-26T03:00:26.426188+0000","last_fullsized":"2024-03-26T03:00:26.426188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.426122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.618075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457964+0000","last_change":"2024-03-26T02:59:58.637319+0000","last_active":"2024-03-26T03:00:24.457964+0000","last_peered":"2024-03-26T03:00:24.457964+0000","last_clean":"2024-03-26T03:00:24.457964+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:00:24.457964+0000","last_undegraded":"2024-03-26T03:00:24.457964+0000","last_fullsized":"2024-03-26T03:00:24.457964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:58.637207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.637207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.247095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":132,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.080607+0000","last_change":"2024-03-26T02:59:30.881595+0000","last_active":"2024-03-26T03:00:24.080607+0000","last_peered":"2024-03-26T03:00:24.080607+0000","last_clean":"2024-03-26T03:00:24.080607+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:00:24.080607+0000","last_undegraded":"2024-03-26T03:00:24.080607+0000","last_fullsized":"2024-03-26T03:00:24.080607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.871610+0000","last_change":"2024-03-26T02:59:02.948314+0000","last_active":"2024-03-26T03:00:23.871610+0000","last_peered":"2024-03-26T03:00:23.871610+0000","last_clean":"2024-03-26T03:00:23.871610+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:00:23.871610+0000","last_undegraded":"2024-03-26T03:00:23.871610+0000","last_fullsized":"2024-03-26T03:00:23.871610+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:26.533166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.457747+0000","last_change":"2024-03-26T02:59:00.941640+0000","last_active":"2024-03-26T03:00:24.457747+0000","last_peered":"2024-03-26T03:00:24.457747+0000","last_clean":"2024-03-26T03:00:24.457747+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:00:24.457747+0000","last_undegraded":"2024-03-26T03:00:24.457747+0000","last_fullsized":"2024-03-26T03:00:24.457747+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:28.227740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.976778+0000","last_change":"2024-03-26T02:59:57.244851+0000","last_active":"2024-03-26T03:00:24.976778+0000","last_peered":"2024-03-26T03:00:24.976778+0000","last_clean":"2024-03-26T03:00:24.976778+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:00:24.976778+0000","last_undegraded":"2024-03-26T03:00:24.976778+0000","last_fullsized":"2024-03-26T03:00:24.976778+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_clean_scrub_stamp":"2024-03-26T02:59:57.244756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.738002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:27.238941+0000","last_change":"2024-03-26T03:00:27.238978+0000","last_active":"2024-03-26T03:00:27.238941+0000","last_peered":"2024-03-26T03:00:27.238941+0000","last_clean":"2024-03-26T03:00:27.238941+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:00:27.238941+0000","last_undegraded":"2024-03-26T03:00:27.238941+0000","last_fullsized":"2024-03-26T03:00:27.238941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.238859+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:48.977374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":52,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057150+0000","last_change":"2024-03-26T02:59:03.206522+0000","last_active":"2024-03-26T03:00:24.057150+0000","last_peered":"2024-03-26T03:00:24.057150+0000","last_clean":"2024-03-26T03:00:24.057150+0000","last_became_active":"2024-03-26T02:59:03.206195+0000","last_became_peered":"2024-03-26T02:59:03.206195+0000","last_unstale":"2024-03-26T03:00:24.057150+0000","last_undegraded":"2024-03-26T03:00:24.057150+0000","last_fullsized":"2024-03-26T03:00:24.057150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:00:27.119564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.458006+0000","last_change":"2024-03-26T02:58:56.928192+0000","last_active":"2024-03-26T03:00:24.458006+0000","last_peered":"2024-03-26T03:00:24.458006+0000","last_clean":"2024-03-26T03:00:24.458006+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:00:24.458006+0000","last_undegraded":"2024-03-26T03:00:24.458006+0000","last_fullsized":"2024-03-26T03:00:24.458006+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:24.057028+0000","last_change":"2024-03-26T03:00:23.030834+0000","last_active":"2024-03-26T03:00:24.057028+0000","last_peered":"2024-03-26T03:00:24.057028+0000","last_clean":"2024-03-26T03:00:24.057028+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:00:24.057028+0000","last_undegraded":"2024-03-26T03:00:24.057028+0000","last_fullsized":"2024-03-26T03:00:24.057028+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:00:23.030734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.030734+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.236468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-26T03:00:23.870326+0000","last_change":"2024-03-26T02:59:00.938985+0000","last_active":"2024-03-26T03:00:23.870326+0000","last_peered":"2024-03-26T03:00:23.870326+0000","last_clean":"2024-03-26T03:00:23.870326+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:00:23.870326+0000","last_undegraded":"2024-03-26T03:00:23.870326+0000","last_fullsized":"2024-03-26T03:00:23.870326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103191,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":965072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430235,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28412,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716740,"statfs":{"total":95995035648,"available":95965941760,"internally_reserved":0,"allocated":1429504,"data_stored":964454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":872448,"data_stored":375906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28428,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716724,"statfs":{"total":95995035648,"available":95965925376,"internally_reserved":0,"allocated":1445888,"data_stored":965725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":375143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":374515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32012,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713140,"statfs":{"total":95995035648,"available":95962255360,"internally_reserved":0,"allocated":856064,"data_stored":374613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":374164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:00:34.001 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T03:00:34.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump 2024-03-26T03:00:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:00:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:00:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=infra.usagestats.collector t=2024-03-26T03:00:33.923282611Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T03:00:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=provisioning.datasources t=2024-03-26T03:00:33.924371247Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T03:00:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=provisioning.datasources t=2024-03-26T03:00:33.935443729Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T03:00:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=provisioning.alerting t=2024-03-26T03:00:33.946373469Z level=info msg="starting to provision alerting" 2024-03-26T03:00:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=provisioning.alerting t=2024-03-26T03:00:33.946389007Z level=info msg="finished to provision alerting" 2024-03-26T03:00:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=grafanaStorageLogger t=2024-03-26T03:00:33.947624717Z level=info msg="storage starting" 2024-03-26T03:00:34.239 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=ngalert.state.manager t=2024-03-26T03:00:33.947824719Z level=info msg="Warming state cache for startup" 2024-03-26T03:00:34.239 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=http.server t=2024-03-26T03:00:33.954569099Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T03:00:34.239 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=ngalert.state.manager t=2024-03-26T03:00:33.969095983Z level=info msg="State cache has been initialized" states=0 duration=21.269695ms 2024-03-26T03:00:34.239 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=ticker t=2024-03-26T03:00:33.969163561Z level=info msg=starting first_tick=2024-03-26T03:00:40Z 2024-03-26T03:00:34.239 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:00:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=ngalert.multiorg.alertmanager t=2024-03-26T03:00:33.969154064Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T03:00:34.271 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:34.527 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[37920]: Deploying daemon node-exporter.a on smithi066 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[37920]: from='client.14778 v1:172.21.15.66:0/3919437946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[29666]: Deploying daemon node-exporter.a on smithi066 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[29666]: from='client.14778 v1:172.21.15.66:0/3919437946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:00:34.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:34 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:00:34.528 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:34 smithi066 bash[73704]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T03:00:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:34 smithi192 ceph-mon[33110]: Deploying daemon node-exporter.a on smithi066 2024-03-26T03:00:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:34 smithi192 ceph-mon[33110]: from='client.14778 v1:172.21.15.66:0/3919437946' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:00:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:34 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:00:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:34 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:00:34.858 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:34 smithi066 bash[73704]: Getting image source signatures 2024-03-26T03:00:34.858 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:34 smithi066 bash[73704]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T03:00:34.858 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:34 smithi066 bash[73704]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T03:00:34.858 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:34 smithi066 bash[73704]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T03:00:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:00:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: 6.11 scrub starts 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[29666]: 6.11 scrub ok 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: 6.11 scrub starts 2024-03-26T03:00:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:35 smithi066 ceph-mon[37920]: 6.11 scrub ok 2024-03-26T03:00:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:00:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:00:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:00:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:00:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:00:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: 6.11 scrub starts 2024-03-26T03:00:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:35 smithi192 ceph-mon[33110]: 6.11 scrub ok 2024-03-26T03:00:36.590 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:36 smithi066 bash[73704]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[29666]: 6.4 scrub starts 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[29666]: 6.4 scrub ok 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[29666]: 6.b scrub starts 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[29666]: 6.b scrub ok 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:00:36.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:00:36.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[37920]: 6.4 scrub starts 2024-03-26T03:00:36.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[37920]: 6.4 scrub ok 2024-03-26T03:00:36.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[37920]: 6.b scrub starts 2024-03-26T03:00:36.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:36 smithi066 ceph-mon[37920]: 6.b scrub ok 2024-03-26T03:00:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:36 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:00:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:36 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:00:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:36 smithi192 ceph-mon[33110]: 6.4 scrub starts 2024-03-26T03:00:36.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:36 smithi192 ceph-mon[33110]: 6.4 scrub ok 2024-03-26T03:00:36.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:36 smithi192 ceph-mon[33110]: 6.b scrub starts 2024-03-26T03:00:36.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:36 smithi192 ceph-mon[33110]: 6.b scrub ok 2024-03-26T03:00:36.858 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:36 smithi066 bash[73704]: Writing manifest to image destination 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:epoch 91 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:fsid eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:created 2024-03-26T02:53:28.887014+0000 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:modified 2024-03-26T03:00:23.866603+0000 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:crush_version 18 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:full_ratio 0.95 2024-03-26T03:00:37.057 INFO:teuthology.orchestra.run.smithi066.stdout:backfillfull_ratio 0.9 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:nearfull_ratio 0.85 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:require_min_compat_client luminous 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:min_compat_client luminous 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:require_osd_release squid 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:stretch_mode_enabled false 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 68 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:max_osd 8 2024-03-26T03:00:37.058 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0 up in weight 1 up_from 8 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6801/1028658447 v1:172.21.15.66:6802/1028658447 exists,up 90c9bf50-0b39-4fd1-9f00-b02800f07bf9 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1 up in weight 1 up_from 13 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6805/3170415936 v1:172.21.15.66:6806/3170415936 exists,up b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2 up in weight 1 up_from 18 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6809/2520642381 v1:172.21.15.66:6810/2520642381 exists,up 76639ea3-00c7-4693-9eb2-3421f234283b 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6813/2366374823 v1:172.21.15.66:6814/2366374823 exists,up 2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4 up in weight 1 up_from 31 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6800/1766416869 v1:172.21.15.192:6801/1766416869 exists,up 4fae794b-e89c-4931-9ed3-0cc32b445883 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5 up in weight 1 up_from 40 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6804/2546913262 v1:172.21.15.192:6805/2546913262 exists,up 04b178c3-9be8-466c-822c-0acfca8a92b0 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6 up in weight 1 up_from 48 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6808/4039268292 v1:172.21.15.192:6809/4039268292 exists,up fc63b52a-e4fe-4396-bf54-2e38696aedcb 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7 up in weight 1 up_from 58 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6812/3100313121 v1:172.21.15.192:6813/3100313121 exists,up e2f4cedd-43af-4261-9f86-c4b72c97bd1b 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:pg_upmap_items 2.3 [7,4] 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:pg_upmap_items 4.9 [1,5] 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:pg_upmap_items 4.a [1,2] 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:pg_upmap_items 4.17 [1,2] 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:6800/2473255846 expires 2024-03-27T02:59:37.893358+0000 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/3359773304 expires 2024-03-27T02:59:37.893358+0000 2024-03-26T03:00:37.059 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/3105618869 expires 2024-03-27T02:59:37.893358+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/1321389422 expires 2024-03-27T02:59:37.893358+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/3068874001 expires 2024-03-27T02:59:37.893358+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/2719981294 expires 2024-03-27T02:54:29.078830+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/858010278 expires 2024-03-27T02:54:29.078830+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:6800/2558488368 expires 2024-03-27T02:54:01.878928+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/2101239378 expires 2024-03-27T02:59:37.893358+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/2641107787 expires 2024-03-27T02:54:01.878928+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/5961410 expires 2024-03-27T02:54:01.878928+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/3720619020 expires 2024-03-27T02:54:01.878928+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:6800/2059693071 expires 2024-03-27T02:54:29.078830+0000 2024-03-26T03:00:37.060 INFO:teuthology.orchestra.run.smithi066.stdout:blocklist 172.21.15.66:0/952321413 expires 2024-03-27T02:54:29.078830+0000 2024-03-26T03:00:37.481 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 podman[73704]: 2024-03-26 03:00:37.190415209 +0000 UTC m=+3.103201214 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T03:00:37.481 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 podman[73704]: 2024-03-26 03:00:37.481076292 +0000 UTC m=+3.393862279 container create 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:00:37.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:37.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 4.1e scrub starts 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 4.1e scrub ok 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 4.11 scrub starts 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 4.11 scrub ok 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3222061899' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:00:37.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 4.1e scrub starts 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 4.1e scrub ok 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 4.11 scrub starts 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 4.11 scrub ok 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:00:37.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3222061899' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T03:00:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:00:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:00:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 4.1e scrub starts 2024-03-26T03:00:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 4.1e scrub ok 2024-03-26T03:00:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 4.11 scrub starts 2024-03-26T03:00:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 4.11 scrub ok 2024-03-26T03:00:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:00:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:00:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3222061899' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 podman[73704]: 2024-03-26 03:00:37.70015285 +0000 UTC m=+3.612938838 container init 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 podman[73704]: 2024-03-26 03:00:37.703109519 +0000 UTC m=+3.615895511 container start 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.707Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.707Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.707Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T03:00:37.784 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T03:00:37.785 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T03:00:37.786 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.708Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T03:00:37.787 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a[73894]: ts=2024-03-26T03:00:37.709Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T03:00:37.788 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 bash[73704]: 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 2024-03-26T03:00:38.108 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:00:37 smithi066 systemd[1]: Started Ceph node-exporter.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:00:38.323 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6801/1028658447 v1:172.21.15.66:6802/1028658447 exists,up 90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'osd.1 up in weight 1 up_from 13 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6805/3170415936 v1:172.21.15.66:6806/3170415936 exists,up b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'osd.2 up in weight 1 up_from 18 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6809/2520642381 v1:172.21.15.66:6810/2520642381 exists,up 76639ea3-00c7-4693-9eb2-3421f234283b', 'osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.66:6813/2366374823 v1:172.21.15.66:6814/2366374823 exists,up 2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'osd.4 up in weight 1 up_from 31 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6800/1766416869 v1:172.21.15.192:6801/1766416869 exists,up 4fae794b-e89c-4931-9ed3-0cc32b445883', 'osd.5 up in weight 1 up_from 40 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6804/2546913262 v1:172.21.15.192:6805/2546913262 exists,up 04b178c3-9be8-466c-822c-0acfca8a92b0', 'osd.6 up in weight 1 up_from 48 up_thru 76 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6808/4039268292 v1:172.21.15.192:6809/4039268292 exists,up fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'osd.7 up in weight 1 up_from 58 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.192:6812/3100313121 v1:172.21.15.192:6813/3100313121 exists,up e2f4cedd-43af-4261-9f86-c4b72c97bd1b'] 2024-03-26T03:00:38.323 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon mon.a config show 2024-03-26T03:00:38.580 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: 6.f scrub starts 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: 6.f scrub ok 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: 5.0 deep-scrub starts 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: 5.0 deep-scrub ok 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: 6.f scrub starts 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: 6.f scrub ok 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: 5.0 deep-scrub starts 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: 5.0 deep-scrub ok 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:38.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: 6.f scrub starts 2024-03-26T03:00:38.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: 6.f scrub ok 2024-03-26T03:00:38.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: 5.0 deep-scrub starts 2024-03-26T03:00:38.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: 5.0 deep-scrub ok 2024-03-26T03:00:38.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:38.682 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:38.966 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:38 smithi192 systemd[1]: Starting Ceph node-exporter.b for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:00:39.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:00:38 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:00:39.238 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:39 smithi192 bash[64218]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T03:00:39.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[29666]: Deploying daemon node-exporter.b on smithi192 2024-03-26T03:00:39.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[29666]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:39.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:00:39.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:00:39.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[37920]: Deploying daemon node-exporter.b on smithi192 2024-03-26T03:00:39.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[37920]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:39.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:00:39.597 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:39 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:00:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:39 smithi192 ceph-mon[33110]: Deploying daemon node-exporter.b on smithi192 2024-03-26T03:00:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:39 smithi192 ceph-mon[33110]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:39 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:00:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:39 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:00:39.738 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:39 smithi192 bash[64218]: Getting image source signatures 2024-03-26T03:00:39.738 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:39 smithi192 bash[64218]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T03:00:39.738 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:39 smithi192 bash[64218]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T03:00:39.738 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:39 smithi192 bash[64218]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T03:00:40.424 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:00:40.424 INFO:teuthology.orchestra.run.smithi066.stdout: "name": "mon.a", 2024-03-26T03:00:40.424 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster": "ceph", 2024-03-26T03:00:40.424 INFO:teuthology.orchestra.run.smithi066.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-26T03:00:40.424 INFO:teuthology.orchestra.run.smithi066.stdout: "admin_socket_mode": "", 2024-03-26T03:00:40.424 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_client_required": "cephx, none", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_cluster_required": "cephx", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_debug": "true", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_service_required": "cephx", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "auth_supported": "", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_aio": "true", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_aio_poll_ms": "250", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_aio_reap_max": "16", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_async_discard_threads": "0", 2024-03-26T03:00:40.425 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_block_size": "4096", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_debug_aio": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_debug_inflight_ios": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_enable_discard": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_flock_retry": "3", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_inject_crash": "0", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_ioring": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_ioring_hipri": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-26T03:00:40.426 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bdev_type": "", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_alloc_size": "1048576", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_allocator": "hybrid", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_buffered_io": "true", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_check_for_zeros": "false", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_compact_log_sync": "false", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-26T03:00:40.427 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_max_log_runway": "4194304", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_max_prefetch": "1048576", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_min_flush_size": "524288", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_min_log_runway": "1048576", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_replay_recovery": "false", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluefs_sync_write": "false", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_allocation_from_file": "true", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_allocator": "hybrid", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-26T03:00:40.428 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_create": "true", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_db_create": "false", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_db_path": "", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_db_size": "0", 2024-03-26T03:00:40.429 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_path": "", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_preallocate_file": "false", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_size": "107374182400", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_wal_create": "false", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_wal_path": "", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_block_wal_size": "100663296", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_bluefs": "true", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-26T03:00:40.430 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_autotune": "true", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_size": "0", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-26T03:00:40.431 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cache_type": "2q", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_clone_cow": "true", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_mode": "none", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_csum_type": "crc32c", 2024-03-26T03:00:40.432 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_freelist": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-26T03:00:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_small_allocations": "0", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_default_buffered_read": "true", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_default_buffered_write": "false", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-26T03:00:40.434 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fail_eio": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_on_mount": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_on_umount": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-26T03:00:40.435 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_ignore_data_csum": "false", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_log_op_age": "5.000000", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_max_alloc_size": "0", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_max_blob_size": "0", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-26T03:00:40.436 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_max_deferred_txc": "32", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_min_alloc_size": "0", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_nid_prealloc": "1024", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_retry_disk_reads": "3", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_rocksdb_cf": "true", 2024-03-26T03:00:40.437 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-26T03:00:40.438 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-26T03:00:40.438 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-26T03:00:40.438 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_spdk_mem": "512", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_tracing": "false", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-26T03:00:40.439 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "bluestore_zero_block_detection": "false", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-26T03:00:40.440 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_cluster_require_signatures": "false", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_cluster_require_version": "2", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_require_signatures": "false", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_require_version": "2", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_service_require_signatures": "false", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_service_require_version": "2", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cephx_sign_messages": "true", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "chdir": "", 2024-03-26T03:00:40.441 INFO:teuthology.orchestra.run.smithi066.stdout: "client_acl_type": "", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_asio_thread_count": "2", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_cache_mid": "0.750000", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_cache_size": "16384", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_caps_release_delay": "5", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_check_pool_perm": "true", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_debug_force_sync_read": "false", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_debug_getattr_caps": "false", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_debug_inject_tick_delay": "0", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_die_on_failed_remount": "false", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_dirsize_rbytes": "true", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_force_lazyio": "false", 2024-03-26T03:00:40.442 INFO:teuthology.orchestra.run.smithi066.stdout: "client_fs": "", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_inject_release_failure": "false", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_max_inline_size": "4096", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_mds_namespace": "", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_metadata": "", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_mount_gid": "-1", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_mount_timeout": "300", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_mount_uid": "-1", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_mountpoint": "/", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_notify_timeout": "10", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_oc": "true", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_oc_max_dirty": "104857600", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-26T03:00:40.443 INFO:teuthology.orchestra.run.smithi066.stdout: "client_oc_max_objects": "1000", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_oc_size": "209715200", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_oc_target_dirty": "8388608", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_permissions": "true", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_quota": "true", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_quota_df": "true", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_readahead_max_bytes": "0", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_readahead_max_periods": "4", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_readahead_min": "131072", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_reconnect_stale": "false", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_shutdown_timeout": "30", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_snapdir": ".snap", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_tick_interval": "1", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_trace": "", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_try_dentry_invalidate": "false", 2024-03-26T03:00:40.444 INFO:teuthology.orchestra.run.smithi066.stdout: "client_use_faked_inos": "false", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "client_use_random_mds": "false", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_graylog": "false", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_graylog_port": "12201", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_monitors": "default=true", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_syslog": "false", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "clog_to_syslog_level": "info", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "-", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_network": "", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_network_interface": "", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "compressor_zlib_isal": "false", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "compressor_zlib_level": "5", 2024-03-26T03:00:40.445 INFO:teuthology.orchestra.run.smithi066.stdout: "compressor_zlib_winsize": "-15", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "compressor_zstd_level": "1", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crimson_alien_op_num_threads": "6", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crimson_seastar_cpu_cores": "", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_location": "", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_location_hook": "", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_location_hook_timeout": "10", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "daemonize": "false", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "daos_pool": "tank", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-26T03:00:40.446 INFO:teuthology.orchestra.run.smithi066.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_alienstore": "0/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_allow_any_pool_priority": "false", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_asok": "1/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_asok_assert_abort": "false", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_asserts_on_shutdown": "true", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_auth": "1/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_bdev": "1/3", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_bluefs": "1/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_bluestore": "1/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_buffer": "0/1", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_ceph_exporter": "1/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_cephfs_mirror": "0/5", 2024-03-26T03:00:40.447 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_cephsqlite": "0/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_client": "0/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_compressor": "1/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_context": "0/1", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_crimson_interrupt": "0/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_crush": "1/1", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_crypto": "1/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_cyanstore": "0/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_deliberately_leak_memory": "false", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_disable_randomized_ping": "false", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_dpdk": "1/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_eventtrace": "1/5", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_filer": "0/1", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_filestore": "1/3", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_finisher": "1/1", 2024-03-26T03:00:40.448 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_fuse": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_heartbeat_testing_span": "0", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_heartbeatmap": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_javaclient": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_journal": "1/3", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_journaler": "0/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_kstore": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_lockdep": "0/1", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mclock": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds_balancer": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds_locker": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds_log": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds_log_expire": "1/5", 2024-03-26T03:00:40.449 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds_migrator": "1/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mds_quiesce": "3/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_memstore": "1/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mgr": "2/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mgrc": "1/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_mon": "20/20", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_monc": "0/10", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_ms": "1/1", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_none": "0/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_objclass": "0/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_objectcacher": "0/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_objecter": "0/1", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_optracker": "0/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_osd": "1/5", 2024-03-26T03:00:40.450 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_paxos": "20/20", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_perfcounter": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_prioritycache": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rados": "0/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rbd": "0/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rbd_mirror": "0/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rbd_pwl": "0/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rbd_replay": "0/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_refs": "0/0", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_reserver": "1/1", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_access": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_datacache": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_dbstore": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_flight": "1/5", 2024-03-26T03:00:40.451 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_notification": "1/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rgw_sync": "1/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_rocksdb": "4/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_backref": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_cache": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_cleaner": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_device": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_epm": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_journal": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_lba": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_odata": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_omap": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_onode": "0/5", 2024-03-26T03:00:40.452 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_t": "0/5", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_seastore_tm": "0/5", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_striper": "0/1", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_test": "0/5", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_throttle": "1/1", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_timer": "0/1", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_tp": "0/5", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "debug_trace": "1/5", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "device_failure_prediction_mode": "none", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "err_to_graylog": "false", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "err_to_journald": "false", 2024-03-26T03:00:40.453 INFO:teuthology.orchestra.run.smithi066.stdout: "err_to_stderr": "true", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "err_to_syslog": "false", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "event_tracing": "false", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "exporter_addr": "0.0.0.0", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "exporter_http_port": "9926", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "exporter_prio_limit": "5", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "exporter_sort_metrics": "true", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "exporter_stats_period": "5", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "fake_statfs_for_testing": "0", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "fatal_signal_handlers": "true", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "filer_max_purge_ops": "10", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "filer_max_truncate_ops": "128", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_apply_finisher_threads": "1", 2024-03-26T03:00:40.454 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_blackhole": "false", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_btrfs_clone_range": "true", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_btrfs_snap": "true", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_caller_concurrency": "10", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_collect_device_partition_information": "true", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_commit_timeout": "600.000000", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_debug_inject_read_err": "false", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_debug_omap_check": "false", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_debug_verify_split": "false", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_dump_file": "", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fadvise": "true", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fail_eio": "true", 2024-03-26T03:00:40.455 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fd_cache_shards": "16", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fd_cache_size": "128", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fiemap": "false", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fiemap_threshold": "4096", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_inject_stall": "0", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_journal_parallel": "false", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_journal_trailing": "false", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_journal_writeahead": "false", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_kill_at": "0", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-26T03:00:40.456 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattrs": "0", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_xattr_value_size": "0", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_merge_threshold": "-10", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_odsync_write": "false", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_omap_backend": "rocksdb", 2024-03-26T03:00:40.457 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_omap_backend_path": "", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_op_thread_timeout": "60", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_op_threads": "2", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_punch_hole": "false", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-26T03:00:40.458 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_queue_max_ops": "50", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_seek_data_hole": "false", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_sloppy_crc": "false", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_splice": "false", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_split_multiple": "2", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_split_rand_factor": "20", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_update_to": "1000", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-26T03:00:40.459 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_enable": "true", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_xfs_extsize": "false", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "filestore_zfs_snap": "false", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "fio_dir": "/tmp/fio", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_allow_other": "true", 2024-03-26T03:00:40.460 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_atomic_o_trunc": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_big_writes": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_debug": "false", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_default_permissions": "false", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_disable_pagecache": "false", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_max_write": "0", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_multithreaded": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_require_active_mds": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_set_user_groups": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_splice_move": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_splice_read": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_splice_write": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "fuse_use_invalidate_cb": "true", 2024-03-26T03:00:40.461 INFO:teuthology.orchestra.run.smithi066.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "gss_target_name": "ceph", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_file": "", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_inject_failure": "0", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_interval": "5", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "host": "smithi066", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-26T03:00:40.462 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "inject_early_sigterm": "false", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "jaeger_agent_port": "6799", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "jaeger_tracing_enable": "false", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_aio": "true", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_align_min_size": "65536", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_block_align": "true", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_block_size": "4096", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_dio": "true", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_discard": "false", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_force_aio": "false", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_ignore_corruption": "false", 2024-03-26T03:00:40.463 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_max_write_bytes": "10485760", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_max_write_entries": "100", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_replay_from": "0", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_write_header_frequency": "0", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journal_zero_on_create": "false", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journaler_prefetch_periods": "10", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journaler_prezero_periods": "5", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "journaler_write_head_interval": "15", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "key": "", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "keyfile": "", 2024-03-26T03:00:40.464 INFO:teuthology.orchestra.run.smithi066.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_backend": "rocksdb", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_default_stripe_size": "65536", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_fsck_on_mount": "false", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_max_bytes": "67108864", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_max_ops": "512", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_nid_prealloc": "1024", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_onode_map_size": "1024", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_sync_submit_transaction": "false", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "kstore_sync_transaction": "false", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "librados_thread_count": "2", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "lockdep": "false", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "lockdep_force_backtrace": "false", 2024-03-26T03:00:40.465 INFO:teuthology.orchestra.run.smithi066.stdout: "log_coarse_timestamps": "true", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_flush_on_exit": "false", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_graylog_host": "127.0.0.1", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_graylog_port": "12201", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_max_new": "1000", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_max_recent": "10000", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_stderr_prefix": "", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_stop_at_utilization": "0.970000", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_to_file": "true", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_to_graylog": "false", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_to_journald": "false", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_to_stderr": "false", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "log_to_syslog": "false", 2024-03-26T03:00:40.466 INFO:teuthology.orchestra.run.smithi066.stdout: "max_rotating_auth_attempts": "10", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_action_on_write_error": "1", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_alternate_name_max": "8192", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_asio_thread_count": "2", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_export_pin": "true", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_fragment_dirs": "true", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_fragment_interval": "5", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_interval": "10", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_max": "-1", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_max_until": "-1", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_merge_size": "50", 2024-03-26T03:00:40.467 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_midchunk": "0.300000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_min_start": "0.200000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_minchunk": "0.001000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_mode": "0", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_need_max": "1.200000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_need_min": "0.800000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_overload_epochs": "2", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_split_bits": "3", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_split_size": "10000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_target_decay": "10.000000", 2024-03-26T03:00:40.468 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_beacon_grace": "15.000000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_beacon_interval": "4.000000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_mid": "0.700000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_quiesce_delay": "0", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_quiesce_sleep": "200", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_quiesce_splitauth": "true", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_quiesce_threshold": "524288", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_release_free_interval": "10", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_reservation": "0.050000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_trim_interval": "1", 2024-03-26T03:00:40.469 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cache_trim_threshold": "262144", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_client_prealloc_inos": "1000", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_connect_bootstrapping": "false", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_damage_table_max_entries": "10000", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_debug_auth_pins": "false", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_debug_frag": "false", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_debug_scatterstat": "false", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_debug_subtrees": "false", 2024-03-26T03:00:40.470 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_decay_halflife": "5.000000", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_default_dir_hash": "2", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_defer_session_stale": "true", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_deny_all_reconnect": "false", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dir_keys_per_op": "16384", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dir_max_commit_size": "10", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dir_max_entries": "0", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dir_prefetch": "true", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dump_cache_on_map": "false", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_early_reply": "true", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_enable_op_tracker": "true", 2024-03-26T03:00:40.471 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_enforce_unique_name": "true", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_export_ephemeral_random": "true", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_health_summarize_threshold": "10", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-26T03:00:40.472 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_inject_health_dummy": "false", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_inject_migrator_session_race": "false", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_join_fs": "", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_journal_format": "1", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_create_at": "0", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_export_at": "0", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_import_at": "0", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_journal_at": "0", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_journal_expire_at": "0", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_journal_replay_at": "0", 2024-03-26T03:00:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_link_at": "0", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_mdstable_at": "0", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_openc_at": "0", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_rename_at": "0", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_shutdown_at": "0", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_event_large_threshold": "524288", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_events_per_segment": "1024", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_max_events": "-1", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_max_segments": "128", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_pause": "false", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_segment_size": "0", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-26T03:00:40.474 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_trim_threshold": "128", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_log_warn_factor": "2.000000", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_caps_per_client": "1048576", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_completed_flushes": "100000", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_completed_requests": "100000", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_export_size": "20971520", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_file_recover": "32", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_purge_files": "64", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_purge_ops": "8192", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_max_snaps_per_dir": "100", 2024-03-26T03:00:40.475 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_metrics_update_interval": "2", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_min_caps_per_client": "100", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_min_caps_working_set": "10000", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_numa_node": "-1", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_op_complaint_time": "30.000000", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_op_history_duration": "600", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_op_history_size": "20", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_op_history_slow_op_size": "20", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_op_log_threshold": "5", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_ping_grace": "15", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_ping_interval": "5", 2024-03-26T03:00:40.476 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_recall_max_caps": "30000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_recall_warning_threshold": "262144", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_replay_interval": "1.000000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_root_ino_gid": "0", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_root_ino_uid": "0", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-26T03:00:40.477 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_shutdown_check": "0", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_skip_ino": "0", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_snap_max_uid": "4294967294", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_snap_min_uid": "0", 2024-03-26T03:00:40.478 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_snap_rstat": "false", 2024-03-26T03:00:40.479 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_standby_replay_damaged": "false", 2024-03-26T03:00:40.479 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_symlink_recovery": "true", 2024-03-26T03:00:40.479 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-26T03:00:40.479 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_thrash_exports": "0", 2024-03-26T03:00:40.479 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_thrash_fragments": "0", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_tick_interval": "5.000000", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_valgrind_exit": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_verify_backtrace": "1", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_verify_scatter": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mds_wipe_sessions": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mempool_debug": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "memstore_device_bytes": "1073741824", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "memstore_page_set": "false", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "memstore_page_size": "65536", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_client_bytes": "134217728", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_client_messages": "512", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-26T03:00:40.480 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_disabled_modules": "", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_max_pg_num_change": "128", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_mds_bytes": "134217728", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_mds_messages": "128", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_mon_bytes": "134217728", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_mon_messages": "128", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_osd_bytes": "536870912", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_osd_messages": "8192", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_pool": "true", 2024-03-26T03:00:40.481 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_standby_modules": "true", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_stats_period": "5", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_stats_threshold": "5", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_tick_period": "2", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_allow_pool_delete": "true", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_allow_pool_size_one": "false", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_auth_validate_all_caps": "true", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_bytes": "104857600", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_directed_command_retry": "2", 2024-03-26T03:00:40.482 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_hunt_parallel": "3", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_log_interval": "1.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_ping_interval": "10.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_client_target_rank": "-1", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_file_level": "debug", 2024-03-26T03:00:40.483 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_file": "true", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_journald": "true", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_compact_on_bootstrap": "false", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_compact_on_start": "false", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_compact_on_trim": "true", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-26T03:00:40.484 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_cpu_threads": "4", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_crush_min_required_version": "hammer", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_daemon_bytes": "419430400", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_data_avail_crit": "5", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_data_avail_warn": "5", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_data_size_warn": "16106127360", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_dump_json": "false", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_dump_transactions": "false", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_extra_checks": "false", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-26T03:00:40.485 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_no_require_reef": "false", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_no_require_squid": "false", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_down_mkfs_grace": "60", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_down_uptime_grace": "60", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_election_default_strategy": "1", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_election_timeout": "5.000000", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_elector_ping_divisor": "2", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_enable_op_tracker": "true", 2024-03-26T03:00:40.486 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_fake_pool_delete": "false", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_force_quorum_join": "false", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_globalid_prealloc": "10000", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_health_detail_to_clog": "true", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_health_log_update_period": "5", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_health_max_detail": "50", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_health_to_clog": "true", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_health_to_clog_interval": "600", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_host": "172.21.15.66:6789/0", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_host_override": "", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_initial_members": "", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-26T03:00:40.487 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_lease": "5.000000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_log_full_interval": "50", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_log_max": "10000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_log_max_summary": "50", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_log_epochs": "500", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_osd": "10000", 2024-03-26T03:00:40.488 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_pg_per_osd": "10000", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_pool_pg_num": "65536", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mds_force_trim_to": "0", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mds_skip_sanity": "false", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_memory_autotune": "true", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_memory_target": "2147483648", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mgr_beacon_grace": "30", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mgr_digest_period": "5", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mgr_inactive_grace": "60", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_min_osdmap_epochs": "50", 2024-03-26T03:00:40.489 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_op_complaint_time": "30", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_op_history_duration": "600", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_op_history_size": "20", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_op_history_slow_op_size": "20", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_op_log_threshold": "5", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_auto_mark_in": "false", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_cache_size": "500", 2024-03-26T03:00:40.490 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_down_out_interval": "600", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_force_trim_to": "0", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-26T03:00:40.491 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_min_down_reporters": "2", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_report_timeout": "900", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-26T03:00:40.492 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pg_stuck_threshold": "60", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_probe_timeout": "2.000000", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_reweight_max_change": "0.050000", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_reweight_max_osds": "4", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-26T03:00:40.493 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_scrub_interval": "86400", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_scrub_max_keys": "100", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_scrub_timeout": "300", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_session_timeout": "300", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_smart_report_timeout": "5", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_stat_smooth_intervals": "6", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-26T03:00:40.494 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_stretch_pool_min_size": "2", 2024-03-26T03:00:40.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:40.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:40.496 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_stretch_pool_size": "4", 2024-03-26T03:00:40.496 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-26T03:00:40.496 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-26T03:00:40.496 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_sync_debug": "false", 2024-03-26T03:00:40.496 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-26T03:00:40.496 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_sync_provider_kill_at": "0", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_sync_requester_kill_at": "0", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_sync_timeout": "60.000000", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_target_pg_per_osd": "100", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_tick_interval": "5", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_timecheck_interval": "300.000000", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_use_min_delay_socket": "false", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_older_version_delay": "604800", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-26T03:00:40.497 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_misplaced": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_older_version": "true", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-26T03:00:40.498 INFO:teuthology.orchestra.run.smithi066.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "monmap": "", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "motr_tracing_enabled": "false", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_op_threads": "3", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_cm": "false", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_device_name": "", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_dscp": "96", 2024-03-26T03:00:40.499 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_local_gid": "", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_port_num": "1", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_sl": "3", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_support_srq": "true", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_rdma_type": "ib", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_async_reap_threshold": "5", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_before_connect": "false", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_ipv4": "true", 2024-03-26T03:00:40.500 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_ipv6": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_msgr1": "true", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_msgr2": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_port_max": "7568", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_port_min": "6800", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_retry_count": "3", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_bind_retry_delay": "5", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_blackhole_client": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_blackhole_mds": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_blackhole_mgr": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_blackhole_mon": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_blackhole_osd": "false", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_client_mode": "crc secure", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_cluster_mode": "crc secure", 2024-03-26T03:00:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_cluster_type": "", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_compress_secure": "false", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_connection_idle_timeout": "900", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_connection_ready_timeout": "10", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_crc_data": "true", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_crc_header": "true", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_die_on_bad_msg": "false", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_die_on_bug": "true", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_die_on_old_message": "true", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_die_on_skipped_message": "false", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_coremask": "0xF", 2024-03-26T03:00:40.502 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_enable_tso": "true", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_hugepages": "", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_lro": "true", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_memory_channel": "4", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_pmd": "", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_port_id": "0", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-26T03:00:40.503 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_dump_on_send": "false", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_initial_backoff": "0.200000", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_inject_delay_max": "1.000000", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_inject_delay_type": "", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_inject_network_congestion": "0", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_inject_socket_failures": "0", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_learn_addr_from_peer": "true", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_max_accept_failures": "4", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_max_backoff": "15.000000", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_mon_client_mode": "secure crc", 2024-03-26T03:00:40.504 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_mon_service_mode": "secure crc", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_osd_compress_min_size": "1024", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_osd_compress_mode": "none", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_pq_min_cost": "65536", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_public_type": "", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_service_mode": "crc secure", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_tcp_listen_backlog": "512", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_tcp_nodelay": "true", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_tcp_rcvbuf": "0", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "ms_type": "async+posix", 2024-03-26T03:00:40.505 INFO:teuthology.orchestra.run.smithi066.stdout: "no_config_file": "false", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_completion_locks_per_session": "32", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_inflight_ops": "1024", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_tick_interval": "5.000000", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objecter_timeout": "10.000000", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objectstore_blackhole": "false", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "openssl_engine_opts": "", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_delay_time": "5.000000", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_hist_halflife": "1000", 2024-03-26T03:00:40.506 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_max_low_ops": "2", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_max_ops": "4", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_agent_slop": "0.020000", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_asio_thread_count": "2", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_async_recovery_min_cost": "100", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_backfill_scan_max": "512", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_backfill_scan_min": "64", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_backoff_on_degraded": "false", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_backoff_on_peering": "false", 2024-03-26T03:00:40.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_backoff_on_unfound": "true", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_beacon_report_interval": "300", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bench_duration": "30", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bench_max_block_size": "67108864", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_blkin_trace_all": "false", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_check_for_log_corruption": "false", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T03:00:40.508 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_class_update_on_start": "true", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_client_message_cap": "256", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_client_message_size_cap": "524288000", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_client_op_priority": "63", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_client_watch_timeout": "30", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_command_max_records": "256", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_compact_on_start": "false", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_crush_update_on_start": "true", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_crush_update_weight_set": "true", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-26T03:00:40.509 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_feed_pullee": "-1", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_misdirected_ops": "false", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_no_acting_change": "false", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_no_purge_strays": "false", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_op_order": "false", 2024-03-26T03:00:40.510 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_shutdown": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_trim_objects": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_verify_snaps": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_keys": "1024", 2024-03-26T03:00:40.511 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_stride": "524288", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_default_notify_timeout": "30", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_delete_sleep": "0.000000", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_discard_disconnected_ops": "true", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_enable_op_tracker": "true", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-26T03:00:40.512 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_fast_info": "true", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_fast_shutdown": "true", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_function_tracing": "false", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_grace": "20", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_interval": "6", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_min_peers": "10", 2024-03-26T03:00:40.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_min_size": "2000", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_stale": "600", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_hit_set_max_size": "100000", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_hit_set_min_size": "1000", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_journal_size": "5120", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_kill_backfill_at": "0", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-26T03:00:40.514 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_map_cache_size": "50", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_map_dedup": "true", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_map_message_max": "40", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_map_share_max_epochs": "40", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_attr_name_len": "100", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_attr_size": "0", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_backfills": "1", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_markdown_count": "5", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_markdown_period": "600", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_object_name_len": "2048", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_object_namespace_len": "256", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_object_size": "134217728", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-26T03:00:40.515 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_pg_blocked_by": "16", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_pg_log_entries": "10000", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_pgls": "1024", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_push_cost": "8388608", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_push_objects": "10", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_scrubs": "3", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_trimming_pgs": "2", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_write_op_reply_len": "64", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_max_write_size": "90", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-26T03:00:40.516 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_profile": "balanced", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-26T03:00:40.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_base": "805306368", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_cache_min": "134217728", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_target": "4294967296", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_target_autotune": "false", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_min_pg_log_entries": "250", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_min_recovery_priority": "0", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mon_report_interval": "5", 2024-03-26T03:00:40.518 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_num_cache_shards": "32", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_num_op_tracker_shard": "32", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_numa_auto_affinity": "true", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_numa_node": "-1", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_numa_prefer_iface": "true", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_objecter_finishers": "1", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_objectstore": "bluestore", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_objectstore_fuse": "false", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_objectstore_tracing": "false", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_complaint_time": "30.000000", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_history_duration": "600", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_history_size": "20", 2024-03-26T03:00:40.519 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_history_slow_op_size": "20", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_log_threshold": "5", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_num_shards": "0", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_num_shards_hdd": "5", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_num_shards_ssd": "8", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_pq_min_cost": "65536", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_queue_cut_off": "high", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_op_thread_timeout": "15", 2024-03-26T03:00:40.520 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_open_classes_on_start": "true", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_os_flags": "0", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_peering_op_priority": "255", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_delete_cost": "1048576", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_delete_priority": "5", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_log_trim_max": "10000", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_log_trim_min": "100", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-26T03:00:40.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_crimson": "false", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-26T03:00:40.522 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_flags": "0", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_min_size": "0", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_pg_num": "32", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_pgp_num": "0", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_read_ratio": "70", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_size": "3", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_default_type": "replicated", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_push_per_object_cost": "1000", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-26T03:00:40.523 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recover_clone_overlap": "true", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_cost": "20971520", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_max_active": "0", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_max_single_start": "1", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_op_priority": "3", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_priority": "5", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-26T03:00:40.524 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_sleep": "0.000000", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_repair_during_recovery": "false", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_requested_scrub_priority": "120", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_auto_repair": "false", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_begin_hour": "0", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_begin_week_day": "0", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_chunk_max": "25", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_chunk_min": "5", 2024-03-26T03:00:40.525 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_cost": "52428800", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_during_recovery": "false", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_end_hour": "0", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_end_week_day": "0", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_event_cost": "4096", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_invalid_stats": "true", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_max_preemptions": "5", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_priority": "5", 2024-03-26T03:00:40.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_scrub_sleep": "0.000000", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_skip_data_digest": "false", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_smart_report_timeout": "5", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_snap_trim_cost": "1048576", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_snap_trim_priority": "5", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-26T03:00:40.527 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_target_transaction_size": "30", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_tracing": "false", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_use_stale_snap": "false", 2024-03-26T03:00:40.528 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "osdc_blkin_trace_all": "false", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_kill_at": "0", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_max_join_drift": "10", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_min": "500", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_min_wait": "0.050000", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_propose_interval": "1.000000", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_service_trim_max": "500", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_service_trim_min": "10", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_stash_full_interval": "25", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_trim_max": "500", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "paxos_trim_min": "250", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "perf": "true", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "pid_file": "", 2024-03-26T03:00:40.529 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "-", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrv": "", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "public_bind_addr": "-", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "public_network": "", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "public_network_interface": "", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "qat_compressor_enabled": "false", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "qat_compressor_session_max_number": "256", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "rados_mon_op_timeout": "0", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "rados_osd_op_timeout": "0", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "rados_tracing": "false", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_atime_update_interval": "60", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-26T03:00:40.530 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_balance_parent_reads": "false", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_balance_snap_reads": "false", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_blkin_trace_all": "false", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache": "true", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_policy": "writearound", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_size": "33554432", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_clone_copy_on_read": "false", 2024-03-26T03:00:40.531 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_compression_hint": "none", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_concurrent_management_ops": "10", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_clone_format": "auto", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_data_pool": "", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_features": "61", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_format": "2", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_map_options": "", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_order": "22", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_pool": "rbd", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_stripe_count": "0", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_default_stripe_unit": "0", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-26T03:00:40.532 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_enable_alloc_hint": "true", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_io_scheduler": "simple", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_order": "24", 2024-03-26T03:00:40.533 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_pool": "", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_journal_splay_width": "4", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_localize_parent_reads": "false", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_localize_snap_reads": "false", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-26T03:00:40.534 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-26T03:00:40.535 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_mtime_update_interval": "60", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_non_blocking_aio": "true", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_op_thread_timeout": "60", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_op_threads": "1", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_parent_cache_enabled": "false", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_plugins": "", 2024-03-26T03:00:40.536 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_bps_burst": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_bps_limit": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_exclude_ops": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_iops_burst": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_iops_limit": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-26T03:00:40.537 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_read_from_replica_policy": "default", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_skip_partial_discard": "true", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-26T03:00:40.538 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_tracing": "false", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_validate_names": "true", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd_validate_pool": "true", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "restapi_base_url": "", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "restapi_log_level": "", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_acl_grants_max_num": "100", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_admin_entry": "admin", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_backend_store": "rados", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_barbican_url": "", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_beast_enable_async": "true", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_counters_cache": "false", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-26T03:00:40.539 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_cache_enabled": "true", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_cache_expiry_interval": "900", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_cache_lru_size": "10000", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_config_store": "rados", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_content_length_compat": "false", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_copy_obj_progress": "true", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_cors_rules_max_num": "100", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_cross_domain_policy": "", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-26T03:00:40.540 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_addr": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_password": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_kmip_username": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_require_ssl": "true", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-26T03:00:40.541 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_addr": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_auth": "token", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_namespace": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_prefix": "", 2024-03-26T03:00:40.542 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_token_file": "", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_curl_buffersize": "524288", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_curl_low_speed_time": "300", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-26T03:00:40.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_d4n_port": "6379", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_log_changes_size": "1000", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_log_num_shards": "128", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_log_window": "30", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-26T03:00:40.544 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-26T03:00:40.545 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dns_name": "", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dns_s3website_name": "", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_dynamic_resharding": "true", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_gc_threads": "true", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_lc_threads": "true", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_ops_log": "false", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_quota_threads": "true", 2024-03-26T03:00:40.546 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_static_website": "false", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enable_usage_log": "false", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_enforce_swift_acls": "true", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_exit_timeout_secs": "120", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_expose_bucket": "false", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_extended_http_attrs": "", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_filter": "none", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_frontends": "beast port=7480", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_max_deferred": "50", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_max_objs": "32", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-26T03:00:40.547 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_gc_processor_period": "3600", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_init_timeout": "300", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_kafka_connection_idle": "300", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-26T03:00:40.548 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_domain": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_password": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_password_path": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_project": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_tenant": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_token": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_token_path": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_admin_user": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_api_version": "2", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_barbican_domain": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_barbican_password": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_barbican_project": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_barbican_user": "", 2024-03-26T03:00:40.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_url": "", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_debug_interval": "-1", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_lock_max_time": "90", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_max_objs": "32", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_max_rules": "1000", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_max_worker": "3", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lc_thread_delay": "0", 2024-03-26T03:00:40.550 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ldap_dnattr": "uid", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ldap_searchfilter": "", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_log_http_headers": "", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_log_object_name_utc": "false", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-26T03:00:40.551 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_attr_name_len": "0", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_attr_size": "0", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_chunk_size": "4194304", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_listing_results": "1000", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_notify_retries": "10", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_put_param_size": "1048576", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_put_size": "5368709120", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_max_slo_entries": "1000", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_md_log_max_shards": "64", 2024-03-26T03:00:40.552 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_mp_lock_max_time": "600", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_max_gc": "300", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-26T03:00:40.553 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_num_async_rados_threads": "32", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_num_control_oids": "8", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_numa_node": "-1", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_objexp_chunk_size": "100", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_objexp_gc_interval": "600", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-26T03:00:40.554 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_op_thread_timeout": "600", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_op_tracing": "false", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_opa_token": "", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_opa_url": "", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_opa_verify_ssl": "true", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ops_log_rados": "false", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_ops_log_socket_path": "", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-26T03:00:40.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_period_push_interval": "2.000000", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_posix_cache_lanes": "3", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_posix_cache_partitions": "3", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_print_continue": "true", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-26T03:00:40.556 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_rados_tracing": "false", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_realm": "", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_realm_id": "", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_region": "", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_request_uri": "", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_reshard_batch_size": "64", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_reshard_max_aio": "128", 2024-03-26T03:00:40.557 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_reshard_num_logs": "16", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_reshard_thread_interval": "600", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_resolve_cname": "false", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_run_sync_thread": "true", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_scheduler_type": "throttler", 2024-03-26T03:00:40.558 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_script_uri": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_service_provider_name": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_client_id": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_client_secret": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_entry": "sts", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_key": "sts", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_min_session_duration": "900", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sts_token_introspection_url": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_account_in_url": "false", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_auth_entry": "auth", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_auth_url": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_custom_header": "", 2024-03-26T03:00:40.559 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_need_stats": "true", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_tenant_name": "", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_token_expiration": "86400", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_url": "", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_url_prefix": "swift", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_lease_period": "120", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-26T03:00:40.560 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_thread_pool_size": "512", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_comment": "", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_createby": "", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_encoding": "", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_flag": "false", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_origin": "", 2024-03-26T03:00:40.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_torrent_tracker": "", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_trust_forwarded_https": "false", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_usage_max_shards": "32", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_usage_max_user_shards": "1", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_use_opa_authz": "false", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_counters_cache": "false", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_max_buckets": "1000", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_policies_max_num": "100", 2024-03-26T03:00:40.562 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_user_unique_email": "true", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_verify_ssl": "true", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_zone": "", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_zone_id": "", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_zonegroup": "", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_zonegroup_id": "", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgwlc_auto_session_clear": "true", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-26T03:00:40.563 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_block_size": "4096", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cache_size": "536870912", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-26T03:00:40.564 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_index_type": "binary_search", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_partition_filters": "false", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_perf": "false", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "run_dir": "/var/run/ceph", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_block_create": "true", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_cache_lru_size": "67108864", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_default_max_object_size": "16777216", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-26T03:00:40.565 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_device_size": "53687091200", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_journal_batch_capacity": "16", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_main_device_type": "SSD", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "seastore_segment_size": "67108864", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "set_keepcaps": "false", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "setgroup": "ceph", 2024-03-26T03:00:40.566 INFO:teuthology.orchestra.run.smithi066.stdout: "setuser": "ceph", 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: "setuser_match_path": "", 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: "thp": "false", 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: "threadpool_default_timeout": "60", 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout: "throttler_perf_counter": "true" 2024-03-26T03:00:40.567 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:00:40.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:40.987 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:40 smithi192 bash[64218]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T03:00:41.334 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:41 smithi192 bash[64218]: Writing manifest to image destination 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 5.a scrub starts 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 5.a scrub ok 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 4.16 scrub starts 2024-03-26T03:00:41.592 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[37920]: 4.16 scrub ok 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 5.a scrub starts 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 5.a scrub ok 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 4.16 scrub starts 2024-03-26T03:00:41.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:41 smithi066 ceph-mon[29666]: 4.16 scrub ok 2024-03-26T03:00:41.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-26T03:00:41.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:41.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:00:41.650 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:00:41.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:00:41.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:00:41.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 5.a scrub starts 2024-03-26T03:00:41.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 5.a scrub ok 2024-03-26T03:00:41.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 4.16 scrub starts 2024-03-26T03:00:41.651 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:41 smithi192 ceph-mon[33110]: 4.16 scrub ok 2024-03-26T03:00:41.874 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 6.d deep-scrub starts 2024-03-26T03:00:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 6.d deep-scrub ok 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 6.5 deep-scrub starts 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 6.5 deep-scrub ok 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 2.1e scrub starts 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: 2.1e scrub ok 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[29666]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 6.d deep-scrub starts 2024-03-26T03:00:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 6.d deep-scrub ok 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 6.5 deep-scrub starts 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 6.5 deep-scrub ok 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 2.1e scrub starts 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: 2.1e scrub ok 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:42 smithi066 ceph-mon[37920]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 6.d deep-scrub starts 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 6.d deep-scrub ok 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 6.5 deep-scrub starts 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 6.5 deep-scrub ok 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 2.1e scrub starts 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: 2.1e scrub ok 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:42 smithi192 ceph-mon[33110]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:43.237 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:42 smithi192 podman[64218]: 2024-03-26 03:00:42.70321861 +0000 UTC m=+3.738672010 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T03:00:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-mon[29666]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T03:00:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-mon[29666]: 4.2 deep-scrub starts 2024-03-26T03:00:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-mon[29666]: 4.2 deep-scrub ok 2024-03-26T03:00:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-mon[37920]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T03:00:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-mon[37920]: 4.2 deep-scrub starts 2024-03-26T03:00:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-mon[37920]: 4.2 deep-scrub ok 2024-03-26T03:00:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:43 smithi192 ceph-mon[33110]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T03:00:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:43 smithi192 ceph-mon[33110]: 4.2 deep-scrub starts 2024-03-26T03:00:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:43 smithi192 ceph-mon[33110]: 4.2 deep-scrub ok 2024-03-26T03:00:43.796 INFO:teuthology.orchestra.run.smithi066.stdout:mon.a: {} 2024-03-26T03:00:43.807 INFO:teuthology.orchestra.run.smithi066.stdout:mon.b: {} 2024-03-26T03:00:43.817 INFO:teuthology.orchestra.run.smithi066.stdout:mon.c: {} 2024-03-26T03:00:43.818 INFO:teuthology.orchestra.run.smithi066.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:00:43.818 INFO:teuthology.orchestra.run.smithi066.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:00:43.818 INFO:teuthology.orchestra.run.smithi066.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:00:44.022 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:43 smithi192 podman[64218]: 2024-03-26 03:00:43.744604786 +0000 UTC m=+4.780058228 container create b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:00:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:00:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 podman[64218]: 2024-03-26 03:00:44.022275417 +0000 UTC m=+5.057728817 container init b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 podman[64218]: 2024-03-26 03:00:44.025258255 +0000 UTC m=+5.060711656 container start b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.029Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.029Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.030Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.030Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.030Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.030Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.030Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T03:00:44.356 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T03:00:44.357 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T03:00:44.358 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T03:00:44.359 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b[64268]: ts=2024-03-26T03:00:44.031Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 bash[64218]: b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec 2024-03-26T03:00:44.360 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:00:44 smithi192 systemd[1]: Started Ceph node-exporter.b for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4136558293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.525 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4136558293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.526 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:44 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.554 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-26T03:00:44.564 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:00:44.568 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:00:44.568 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-26T03:00:44.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-26T03:00:44.569 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-26T03:00:44.569 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:00:44.569 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:00:44.569 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T03:00:44.569 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:00:44.570 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-26T03:00:44.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:44.570 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-26T03:00:44.570 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:00:44.571 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-26T03:00:44.571 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:00:44.571 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-26T03:00:44.571 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:00:44.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:44.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:44.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4136558293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:00:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:00:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:44 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:44.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:44.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:44.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:44.882 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:44.889 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:44.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:44.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:45.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:45.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:45.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:45.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:45.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:45.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.400 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:45.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T03:00:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:45 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:00:45.502 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.530 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:45.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:00:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:00:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:00:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:00:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T03:00:45.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:00:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:00:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:00:45.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:00:45.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:00:45.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T03:00:45.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:45 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:00:45.632 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:45.654 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:45.662 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:45.664 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:45.665 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:45.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:45.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:45.797 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:45.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:45.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:45.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:46.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:46.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:46.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:46.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:46.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:46.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:46.401 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.430 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:46.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:46.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:46.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: 2.13 scrub starts 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: 2.13 scrub ok 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: 5.c deep-scrub starts 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: 5.c deep-scrub ok 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: 3.2 scrub starts 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: 3.2 scrub ok 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[29666]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: 2.13 scrub starts 2024-03-26T03:00:46.589 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: 2.13 scrub ok 2024-03-26T03:00:46.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: 5.c deep-scrub starts 2024-03-26T03:00:46.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: 5.c deep-scrub ok 2024-03-26T03:00:46.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: 3.2 scrub starts 2024-03-26T03:00:46.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: 3.2 scrub ok 2024-03-26T03:00:46.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:46 smithi066 ceph-mon[37920]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:46.608 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:00:46.609 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:00:46.627 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:00:46.647 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:00:46.657 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:46.668 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:00:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.686 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:46.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:46.687 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:00:46.708 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:00:46.727 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:00:46.729 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: 2.13 scrub starts 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: 2.13 scrub ok 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: 5.c deep-scrub starts 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: 5.c deep-scrub ok 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: 3.2 scrub starts 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: 3.2 scrub ok 2024-03-26T03:00:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:46 smithi192 ceph-mon[33110]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:46.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:46.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:46.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:46.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:46.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:47.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:47.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:47.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:47.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:47.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:47.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:47.384 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:47 smithi066 ceph-mon[29666]: 5.1c scrub starts 2024-03-26T03:00:47.384 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:47 smithi066 ceph-mon[29666]: 5.1c scrub ok 2024-03-26T03:00:47.384 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:47 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/837449731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:00:47.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:47 smithi066 ceph-mon[37920]: 5.1c scrub starts 2024-03-26T03:00:47.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:47 smithi066 ceph-mon[37920]: 5.1c scrub ok 2024-03-26T03:00:47.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:47 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/837449731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:00:47.421 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.449 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:47.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:47.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.578 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:47.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:47.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:47.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:47 smithi192 ceph-mon[33110]: 5.1c scrub starts 2024-03-26T03:00:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:47 smithi192 ceph-mon[33110]: 5.1c scrub ok 2024-03-26T03:00:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:47 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/837449731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:00:47.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:47.835 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:47.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:47.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:47.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:48.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:48.092 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:48.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:48.222 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:48.324 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:48.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:48.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:48.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:48 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:00:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:48 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:00:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:48 smithi192 ceph-mon[33110]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:48 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:48.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:48.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[29666]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:00:48.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[37920]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:48.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:48 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:48.707 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.735 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:48.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:48.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:48.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:48.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:48.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:48.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:49.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:49.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:49.217 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:49.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:00:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:00:49.246 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:49.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:49.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:49.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.506 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:49.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:49.607 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: 6.9 deep-scrub starts 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: 6.9 deep-scrub ok 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:49.635 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: 6.9 deep-scrub starts 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: 6.9 deep-scrub ok 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:49.636 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:49 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:49.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:00:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:00:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: 6.9 deep-scrub starts 2024-03-26T03:00:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: 6.9 deep-scrub ok 2024-03-26T03:00:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:00:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:00:49.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:49.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:49.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:49 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:49.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:49.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:49.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:49.895 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:49.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:49.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:50.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:50.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:50.154 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:50.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:50.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:50.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:50.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:50.512 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:50.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:50.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:50.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:00:50.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:00:50.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[29666]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T03:00:50.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[29666]: 2.19 scrub starts 2024-03-26T03:00:50.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[29666]: 2.19 scrub ok 2024-03-26T03:00:50.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[29666]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:00:50.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:00:50.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:00:50.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[37920]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T03:00:50.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[37920]: 2.19 scrub starts 2024-03-26T03:00:50.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[37920]: 2.19 scrub ok 2024-03-26T03:00:50.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:50 smithi066 ceph-mon[37920]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:00:50.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:50 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:00:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:50 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:00:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:50 smithi192 ceph-mon[33110]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T03:00:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:50 smithi192 ceph-mon[33110]: 2.19 scrub starts 2024-03-26T03:00:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:50 smithi192 ceph-mon[33110]: 2.19 scrub ok 2024-03-26T03:00:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:50 smithi192 ceph-mon[33110]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:00:50.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:50.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:50.899 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:50.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:50.929 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:50.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:51.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.061 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:51.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:51.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.191 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:51.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:51.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:51.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:51.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:51.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:51.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:51.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:51.682 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:51.687 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:51 smithi066 ceph-mon[29666]: 4.19 scrub starts 2024-03-26T03:00:51.687 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:51 smithi066 ceph-mon[29666]: 4.19 scrub ok 2024-03-26T03:00:51.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:51 smithi066 ceph-mon[37920]: 4.19 scrub starts 2024-03-26T03:00:51.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:51 smithi066 ceph-mon[37920]: 4.19 scrub ok 2024-03-26T03:00:51.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:51 smithi192 ceph-mon[33110]: 4.19 scrub starts 2024-03-26T03:00:51.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:51 smithi192 ceph-mon[33110]: 4.19 scrub ok 2024-03-26T03:00:51.712 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:51.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:51.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:51.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:51.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:51.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:52.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:00:52.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:52.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:52.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:52.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:52.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:52.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:52.405 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:00:52.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:52.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:52.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:52.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:52.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: 6.1b scrub starts 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: 6.1b scrub ok 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/525718643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:00:52.702 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:00:52.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:00:52.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:00:52.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: 6.1b scrub starts 2024-03-26T03:00:52.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: 6.1b scrub ok 2024-03-26T03:00:52.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:52.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:52 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/525718643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:00:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:00:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:00:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:00:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:00:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: 6.1b scrub starts 2024-03-26T03:00:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: 6.1b scrub ok 2024-03-26T03:00:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:00:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:52 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/525718643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:00:52.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:52.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:52.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:52.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:52.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:53.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:53.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:53.126 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:53.158 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:53.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.289 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:53.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:53.373 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:00:53.373 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:00:53.373 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 95, 2024-03-26T03:00:53.373 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:00:53.373 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:00:52.398649+0000", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T02:58:43.841339+0000", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5801984, 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "noscrub", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:00:53.374 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 6, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.375 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:00:53.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 1, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 1, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 1, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 1, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 1, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:00:53.377 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "22", 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:00:53.378 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:00:53.379 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:00:53.379 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:00:53.379 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:00:53.379 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:00:53.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:00:53.381 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 7.8899998664855957, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 7.8899998664855957, 2024-03-26T03:00:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 3, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 3, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:00:53.383 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:00:53.384 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:00:53.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:00:53.386 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:00:53.387 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:00:53.388 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.5, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.5, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.5, 2024-03-26T03:00:53.389 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.5, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:00:53.390 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:00:53.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:00:53.392 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:00:53.393 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:00:53.394 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6500000953674316, 2024-03-26T03:00:53.395 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6500000953674316, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:00:53.396 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:00:53.397 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:00:53.398 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:00:53.399 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:00:53.400 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.25, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.25, 2024-03-26T03:00:53.401 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.25, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.25, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:00:53.402 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:00:53.403 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:00:53.404 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:00:53.405 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:00:53.406 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.25, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.25, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.25, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.25, 2024-03-26T03:00:53.407 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:00:53.408 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:00:53.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:53.409 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:00:53.410 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:00:53.411 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:00:53.412 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:00:53.413 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.414 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.75, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.75, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:00:53.415 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.416 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 69, 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.417 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:00:53.418 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:00:53.419 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.420 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 69, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.422 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.423 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.424 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 69, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:00:53.425 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.426 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:00:53.427 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 76, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:00:53.430 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.431 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.432 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 76, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.433 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.434 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:00:53.435 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.436 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.437 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.438 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 76, 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.439 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.440 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:00:53.441 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.442 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 76, 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.443 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.445 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:00:53.446 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 69, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.447 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.448 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.449 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.450 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:43.767821+0000", 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.451 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:14.332126+0000", 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:19.644109+0000", 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:41.569980+0000", 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.453 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:07.097522+0000", 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.454 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:18.462571+0000", 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:48.408696+0000", 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:00:53.455 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:00:10.625886+0000", 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [], 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:00:53.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.457 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:00:53.458 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [], 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:00:53.459 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:00:53.460 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:00:53.461 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:00:53.462 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:00:53.462 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:00:53.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:53.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: 6.1f scrub starts 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: 6.1f scrub ok 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/525718643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: 2.9 scrub starts 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: 2.9 scrub ok 2024-03-26T03:00:53.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/478266292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: 6.1f scrub starts 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: 6.1f scrub ok 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/525718643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: 2.9 scrub starts 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: 2.9 scrub ok 2024-03-26T03:00:53.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:53.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:53.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/478266292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:00:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:53.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:53.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: 6.1f scrub starts 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: 6.1f scrub ok 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/525718643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: 2.9 scrub starts 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: 2.9 scrub ok 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:00:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/478266292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:00:53.747 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:53.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:53.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:00:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:00:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:00:53.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:53.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:53.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:54.003 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:54.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.133 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:54.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:54.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:54.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:54.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:54.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:54.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:54.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:54.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:54.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:54.735 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3466452317' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:00:54.735 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:54 smithi066 ceph-mon[29666]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:54.735 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:54 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3466452317' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:00:54.735 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:54 smithi066 ceph-mon[37920]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:54 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3466452317' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:00:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:54 smithi192 ceph-mon[33110]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:00:54.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:54.763 INFO:tasks.thrashosds.thrasher:j is {'epoch': 95, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:00:52.398649+0000', 'last_up_change': '2024-03-26T02:58:43.841339+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 76, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 69, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:43.767821+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:14.332126+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:19.644109+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:41.569980+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:07.097522+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:18.462571+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:48.408696+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:00:10.625886+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:00:54.763 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-26T03:00:54.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:54.873 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:54.902 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:54.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:55.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:55.005 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_0' created 2024-03-26T03:00:55.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:55.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:55.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:55.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.290 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:55.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:55.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:55.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:55.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:55.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:55.647 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:55.675 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3466452317' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T03:00:55.807 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:55 smithi192 ceph-mon[33110]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T03:00:55.808 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:55.836 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3466452317' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T03:00:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:55 smithi066 ceph-mon[29666]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T03:00:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3466452317' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T03:00:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:55 smithi066 ceph-mon[37920]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T03:00:55.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-26T03:00:55.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:55.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:55.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:56.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:56.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:56.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:56.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:56.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:56.354 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:56.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:56.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:56.578 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:56.582 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:56.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:56.590 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:00:56.591 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:00:56.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:56 smithi066 ceph-mon[37920]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T03:00:56.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:56 smithi066 ceph-mon[37920]: pgmap v59: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:56.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:56 smithi066 ceph-mon[29666]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T03:00:56.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:56 smithi066 ceph-mon[29666]: pgmap v59: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:56.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:56.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:56 smithi192 ceph-mon[33110]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T03:00:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:56 smithi192 ceph-mon[33110]: pgmap v59: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:00:56.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:56.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:56.873 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:56.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:56.975 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.005 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:57.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:57.106 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:57.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:57.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:57.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:57.367 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.395 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:57.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:57.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:57.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:57 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:00:57.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:57 smithi066 ceph-mon[29666]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T03:00:57.513 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:57 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:00:57.513 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:57 smithi066 ceph-mon[37920]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T03:00:57.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:57.626 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.654 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:57.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:57 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:00:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:57 smithi192 ceph-mon[33110]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T03:00:57.755 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.786 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:57.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:57.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:57.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:57.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:58.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:58.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:58.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.172 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:58.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:58.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:58.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:58.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:00:58.406 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:58.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:58.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:58.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:58.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:58.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:58 smithi192 ceph-mon[33110]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-03-26T03:00:58.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:58.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:58.827 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:58 smithi066 ceph-mon[29666]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-03-26T03:00:58.828 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:58 smithi066 ceph-mon[37920]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 926 B/s rd, 0 op/s 2024-03-26T03:00:58.858 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:00:58.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:00:58.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:58.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:00:59.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:59.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:00:59.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:00:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:59.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:00:59.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:00:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:00:59.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:00:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:59.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:00:59.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:00:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.475 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:59.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:00:59.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:00:59.603 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:00:59.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:00:59 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:59.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:00:59 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:59.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:00:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.734 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:59.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:00:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:00:59 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:00:59.764 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:00:59.765 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:00:59.765 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-26T03:00:59.765 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:00:59.765 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:00:59.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-26T03:00:59.778 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:00:59.835 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:00:59.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.864 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:00:59.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:00:59.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:00:59.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:00:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:59.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:00.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:00.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:00.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:00.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:00.346 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:01:00.354 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:00.365 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:01:00.382 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:00.385 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:01:00.404 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:01:00.424 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:01:00.447 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:01:00.468 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:01:00.484 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:00.487 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:01:00.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:00 smithi066 ceph-mon[29666]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:01:00.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2269430232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:00.488 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:00 smithi066 ceph-mon[37920]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:01:00.488 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2269430232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:01:00.489 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:01:00.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:00.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:00.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:00 smithi192 ceph-mon[33110]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:01:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2269430232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:00.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:00.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:00.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:00.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:00.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:01.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:01.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:01.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:01.161 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:01.262 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.293 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:01.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:01.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.423 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:01.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:01.460 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-26T03:01:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:01 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/545960498' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:01:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:01 smithi192 ceph-mon[33110]: from='client.34706 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:01:01.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:01.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:01 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/545960498' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:01:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:01 smithi066 ceph-mon[29666]: from='client.34706 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:01:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:01 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/545960498' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:01:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:01 smithi066 ceph-mon[37920]: from='client.34706 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:01:01.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:01.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:01.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:01.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:01.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:01.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:01.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:02.044 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:02.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:02.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:02.202 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:02.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.335 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:02.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:02.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:02.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:02.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:02 smithi066 ceph-mon[37920]: from='client.34706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:01:02.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:02 smithi066 ceph-mon[37920]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T03:01:02.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:02 smithi066 ceph-mon[37920]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-26T03:01:02.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:02 smithi066 ceph-mon[29666]: from='client.34706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:01:02.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:02 smithi066 ceph-mon[29666]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T03:01:02.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:02 smithi066 ceph-mon[29666]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-26T03:01:02.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.595 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:02.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:02.696 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.727 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:02.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:02 smithi192 ceph-mon[33110]: from='client.34706 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:01:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:02 smithi192 ceph-mon[33110]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T03:01:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:02 smithi192 ceph-mon[33110]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-03-26T03:01:02.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:02.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:02.870 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:02.958 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:02.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:02.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:03.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:03.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:03.157 DEBUG:teuthology.orchestra.run.smithi066:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-26T03:01:03.160 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:01:03.160 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:01:03.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:03.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:03.250 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:hints = 1 2024-03-26T03:01:03.250 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:01:03.250 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object prefix: benchmark_data_smithi066_76917 2024-03-26T03:01:03.250 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:01:03.251 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:01:03.347 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:03.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:03.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:03.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:01:03.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:03.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:03 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:01:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:03 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:01:03.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:03.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:03 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:01:03.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:03.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:03.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:03.975 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:03.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:04.077 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:01:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:01:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:04.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:04.137 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:04.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:04.250 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 1 15 3081 3066 191.614 191.625 0.00434771 0.00519871 2024-03-26T03:01:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:04.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:04.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:04.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:04.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:04.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:04.657 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:04.758 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.809 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:04.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:04.910 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:04.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:04.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:04.973 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:04 smithi066 ceph-mon[29666]: pgmap v65: 148 pgs: 148 active+clean; 121 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 241 op/s 2024-03-26T03:01:04.974 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:04 smithi066 ceph-mon[37920]: pgmap v65: 148 pgs: 148 active+clean; 121 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 241 op/s 2024-03-26T03:01:05.061 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:05.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:05.208 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:05.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:04 smithi192 ceph-mon[33110]: pgmap v65: 148 pgs: 148 active+clean; 121 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 241 op/s 2024-03-26T03:01:05.253 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:05.254 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 2 16 6024 6008 187.401 183.875 0.00585273 0.00531552 2024-03-26T03:01:05.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:05.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:05.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.516 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:05.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:05.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:05.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:05.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:05.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:05.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:05.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:05.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:06.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:06.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:06.122 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:01:06.122 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:01:06.122 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:01:06.123 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:01:06.123 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:01:06.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:06.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:06.254 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 3 16 8764 8748 182.019 171.25 0.00465168 0.00548371 2024-03-26T03:01:06.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:06.374 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:06.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:06.520 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:06.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:06.678 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:06.780 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:06.829 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:06.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:06.978 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:06.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:07.080 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:07.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:07.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:07.243 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:06 smithi066 ceph-mon[29666]: pgmap v66: 148 pgs: 148 active+clean; 129 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 218 op/s 2024-03-26T03:01:07.243 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:06 smithi066 ceph-mon[37920]: pgmap v66: 148 pgs: 148 active+clean; 129 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 218 op/s 2024-03-26T03:01:07.254 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 4 15 11524 11509 179.653 172.562 0.00558307 0.00555764 2024-03-26T03:01:07.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:07.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:01:07.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:07.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:06 smithi192 ceph-mon[33110]: pgmap v66: 148 pgs: 148 active+clean; 129 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 218 op/s 2024-03-26T03:01:07.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:07.551 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:07.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:07.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:07.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:07.837 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:07.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:07.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:07.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:08.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:08.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:08.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:08.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 5 16 14146 14130 176.484 163.812 0.0117523 0.00565914 2024-03-26T03:01:08.259 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:08.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.400 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:08.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:08.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:08.548 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:08.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:08.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:08.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:08.836 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:08.938 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:08.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:08.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:09.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:09.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:09.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:09.254 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 6 16 16436 16420 170.925 143.125 0.00683519 0.00584342 2024-03-26T03:01:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:09.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:09.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:09.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:08 smithi066 ceph-mon[29666]: pgmap v67: 148 pgs: 148 active+clean; 129 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 207 op/s 2024-03-26T03:01:09.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:08 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:09.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:08 smithi066 ceph-mon[37920]: pgmap v67: 148 pgs: 148 active+clean; 129 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 207 op/s 2024-03-26T03:01:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:08 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:09.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:08 smithi192 ceph-mon[33110]: pgmap v67: 148 pgs: 148 active+clean; 129 MiB data, 318 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 207 op/s 2024-03-26T03:01:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:08 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:09.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:01:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:01:09.491 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:09.534 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:09.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:09.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:09.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:09.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:09.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:09.950 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:09.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:10.017 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:10.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:10.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:10.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:10.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:10.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 7 16 18896 18880 168.47 153.75 0.00666828 0.00592962 2024-03-26T03:01:10.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:10.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:10.332 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.367 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:10.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:10.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:10.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:10.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:10.603 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.658 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:10.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:10.675 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:10.745 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:01:10.746 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:01:10.746 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:01:10.759 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:10.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:10.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:10.984 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:10.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:11.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:11.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:11.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:11.241 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:11.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 8 16 20812 20796 162.381 119.75 0.00330985 0.00615338 2024-03-26T03:01:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:11.300 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:11.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:11.323 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[29666]: pgmap v68: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[37920]: pgmap v68: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:01:11.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.401 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:11.451 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:11.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:11 smithi192 ceph-mon[33110]: pgmap v68: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-03-26T03:01:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:01:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:01:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:11.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:11.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:11.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:11.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:11.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:11.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:11.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:11.864 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:11.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:11.965 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:12.013 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:12.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:12.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:12.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:12.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 9 15 22866 22851 158.61 128.438 0.00794579 0.00629728 2024-03-26T03:01:12.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:12.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:12.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:12 smithi192 ceph-mon[33110]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T03:01:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:12 smithi192 ceph-mon[33110]: Reconfiguring daemon alertmanager.a on smithi066 2024-03-26T03:01:12.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:12.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:12 smithi066 ceph-mon[29666]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T03:01:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:12 smithi066 ceph-mon[29666]: Reconfiguring daemon alertmanager.a on smithi066 2024-03-26T03:01:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:12 smithi066 ceph-mon[37920]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T03:01:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:12 smithi066 ceph-mon[37920]: Reconfiguring daemon alertmanager.a on smithi066 2024-03-26T03:01:12.649 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.700 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:12.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:12.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:12.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:12.934 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:12.982 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:12.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:12.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:13.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:13.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:13.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:13.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:13.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 10 16 25166 25150 157.117 143.688 0.00626388 0.00635887 2024-03-26T03:01:13.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:13.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:13.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:13 smithi192 ceph-mon[33110]: pgmap v69: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 976 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-26T03:01:13.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:13.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:13 smithi066 ceph-mon[29666]: pgmap v69: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 976 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-26T03:01:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:13 smithi066 ceph-mon[37920]: pgmap v69: 148 pgs: 148 active+clean; 973 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 976 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-26T03:01:13.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:13.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:13.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:13.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:13.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:13.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:14.015 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:14.060 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:14.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:01:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:01:14.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:14.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:14.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 11 16 26726 26710 151.698 97.5 0.00308532 0.00658694 2024-03-26T03:01:14.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:14.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:14.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:14 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2439800116' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:14.407 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:14 smithi066 ceph-mon[29666]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:14.407 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:14 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2439800116' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:14.407 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:14 smithi066 ceph-mon[37920]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:14.408 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:01:14.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:14.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:14 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2439800116' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:14 smithi192 ceph-mon[33110]: from='client.34721 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:14.552 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:14.580 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:14.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:14.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:14.813 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.841 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:14.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:14.942 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:14.974 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:14.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:15.075 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:15.122 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:15.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:15.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 12 16 28153 28137 146.49 89.1875 0.00613947 0.006821 2024-03-26T03:01:15.267 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:15.371 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.400 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:15.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:15 smithi192 ceph-mon[33110]: pgmap v70: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-26T03:01:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:15 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:01:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:15 smithi192 ceph-mon[33110]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:01:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:15 smithi192 ceph-mon[33110]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T03:01:15.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.546 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:15.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:15.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[29666]: pgmap v70: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-26T03:01:15.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:01:15.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[29666]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:01:15.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[29666]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T03:01:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[37920]: pgmap v70: 148 pgs: 148 active+clean; 1.5 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-26T03:01:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:01:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[37920]: from='client.34721 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:01:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:15 smithi066 ceph-mon[37920]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T03:01:15.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.676 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:15.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:15.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:15.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:15.910 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:15.953 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:15.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:16.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.103 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:16.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:16.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.232 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:16.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:16.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 13 16 29453 29437 141.472 81.25 0.00236721 0.00697476 2024-03-26T03:01:16.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:16.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:16.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:16.466 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:16.511 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:16.613 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:16.640 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:16.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:16.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:16.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:16.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:16.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:17.005 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.034 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:17.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:17.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:17.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:17.255 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 14 16 30652 30636 136.72 74.9375 0.00213468 0.00726986 2024-03-26T03:01:17.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:17.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:17.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.423 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:17.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:17 smithi192 ceph-mon[33110]: pgmap v72: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-26T03:01:17.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:17.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:17 smithi066 ceph-mon[29666]: pgmap v72: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-26T03:01:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:17 smithi066 ceph-mon[37920]: pgmap v72: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-26T03:01:17.659 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.689 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:17.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:17.710 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:01:17.710 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:01:17.711 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-26T03:01:17.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:01:17.790 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.820 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:17.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:17.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:17.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:17.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:18.052 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:18.108 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:18.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:18.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:18.256 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 15 16 31895 31879 132.778 77.6875 0.0170435 0.00752284 2024-03-26T03:01:18.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:18.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:18.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:18.500 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:18.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:18.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:18.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:18.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:18.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:18.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:18.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:18.999 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:19.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:19.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:19.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:19.256 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 16 15 33469 33454 130.632 98.4375 0.0113613 0.00764977 2024-03-26T03:01:19.283 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.332 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:19.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:19.433 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:19.479 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:19 smithi192 ceph-mon[33110]: pgmap v73: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-26T03:01:19.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:01:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:01:19.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:19 smithi066 ceph-mon[29666]: pgmap v73: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-26T03:01:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:19 smithi066 ceph-mon[37920]: pgmap v73: 148 pgs: 148 active+clean; 1.6 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-26T03:01:19.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:19.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:19.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:19.842 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:19.899 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:19.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:20.001 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:20.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:20.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:20.199 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:20.256 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 17 16 35124 35108 129.028 103.375 0.0147689 0.00774392 2024-03-26T03:01:20.302 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:20.351 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:20.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:20.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:20.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:20.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:20.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:20.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:20.721 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:20.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:20.870 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:20.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:20.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:21.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:21.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:21.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:21.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:21.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:01:21.256 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 18 16 36950 36934 128.2 114.125 0.00300036 0.00779434 2024-03-26T03:01:21.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:21.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:21 smithi066 ceph-mon[37920]: pgmap v74: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-26T03:01:21.300 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:21 smithi066 ceph-mon[29666]: pgmap v74: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-26T03:01:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.331 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:21.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:21.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:21.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:21 smithi192 ceph-mon[33110]: pgmap v74: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-26T03:01:21.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:21.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:21.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:21.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:21.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:21.901 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:21.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:22.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:22.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:22.135 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:22.181 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:22.258 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 19 16 38573 38557 126.779 101.438 0.00545611 0.00788125 2024-03-26T03:01:22.283 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:22.352 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:22.426 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:01:22.427 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:01:22.428 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:01:22.429 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:22 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2372251418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:22.429 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:22 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2372251418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:22.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:22 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2372251418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:22.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:22.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:22.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:22.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:22.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:22.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:22.853 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:22.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:23.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:23.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:23.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:23.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:23.258 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:01:23.258994+0000 min lat: 0.00124684 max lat: 0.19348 avg lat: 0.00794111 2024-03-26T03:01:23.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:01:23.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 20 16 40303 40287 125.846 108.125 0.00676164 0.00794111 2024-03-26T03:01:23.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:23.302 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:23.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:23 smithi066 ceph-mon[29666]: pgmap v75: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-26T03:01:23.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:23 smithi066 ceph-mon[37920]: pgmap v75: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-26T03:01:23.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:23.404 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:23.450 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:23 smithi192 ceph-mon[33110]: pgmap v75: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.74k op/s 2024-03-26T03:01:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:23.552 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:23.595 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:23.700 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:23.747 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:23.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:23.878 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:23.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:23.900 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:01:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:01:23.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:24.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:24.011 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:01:24.011 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:01:24.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:24.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:24.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:24.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 21 16 41994 41978 124.886 105.688 0.00717002 0.00800158 2024-03-26T03:01:24.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:24.345 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:24.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:24.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:24.465 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:24.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:24.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:24.642 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:24.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:24.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:24.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:24.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:24.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:25.059 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:25.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:25.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:25.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:25.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:25.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[29666]: pgmap v76: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T03:01:25.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[29666]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T03:01:25.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:25.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:25.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[37920]: pgmap v76: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T03:01:25.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:24 smithi066 ceph-mon[37920]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T03:01:25.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:24 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:24 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:24 smithi192 ceph-mon[33110]: pgmap v76: 148 pgs: 148 active+clean; 2.3 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T03:01:25.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:24 smithi192 ceph-mon[33110]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T03:01:25.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:25.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:25.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 22 16 43710 43694 124.084 107.25 0.0109869 0.00805335 2024-03-26T03:01:25.340 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:25.388 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:25.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:25.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:25.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:25.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:25.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:25.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:25.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:25.684 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:01:25.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:25.844 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:25.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:25.858 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:25 smithi066 systemd[1]: Stopping Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:01:25.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:26.012 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:26.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.166 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:26.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:26.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 23 15 45295 45280 122.999 99.125 0.00530471 0.00812467 2024-03-26T03:01:26.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:26.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:26.402 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.485 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:26.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:26.586 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:26.633 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:26.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:26.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:26.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:26.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:26.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:27.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:27.048 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:27.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:27.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:26 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:26 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:26 smithi192 ceph-mon[33110]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T03:01:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:26 smithi192 ceph-mon[33110]: pgmap v79: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:01:27.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 24 16 46951 46935 122.184 103.438 0.00977101 0.00817939 2024-03-26T03:01:27.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:27.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:27.345 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[29666]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[29666]: pgmap v79: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[37920]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T03:01:27.346 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:26 smithi066 ceph-mon[37920]: pgmap v79: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:01:27.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.489 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:27.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:27.591 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:27.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:27.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:27.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:27.871 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:27.903 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:27.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:28.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:28.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:28.131 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:28.175 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:28.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 25 16 48307 48291 120.687 84.75 0.00419876 0.00828056 2024-03-26T03:01:28.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:28.319 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:28.421 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.486 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:28.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:28.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:28.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:28.717 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.776 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:28.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:28.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:28.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:28.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:28.964 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:28 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[72822]: ts=2024-03-26T03:01:28.808Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T03:01:29.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:29.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:29.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:29.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:29.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 26 16 49704 49688 119.403 87.3125 0.00988973 0.00836932 2024-03-26T03:01:29.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:29.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:29.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:28 smithi066 ceph-mon[29666]: pgmap v80: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-03-26T03:01:29.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:29 smithi066 ceph-mon[37920]: pgmap v80: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-03-26T03:01:29.402 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.451 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:29.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:28 smithi192 ceph-mon[33110]: pgmap v80: 148 pgs: 148 active+clean; 2.5 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-03-26T03:01:29.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:01:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:01:29.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:29.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:29.685 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.729 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:29.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:29.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:29.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:29.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:29.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:29.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:30.091 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:30.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:30.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:30.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 27 16 51121 51105 118.261 88.5625 0.00579352 0.00845019 2024-03-26T03:01:30.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:30 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:30 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:30 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:30 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:30.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:30.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:30.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:30 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:30 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:30.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:30.553 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:30.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:30.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:30.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:30.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:30.934 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:30.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:01:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:30.979 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:30.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:31.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:31.136 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:31.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:31.258 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:31 smithi066 ceph-mon[37920]: from='client.34733 v1:172.21.15.66:0/4091301828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:01:31.258 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:31 smithi066 ceph-mon[37920]: pgmap v81: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-03-26T03:01:31.259 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:31 smithi066 ceph-mon[37920]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T03:01:31.259 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:31 smithi066 ceph-mon[29666]: from='client.34733 v1:172.21.15.66:0/4091301828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:01:31.259 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:31 smithi066 ceph-mon[29666]: pgmap v81: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-03-26T03:01:31.259 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:31 smithi066 ceph-mon[29666]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T03:01:31.259 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:30 smithi066 podman[79507]: 2024-03-26 03:01:30.90350104 +0000 UTC m=+2.452238716 container died 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:31.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 28 16 52597 52581 117.332 92.25 0.00420613 0.00851657 2024-03-26T03:01:31.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:31.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:31.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:31 smithi192 ceph-mon[33110]: from='client.34733 v1:172.21.15.66:0/4091301828' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:01:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:31 smithi192 ceph-mon[33110]: pgmap v81: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-03-26T03:01:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:31 smithi192 ceph-mon[33110]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T03:01:31.499 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:31.543 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:31.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:31.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:31.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:31.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:31.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:31.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:31.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:32.033 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:31 smithi066 podman[79507]: 2024-03-26 03:01:31.71945446 +0000 UTC m=+3.268192136 container cleanup 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:32.033 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:31 smithi066 bash[79507]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T03:01:32.034 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:31 smithi066 podman[79553]: 2024-03-26 03:01:31.947292313 +0000 UTC m=+3.069903077 container remove 96b8771af999e80c9b4ebf70a62ba00c0848a03171abbcb8ae809e7abfe40d04 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:32.036 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:01:32.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:32.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:32.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:32.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:32.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 29 16 53996 53980 116.301 87.4375 0.00653382 0.00859293 2024-03-26T03:01:32.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:32.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3979448806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:01:32.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:32 smithi066 ceph-mon[29666]: from='client.24769 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:01:32.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3979448806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:01:32.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:32 smithi066 ceph-mon[37920]: from='client.24769 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:01:32.359 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:32 smithi066 podman[79553]: 2024-03-26 03:01:32.047395371 +0000 UTC m=+3.170006137 volume remove bd0cd39abbd83ca0fdf134a1acbb1baa264aa91b64373704dcc210590b52970c 2024-03-26T03:01:32.362 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:32.464 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3979448806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:01:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:32 smithi192 ceph-mon[33110]: from='client.24769 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:01:32.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:32.614 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:32.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:32.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:32.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:32.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:32.926 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:01:32.931 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:01:32.939 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:32.941 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:32.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:33.043 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:33.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:33.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:33.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:33.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 30 16 55389 55373 115.326 87.0625 0.0105799 0.00866567 2024-03-26T03:01:33.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:33.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: pgmap v83: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 659 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:01:33.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:01:33.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: from='client.24769 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: pgmap v83: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 659 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: from='client.24769 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:01:33.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T03:01:33.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:01:33.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:01:33.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:01:33.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:01:33.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:33.480 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: pgmap v83: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 659 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: from='client.24769 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:01:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:33 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:01:33.525 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:33.627 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.668 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:33.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:33.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.819 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:33.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:33.895 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:01:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:01:33.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:33.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:33.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:34.054 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:34.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:34.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:34.174 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 6.6 scrub starts 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[29666]: 6.6 scrub ok 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:01:34.175 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:01:34.176 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:01:34.176 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:01:34.176 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:01:34.176 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 6.6 scrub starts 2024-03-26T03:01:34.176 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:34 smithi066 ceph-mon[37920]: 6.6 scrub ok 2024-03-26T03:01:34.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:34.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:34.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:34.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 31 16 56739 56723 114.328 84.375 0.00702357 0.00874121 2024-03-26T03:01:34.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:34.341 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:34.403 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-03-26T03:01:27.941799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2716473768,"num_objects":41671,"num_object_clones":0,"num_object_copies":125013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41671,"num_whiteouts":0,"num_read":1360,"num_read_kb":1174,"num_write":42248,"num_write_kb":2654685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42271,"ondisk_log_size":42271,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8119580,"kb_used_data":7828508,"kb_used_omap":12,"kb_used_meta":290995,"kb_avail":741841636,"statfs":{"total":767960285184,"available":759645835264,"internally_reserved":0,"allocated":8016392192,"data_stored":8013338792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297979440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":558694400,"num_objects":8525,"num_object_clones":0,"num_object_copies":25575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8525,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8525,"num_write_kb":545600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003881"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239494+0000","last_change":"2024-03-26T03:00:51.383778+0000","last_active":"2024-03-26T03:01:14.239494+0000","last_peered":"2024-03-26T03:01:14.239494+0000","last_clean":"2024-03-26T03:01:14.239494+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:01:14.239494+0000","last_undegraded":"2024-03-26T03:01:14.239494+0000","last_fullsized":"2024-03-26T03:01:14.239494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.383695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.383695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:57.990530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.226961+0000","last_change":"2024-03-26T02:59:00.943722+0000","last_active":"2024-03-26T03:01:14.226961+0000","last_peered":"2024-03-26T03:01:14.226961+0000","last_clean":"2024-03-26T03:01:14.226961+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:01:14.226961+0000","last_undegraded":"2024-03-26T03:01:14.226961+0000","last_fullsized":"2024-03-26T03:01:14.226961+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236724+0000","last_change":"2024-03-26T03:00:22.240686+0000","last_active":"2024-03-26T03:01:14.236724+0000","last_peered":"2024-03-26T03:01:14.236724+0000","last_clean":"2024-03-26T03:01:14.236724+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:01:14.236724+0000","last_undegraded":"2024-03-26T03:01:14.236724+0000","last_fullsized":"2024-03-26T03:01:14.236724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:22.240595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:22.240595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.476581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":162,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238687+0000","last_change":"2024-03-26T03:00:50.380208+0000","last_active":"2024-03-26T03:01:14.238687+0000","last_peered":"2024-03-26T03:01:14.238687+0000","last_clean":"2024-03-26T03:01:14.238687+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:01:14.238687+0000","last_undegraded":"2024-03-26T03:01:14.238687+0000","last_fullsized":"2024-03-26T03:01:14.238687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:00:50.380106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:50.380106+0000","objects_scrubbed":11,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:19.551966+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":78,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227097+0000","last_change":"2024-03-26T03:00:51.327781+0000","last_active":"2024-03-26T03:01:14.227097+0000","last_peered":"2024-03-26T03:01:14.227097+0000","last_clean":"2024-03-26T03:01:14.227097+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:01:14.227097+0000","last_undegraded":"2024-03-26T03:01:14.227097+0000","last_fullsized":"2024-03-26T03:01:14.227097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.327687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.327687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:01.077523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.231700+0000","last_change":"2024-03-26T02:59:00.932288+0000","last_active":"2024-03-26T03:01:14.231700+0000","last_peered":"2024-03-26T03:01:14.231700+0000","last_clean":"2024-03-26T03:01:14.231700+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:01:14.231700+0000","last_undegraded":"2024-03-26T03:01:14.231700+0000","last_fullsized":"2024-03-26T03:01:14.231700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239272+0000","last_change":"2024-03-26T03:00:41.464704+0000","last_active":"2024-03-26T03:01:14.239272+0000","last_peered":"2024-03-26T03:01:14.239272+0000","last_clean":"2024-03-26T03:01:14.239272+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:01:14.239272+0000","last_undegraded":"2024-03-26T03:01:14.239272+0000","last_fullsized":"2024-03-26T03:01:14.239272+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:41.464590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.464590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:09.359276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":114,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227199+0000","last_change":"2024-03-26T03:00:41.276506+0000","last_active":"2024-03-26T03:01:14.227199+0000","last_peered":"2024-03-26T03:01:14.227199+0000","last_clean":"2024-03-26T03:01:14.227199+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:01:14.227199+0000","last_undegraded":"2024-03-26T03:01:14.227199+0000","last_fullsized":"2024-03-26T03:01:14.227199+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:41.276382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.276382+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:09.188576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":78,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236116+0000","last_change":"2024-03-26T03:00:25.999429+0000","last_active":"2024-03-26T03:01:14.236116+0000","last_peered":"2024-03-26T03:01:14.236116+0000","last_clean":"2024-03-26T03:01:14.236116+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:01:14.236116+0000","last_undegraded":"2024-03-26T03:01:14.236116+0000","last_fullsized":"2024-03-26T03:01:14.236116+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.999320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.999320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:43.465119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234488+0000","last_change":"2024-03-26T03:00:18.269455+0000","last_active":"2024-03-26T03:01:14.234488+0000","last_peered":"2024-03-26T03:01:14.234488+0000","last_clean":"2024-03-26T03:01:14.234488+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:01:14.234488+0000","last_undegraded":"2024-03-26T03:01:14.234488+0000","last_fullsized":"2024-03-26T03:01:14.234488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.269349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.269349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:35.865636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234805+0000","last_change":"2024-03-26T03:00:51.169262+0000","last_active":"2024-03-26T03:01:14.234805+0000","last_peered":"2024-03-26T03:01:14.234805+0000","last_clean":"2024-03-26T03:01:14.234805+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:01:14.234805+0000","last_undegraded":"2024-03-26T03:01:14.234805+0000","last_fullsized":"2024-03-26T03:01:14.234805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.169167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.169167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:53.949803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":100,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.228332+0000","last_change":"2024-03-26T02:58:58.931435+0000","last_active":"2024-03-26T03:01:14.228332+0000","last_peered":"2024-03-26T03:01:14.228332+0000","last_clean":"2024-03-26T03:01:14.228332+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:01:14.228332+0000","last_undegraded":"2024-03-26T03:01:14.228332+0000","last_fullsized":"2024-03-26T03:01:14.228332+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.232950+0000","last_change":"2024-03-26T03:00:33.322325+0000","last_active":"2024-03-26T03:01:14.232950+0000","last_peered":"2024-03-26T03:01:14.232950+0000","last_clean":"2024-03-26T03:01:14.232950+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:01:14.232950+0000","last_undegraded":"2024-03-26T03:01:14.232950+0000","last_fullsized":"2024-03-26T03:01:14.232950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:33.322244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:33.322244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:47.435801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236481+0000","last_change":"2024-03-26T03:00:05.805098+0000","last_active":"2024-03-26T03:01:14.236481+0000","last_peered":"2024-03-26T03:01:14.236481+0000","last_clean":"2024-03-26T03:01:14.236481+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:01:14.236481+0000","last_undegraded":"2024-03-26T03:01:14.236481+0000","last_fullsized":"2024-03-26T03:01:14.236481+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:05.804994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.804994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.354036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234689+0000","last_change":"2024-03-26T03:00:40.224810+0000","last_active":"2024-03-26T03:01:14.234689+0000","last_peered":"2024-03-26T03:01:14.234689+0000","last_clean":"2024-03-26T03:01:14.234689+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:01:14.234689+0000","last_undegraded":"2024-03-26T03:01:14.234689+0000","last_fullsized":"2024-03-26T03:01:14.234689+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:40.224697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.224697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:46.550276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":126,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227246+0000","last_change":"2024-03-26T03:00:06.927566+0000","last_active":"2024-03-26T03:01:14.227246+0000","last_peered":"2024-03-26T03:01:14.227246+0000","last_clean":"2024-03-26T03:01:14.227246+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:01:14.227246+0000","last_undegraded":"2024-03-26T03:01:14.227246+0000","last_fullsized":"2024-03-26T03:01:14.227246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:06.927439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:06.927439+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240745+0000","last_change":"2024-03-26T03:00:52.383684+0000","last_active":"2024-03-26T03:01:14.240745+0000","last_peered":"2024-03-26T03:01:14.240745+0000","last_clean":"2024-03-26T03:01:14.240745+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:01:14.240745+0000","last_undegraded":"2024-03-26T03:01:14.240745+0000","last_fullsized":"2024-03-26T03:01:14.240745+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.383607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.383607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:04.843588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227668+0000","last_change":"2024-03-26T03:00:46.268131+0000","last_active":"2024-03-26T03:01:14.227668+0000","last_peered":"2024-03-26T03:01:14.227668+0000","last_clean":"2024-03-26T03:01:14.227668+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:01:14.227668+0000","last_undegraded":"2024-03-26T03:01:14.227668+0000","last_fullsized":"2024-03-26T03:01:14.227668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:46.268052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:46.268052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:54.813867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239592+0000","last_change":"2024-03-26T03:00:02.741809+0000","last_active":"2024-03-26T03:01:14.239592+0000","last_peered":"2024-03-26T03:01:14.239592+0000","last_clean":"2024-03-26T03:01:14.239592+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:01:14.239592+0000","last_undegraded":"2024-03-26T03:01:14.239592+0000","last_fullsized":"2024-03-26T03:01:14.239592+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.741725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.741725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.926475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":122,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239911+0000","last_change":"2024-03-26T03:00:25.578239+0000","last_active":"2024-03-26T03:01:14.239911+0000","last_peered":"2024-03-26T03:01:14.239911+0000","last_clean":"2024-03-26T03:01:14.239911+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:01:14.239911+0000","last_undegraded":"2024-03-26T03:01:14.239911+0000","last_fullsized":"2024-03-26T03:01:14.239911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:00:25.578139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.578139+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.013114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227756+0000","last_change":"2024-03-26T03:00:27.177539+0000","last_active":"2024-03-26T03:01:14.227756+0000","last_peered":"2024-03-26T03:01:14.227756+0000","last_clean":"2024-03-26T03:01:14.227756+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:01:14.227756+0000","last_undegraded":"2024-03-26T03:01:14.227756+0000","last_fullsized":"2024-03-26T03:01:14.227756+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.177404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.177404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:28.371450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229993+0000","last_change":"2024-03-26T03:00:18.300021+0000","last_active":"2024-03-26T03:01:14.229993+0000","last_peered":"2024-03-26T03:01:14.229993+0000","last_clean":"2024-03-26T03:01:14.229993+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:01:14.229993+0000","last_undegraded":"2024-03-26T03:01:14.229993+0000","last_fullsized":"2024-03-26T03:01:14.229993+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.299882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.299882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.440696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225813+0000","last_change":"2024-03-26T03:00:18.484389+0000","last_active":"2024-03-26T03:01:14.225813+0000","last_peered":"2024-03-26T03:01:14.225813+0000","last_clean":"2024-03-26T03:01:14.225813+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:01:14.225813+0000","last_undegraded":"2024-03-26T03:01:14.225813+0000","last_fullsized":"2024-03-26T03:01:14.225813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.484287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.484287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:22.739970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":112,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236130+0000","last_change":"2024-03-26T03:00:04.540480+0000","last_active":"2024-03-26T03:01:14.236130+0000","last_peered":"2024-03-26T03:01:14.236130+0000","last_clean":"2024-03-26T03:01:14.236130+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:01:14.236130+0000","last_undegraded":"2024-03-26T03:01:14.236130+0000","last_fullsized":"2024-03-26T03:01:14.236130+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:04.540375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.540375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229430+0000","last_change":"2024-03-26T02:59:02.938768+0000","last_active":"2024-03-26T03:01:14.229430+0000","last_peered":"2024-03-26T03:01:14.229430+0000","last_clean":"2024-03-26T03:01:14.229430+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:01:14.229430+0000","last_undegraded":"2024-03-26T03:01:14.229430+0000","last_fullsized":"2024-03-26T03:01:14.229430+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240385+0000","last_change":"2024-03-26T03:00:13.118260+0000","last_active":"2024-03-26T03:01:14.240385+0000","last_peered":"2024-03-26T03:01:14.240385+0000","last_clean":"2024-03-26T03:01:14.240385+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:01:14.240385+0000","last_undegraded":"2024-03-26T03:01:14.240385+0000","last_fullsized":"2024-03-26T03:01:14.240385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.118140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.552791+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":93,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238832+0000","last_change":"2024-03-26T03:00:49.420677+0000","last_active":"2024-03-26T03:01:14.238832+0000","last_peered":"2024-03-26T03:01:14.238832+0000","last_clean":"2024-03-26T03:01:14.238832+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:01:14.238832+0000","last_undegraded":"2024-03-26T03:01:14.238832+0000","last_fullsized":"2024-03-26T03:01:14.238832+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:00:49.420583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:49.420583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:49.686992+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":119,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225640+0000","last_change":"2024-03-26T03:00:30.623048+0000","last_active":"2024-03-26T03:01:14.225640+0000","last_peered":"2024-03-26T03:01:14.225640+0000","last_clean":"2024-03-26T03:01:14.225640+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:01:14.225640+0000","last_undegraded":"2024-03-26T03:01:14.225640+0000","last_fullsized":"2024-03-26T03:01:14.225640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:30.622926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:30.622926+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.084029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234906+0000","last_change":"2024-03-26T03:00:25.402119+0000","last_active":"2024-03-26T03:01:14.234906+0000","last_peered":"2024-03-26T03:01:14.234906+0000","last_clean":"2024-03-26T03:01:14.234906+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:01:14.234906+0000","last_undegraded":"2024-03-26T03:01:14.234906+0000","last_fullsized":"2024-03-26T03:01:14.234906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:00:25.402032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.402032+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:40.441905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":125,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.010477+0000","last_change":"2024-03-26T03:00:25.586479+0000","last_active":"2024-03-26T03:01:17.010477+0000","last_peered":"2024-03-26T03:01:17.010477+0000","last_clean":"2024-03-26T03:01:17.010477+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:01:17.010477+0000","last_undegraded":"2024-03-26T03:01:17.010477+0000","last_fullsized":"2024-03-26T03:01:17.010477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:25.586362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.586362+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.794553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236165+0000","last_change":"2024-03-26T03:00:18.957459+0000","last_active":"2024-03-26T03:01:14.236165+0000","last_peered":"2024-03-26T03:01:14.236165+0000","last_clean":"2024-03-26T03:01:14.236165+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:01:14.236165+0000","last_undegraded":"2024-03-26T03:01:14.236165+0000","last_fullsized":"2024-03-26T03:01:14.236165+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.957349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:36.020872+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":115,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240526+0000","last_change":"2024-03-26T03:00:36.337159+0000","last_active":"2024-03-26T03:01:14.240526+0000","last_peered":"2024-03-26T03:01:14.240526+0000","last_clean":"2024-03-26T03:01:14.240526+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:01:14.240526+0000","last_undegraded":"2024-03-26T03:01:14.240526+0000","last_fullsized":"2024-03-26T03:01:14.240526+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:00:36.337006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:36.337006+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.866766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240342+0000","last_change":"2024-03-26T02:59:03.207173+0000","last_active":"2024-03-26T03:01:14.240342+0000","last_peered":"2024-03-26T03:01:14.240342+0000","last_clean":"2024-03-26T03:01:14.240342+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:01:14.240342+0000","last_undegraded":"2024-03-26T03:01:14.240342+0000","last_fullsized":"2024-03-26T03:01:14.240342+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:27.669632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234594+0000","last_change":"2024-03-26T03:00:04.453033+0000","last_active":"2024-03-26T03:01:14.234594+0000","last_peered":"2024-03-26T03:01:14.234594+0000","last_clean":"2024-03-26T03:01:14.234594+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:01:14.234594+0000","last_undegraded":"2024-03-26T03:01:14.234594+0000","last_fullsized":"2024-03-26T03:01:14.234594+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.452915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.452915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.497308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225385+0000","last_change":"2024-03-26T03:00:29.597498+0000","last_active":"2024-03-26T03:01:14.225385+0000","last_peered":"2024-03-26T03:01:14.225385+0000","last_clean":"2024-03-26T03:01:14.225385+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:01:14.225385+0000","last_undegraded":"2024-03-26T03:01:14.225385+0000","last_fullsized":"2024-03-26T03:01:14.225385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:29.597404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:29.597404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:46.719488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"71'17","reported_seq":119,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239146+0000","last_change":"2024-03-26T03:00:36.458059+0000","last_active":"2024-03-26T03:01:14.239146+0000","last_peered":"2024-03-26T03:01:14.239146+0000","last_clean":"2024-03-26T03:01:14.239146+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:01:14.239146+0000","last_undegraded":"2024-03-26T03:01:14.239146+0000","last_fullsized":"2024-03-26T03:01:14.239146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:36.457970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:36.457970+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.468913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234379+0000","last_change":"2024-03-26T03:00:35.380327+0000","last_active":"2024-03-26T03:01:14.234379+0000","last_peered":"2024-03-26T03:01:14.234379+0000","last_clean":"2024-03-26T03:01:14.234379+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:01:14.234379+0000","last_undegraded":"2024-03-26T03:01:14.234379+0000","last_fullsized":"2024-03-26T03:01:14.234379+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.380233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.380233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.717029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225088+0000","last_change":"2024-03-26T03:00:21.571415+0000","last_active":"2024-03-26T03:01:14.225088+0000","last_peered":"2024-03-26T03:01:14.225088+0000","last_clean":"2024-03-26T03:01:14.225088+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:01:14.225088+0000","last_undegraded":"2024-03-26T03:01:14.225088+0000","last_fullsized":"2024-03-26T03:01:14.225088+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:21.571273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:21.571273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.284803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236764+0000","last_change":"2024-03-26T03:00:02.803641+0000","last_active":"2024-03-26T03:01:14.236764+0000","last_peered":"2024-03-26T03:01:14.236764+0000","last_clean":"2024-03-26T03:01:14.236764+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:01:14.236764+0000","last_undegraded":"2024-03-26T03:01:14.236764+0000","last_fullsized":"2024-03-26T03:01:14.236764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.803511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.803511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238339+0000","last_change":"2024-03-26T03:00:23.647837+0000","last_active":"2024-03-26T03:01:14.238339+0000","last_peered":"2024-03-26T03:01:14.238339+0000","last_clean":"2024-03-26T03:01:14.238339+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:01:14.238339+0000","last_undegraded":"2024-03-26T03:01:14.238339+0000","last_fullsized":"2024-03-26T03:01:14.238339+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:00:23.647732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.647732+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.169412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238468+0000","last_change":"2024-03-26T03:00:34.438665+0000","last_active":"2024-03-26T03:01:14.238468+0000","last_peered":"2024-03-26T03:01:14.238468+0000","last_clean":"2024-03-26T03:01:14.238468+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:01:14.238468+0000","last_undegraded":"2024-03-26T03:01:14.238468+0000","last_fullsized":"2024-03-26T03:01:14.238468+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:34.438582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:34.438582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.111003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":76,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.232002+0000","last_change":"2024-03-26T02:59:00.936834+0000","last_active":"2024-03-26T03:01:14.232002+0000","last_peered":"2024-03-26T03:01:14.232002+0000","last_clean":"2024-03-26T03:01:14.232002+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:01:14.232002+0000","last_undegraded":"2024-03-26T03:01:14.232002+0000","last_fullsized":"2024-03-26T03:01:14.232002+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229785+0000","last_change":"2024-03-26T03:00:02.224750+0000","last_active":"2024-03-26T03:01:14.229785+0000","last_peered":"2024-03-26T03:01:14.229785+0000","last_clean":"2024-03-26T03:01:14.229785+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:01:14.229785+0000","last_undegraded":"2024-03-26T03:01:14.229785+0000","last_fullsized":"2024-03-26T03:01:14.229785+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.224639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.224639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.228291+0000","last_change":"2024-03-26T02:58:58.925892+0000","last_active":"2024-03-26T03:01:14.228291+0000","last_peered":"2024-03-26T03:01:14.228291+0000","last_clean":"2024-03-26T03:01:14.228291+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:01:14.228291+0000","last_undegraded":"2024-03-26T03:01:14.228291+0000","last_fullsized":"2024-03-26T03:01:14.228291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.221474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.232450+0000","last_change":"2024-03-26T03:00:44.275620+0000","last_active":"2024-03-26T03:01:14.232450+0000","last_peered":"2024-03-26T03:01:14.232450+0000","last_clean":"2024-03-26T03:01:14.232450+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:01:14.232450+0000","last_undegraded":"2024-03-26T03:01:14.232450+0000","last_fullsized":"2024-03-26T03:01:14.232450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:44.275533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:44.275533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:05.064217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239823+0000","last_change":"2024-03-26T03:00:17.616155+0000","last_active":"2024-03-26T03:01:14.239823+0000","last_peered":"2024-03-26T03:01:14.239823+0000","last_clean":"2024-03-26T03:01:14.239823+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:01:14.239823+0000","last_undegraded":"2024-03-26T03:01:14.239823+0000","last_fullsized":"2024-03-26T03:01:14.239823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:17.616013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.616013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.986671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225530+0000","last_change":"2024-03-26T03:00:07.455788+0000","last_active":"2024-03-26T03:01:14.225530+0000","last_peered":"2024-03-26T03:01:14.225530+0000","last_clean":"2024-03-26T03:01:14.225530+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:01:14.225530+0000","last_undegraded":"2024-03-26T03:01:14.225530+0000","last_fullsized":"2024-03-26T03:01:14.225530+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.455691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.455691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.746322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":96,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.231813+0000","last_change":"2024-03-26T02:58:58.928475+0000","last_active":"2024-03-26T03:01:14.231813+0000","last_peered":"2024-03-26T03:01:14.231813+0000","last_clean":"2024-03-26T03:01:14.231813+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:01:14.231813+0000","last_undegraded":"2024-03-26T03:01:14.231813+0000","last_fullsized":"2024-03-26T03:01:14.231813+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:58:57.908400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.393345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.228017+0000","last_change":"2024-03-26T02:59:02.944972+0000","last_active":"2024-03-26T03:01:14.228017+0000","last_peered":"2024-03-26T03:01:14.228017+0000","last_clean":"2024-03-26T03:01:14.228017+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:01:14.228017+0000","last_undegraded":"2024-03-26T03:01:14.228017+0000","last_fullsized":"2024-03-26T03:01:14.228017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":122,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:22.013917+0000","last_change":"2024-03-26T03:00:32.418258+0000","last_active":"2024-03-26T03:01:22.013917+0000","last_peered":"2024-03-26T03:01:22.013917+0000","last_clean":"2024-03-26T03:01:22.013917+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:01:22.013917+0000","last_undegraded":"2024-03-26T03:01:22.013917+0000","last_fullsized":"2024-03-26T03:01:22.013917+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:32.418162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.418162+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:53.332629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236637+0000","last_change":"2024-03-26T03:00:45.281332+0000","last_active":"2024-03-26T03:01:14.236637+0000","last_peered":"2024-03-26T03:01:14.236637+0000","last_clean":"2024-03-26T03:01:14.236637+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:01:14.236637+0000","last_undegraded":"2024-03-26T03:01:14.236637+0000","last_fullsized":"2024-03-26T03:01:14.236637+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.281248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.281248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:52.951856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":115,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236657+0000","last_change":"2024-03-26T03:00:31.064878+0000","last_active":"2024-03-26T03:01:14.236657+0000","last_peered":"2024-03-26T03:01:14.236657+0000","last_clean":"2024-03-26T03:01:14.236657+0000","last_became_active":"2024-03-26T02:58:58.945477+0000","last_became_peered":"2024-03-26T02:58:58.945477+0000","last_unstale":"2024-03-26T03:01:14.236657+0000","last_undegraded":"2024-03-26T03:01:14.236657+0000","last_fullsized":"2024-03-26T03:01:14.236657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:31.064716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:31.064716+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.171634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239460+0000","last_change":"2024-03-26T03:00:09.986685+0000","last_active":"2024-03-26T03:01:14.239460+0000","last_peered":"2024-03-26T03:01:14.239460+0000","last_clean":"2024-03-26T03:01:14.239460+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:01:14.239460+0000","last_undegraded":"2024-03-26T03:01:14.239460+0000","last_fullsized":"2024-03-26T03:01:14.239460+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:09.986591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:09.986591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":120,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.010526+0000","last_change":"2024-03-26T03:00:10.855859+0000","last_active":"2024-03-26T03:01:17.010526+0000","last_peered":"2024-03-26T03:01:17.010526+0000","last_clean":"2024-03-26T03:01:17.010526+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:01:17.010526+0000","last_undegraded":"2024-03-26T03:01:17.010526+0000","last_fullsized":"2024-03-26T03:01:17.010526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:10.855698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.855698+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.308595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236290+0000","last_change":"2024-03-26T03:00:12.868763+0000","last_active":"2024-03-26T03:01:14.236290+0000","last_peered":"2024-03-26T03:01:14.236290+0000","last_clean":"2024-03-26T03:01:14.236290+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:01:14.236290+0000","last_undegraded":"2024-03-26T03:01:14.236290+0000","last_fullsized":"2024-03-26T03:01:14.236290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:12.868624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:12.868624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.398619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"71'22","reported_seq":127,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238855+0000","last_change":"2024-03-26T03:00:26.545331+0000","last_active":"2024-03-26T03:01:14.238855+0000","last_peered":"2024-03-26T03:01:14.238855+0000","last_clean":"2024-03-26T03:01:14.238855+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:01:14.238855+0000","last_undegraded":"2024-03-26T03:01:14.238855+0000","last_fullsized":"2024-03-26T03:01:14.238855+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'22","last_scrub_stamp":"2024-03-26T03:00:26.545179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.545179+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.405420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234699+0000","last_change":"2024-03-26T02:59:03.208607+0000","last_active":"2024-03-26T03:01:14.234699+0000","last_peered":"2024-03-26T03:01:14.234699+0000","last_clean":"2024-03-26T03:01:14.234699+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:01:14.234699+0000","last_undegraded":"2024-03-26T03:01:14.234699+0000","last_fullsized":"2024-03-26T03:01:14.234699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:15.193408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":82,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236519+0000","last_change":"2024-03-26T03:00:20.999242+0000","last_active":"2024-03-26T03:01:14.236519+0000","last_peered":"2024-03-26T03:01:14.236519+0000","last_clean":"2024-03-26T03:01:14.236519+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:01:14.236519+0000","last_undegraded":"2024-03-26T03:01:14.236519+0000","last_fullsized":"2024-03-26T03:01:14.236519+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:20.999149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.999149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.966830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225172+0000","last_change":"2024-03-26T03:00:04.420607+0000","last_active":"2024-03-26T03:01:14.225172+0000","last_peered":"2024-03-26T03:01:14.225172+0000","last_clean":"2024-03-26T03:01:14.225172+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:01:14.225172+0000","last_undegraded":"2024-03-26T03:01:14.225172+0000","last_fullsized":"2024-03-26T03:01:14.225172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:04.420504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.420504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.414672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":125,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240562+0000","last_change":"2024-03-26T03:00:05.814094+0000","last_active":"2024-03-26T03:01:14.240562+0000","last_peered":"2024-03-26T03:01:14.240562+0000","last_clean":"2024-03-26T03:01:14.240562+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:01:14.240562+0000","last_undegraded":"2024-03-26T03:01:14.240562+0000","last_fullsized":"2024-03-26T03:01:14.240562+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:00:05.813958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.813958+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.910237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.235700+0000","last_change":"2024-03-26T03:00:34.324054+0000","last_active":"2024-03-26T03:01:14.235700+0000","last_peered":"2024-03-26T03:01:14.235700+0000","last_clean":"2024-03-26T03:01:14.235700+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:01:14.235700+0000","last_undegraded":"2024-03-26T03:01:14.235700+0000","last_fullsized":"2024-03-26T03:01:14.235700+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:34.323948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:34.323948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:46.303373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240953+0000","last_change":"2024-03-26T03:00:07.801621+0000","last_active":"2024-03-26T03:01:14.240953+0000","last_peered":"2024-03-26T03:01:14.240953+0000","last_clean":"2024-03-26T03:01:14.240953+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:01:14.240953+0000","last_undegraded":"2024-03-26T03:01:14.240953+0000","last_fullsized":"2024-03-26T03:01:14.240953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.801516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.801516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:30.176058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.237088+0000","last_change":"2024-03-26T03:00:32.335851+0000","last_active":"2024-03-26T03:01:14.237088+0000","last_peered":"2024-03-26T03:01:14.237088+0000","last_clean":"2024-03-26T03:01:14.237088+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:01:14.237088+0000","last_undegraded":"2024-03-26T03:01:14.237088+0000","last_fullsized":"2024-03-26T03:01:14.237088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:00:32.335759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.335759+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:41.002597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":128,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239672+0000","last_change":"2024-03-26T03:00:40.380209+0000","last_active":"2024-03-26T03:01:14.239672+0000","last_peered":"2024-03-26T03:01:14.239672+0000","last_clean":"2024-03-26T03:01:14.239672+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:01:14.239672+0000","last_undegraded":"2024-03-26T03:01:14.239672+0000","last_fullsized":"2024-03-26T03:01:14.239672+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:40.380114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.380114+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:41.557631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"100'2626","reported_seq":2644,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.948615+0000","last_change":"2024-03-26T03:00:55.439899+0000","last_active":"2024-03-26T03:01:23.948615+0000","last_peered":"2024-03-26T03:01:23.948615+0000","last_clean":"2024-03-26T03:01:23.948615+0000","last_became_active":"2024-03-26T03:00:55.439557+0000","last_became_peered":"2024-03-26T03:00:55.439557+0000","last_unstale":"2024-03-26T03:01:23.948615+0000","last_undegraded":"2024-03-26T03:01:23.948615+0000","last_fullsized":"2024-03-26T03:01:23.948615+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:04.990950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172097536,"num_objects":2626,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2626,"num_write_kb":168064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240050+0000","last_change":"2024-03-26T03:00:35.424045+0000","last_active":"2024-03-26T03:01:14.240050+0000","last_peered":"2024-03-26T03:01:14.240050+0000","last_clean":"2024-03-26T03:01:14.240050+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:01:14.240050+0000","last_undegraded":"2024-03-26T03:01:14.240050+0000","last_fullsized":"2024-03-26T03:01:14.240050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.423904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.423904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:00.370227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":101,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227947+0000","last_change":"2024-03-26T02:58:56.919704+0000","last_active":"2024-03-26T03:01:14.227947+0000","last_peered":"2024-03-26T03:01:14.227947+0000","last_clean":"2024-03-26T03:01:14.227947+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:01:14.227947+0000","last_undegraded":"2024-03-26T03:01:14.227947+0000","last_fullsized":"2024-03-26T03:01:14.227947+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:20.213243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":156,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227342+0000","last_change":"2024-03-26T03:00:25.142453+0000","last_active":"2024-03-26T03:01:14.227342+0000","last_peered":"2024-03-26T03:01:14.227342+0000","last_clean":"2024-03-26T03:01:14.227342+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:01:14.227342+0000","last_undegraded":"2024-03-26T03:01:14.227342+0000","last_fullsized":"2024-03-26T03:01:14.227342+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:00:25.142334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.142334+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:37.994864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236945+0000","last_change":"2024-03-26T03:00:08.469473+0000","last_active":"2024-03-26T03:01:14.236945+0000","last_peered":"2024-03-26T03:01:14.236945+0000","last_clean":"2024-03-26T03:01:14.236945+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:01:14.236945+0000","last_undegraded":"2024-03-26T03:01:14.236945+0000","last_fullsized":"2024-03-26T03:01:14.236945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:08.469321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.469321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.714692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"100'2552","reported_seq":2570,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.912046+0000","last_change":"2024-03-26T03:00:55.437426+0000","last_active":"2024-03-26T03:01:23.912046+0000","last_peered":"2024-03-26T03:01:23.912046+0000","last_clean":"2024-03-26T03:01:23.912046+0000","last_became_active":"2024-03-26T03:00:55.437047+0000","last_became_peered":"2024-03-26T03:00:55.437047+0000","last_unstale":"2024-03-26T03:01:23.912046+0000","last_undegraded":"2024-03-26T03:01:23.912046+0000","last_fullsized":"2024-03-26T03:01:23.912046+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2552,"log_dups_size":0,"ondisk_log_size":2552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:14.190036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167247872,"num_objects":2552,"num_object_clones":0,"num_object_copies":7656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2552,"num_write_kb":163328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.232569+0000","last_change":"2024-03-26T03:00:11.023432+0000","last_active":"2024-03-26T03:01:14.232569+0000","last_peered":"2024-03-26T03:01:14.232569+0000","last_clean":"2024-03-26T03:01:14.232569+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:01:14.232569+0000","last_undegraded":"2024-03-26T03:01:14.232569+0000","last_fullsized":"2024-03-26T03:01:14.232569+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:11.023351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:11.023351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.872070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229466+0000","last_change":"2024-03-26T03:00:27.327214+0000","last_active":"2024-03-26T03:01:14.229466+0000","last_peered":"2024-03-26T03:01:14.229466+0000","last_clean":"2024-03-26T03:01:14.229466+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:01:14.229466+0000","last_undegraded":"2024-03-26T03:01:14.229466+0000","last_fullsized":"2024-03-26T03:01:14.229466+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.327116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.327116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.083798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"92'22","reported_seq":121,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.231866+0000","last_change":"2024-03-26T03:00:42.399658+0000","last_active":"2024-03-26T03:01:14.231866+0000","last_peered":"2024-03-26T03:01:14.231866+0000","last_clean":"2024-03-26T03:01:14.231866+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:01:14.231866+0000","last_undegraded":"2024-03-26T03:01:14.231866+0000","last_fullsized":"2024-03-26T03:01:14.231866+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:00:42.399566+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:12.127248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":125,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.010249+0000","last_change":"2024-03-26T03:00:08.968963+0000","last_active":"2024-03-26T03:01:17.010249+0000","last_peered":"2024-03-26T03:01:17.010249+0000","last_clean":"2024-03-26T03:01:17.010249+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:01:17.010249+0000","last_undegraded":"2024-03-26T03:01:17.010249+0000","last_fullsized":"2024-03-26T03:01:17.010249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:08.968848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.968848+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.867218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"100'2647","reported_seq":2665,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.877035+0000","last_change":"2024-03-26T03:00:55.439092+0000","last_active":"2024-03-26T03:01:23.877035+0000","last_peered":"2024-03-26T03:01:23.877035+0000","last_clean":"2024-03-26T03:01:23.877035+0000","last_became_active":"2024-03-26T03:00:55.438713+0000","last_became_peered":"2024-03-26T03:00:55.438713+0000","last_unstale":"2024-03-26T03:01:23.877035+0000","last_undegraded":"2024-03-26T03:01:23.877035+0000","last_fullsized":"2024-03-26T03:01:23.877035+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:04.743800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173473792,"num_objects":2647,"num_object_clones":0,"num_object_copies":7941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2647,"num_write_kb":169408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238627+0000","last_change":"2024-03-26T02:59:02.941781+0000","last_active":"2024-03-26T03:01:14.238627+0000","last_peered":"2024-03-26T03:01:14.238627+0000","last_clean":"2024-03-26T03:01:14.238627+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:01:14.238627+0000","last_undegraded":"2024-03-26T03:01:14.238627+0000","last_fullsized":"2024-03-26T03:01:14.238627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236414+0000","last_change":"2024-03-26T03:00:32.016983+0000","last_active":"2024-03-26T03:01:14.236414+0000","last_peered":"2024-03-26T03:01:14.236414+0000","last_clean":"2024-03-26T03:01:14.236414+0000","last_became_active":"2024-03-26T02:58:56.923317+0000","last_became_peered":"2024-03-26T02:58:56.923317+0000","last_unstale":"2024-03-26T03:01:14.236414+0000","last_undegraded":"2024-03-26T03:01:14.236414+0000","last_fullsized":"2024-03-26T03:01:14.236414+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:32.016849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.016849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:54.441479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":189,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.010219+0000","last_change":"2024-03-26T03:00:13.375248+0000","last_active":"2024-03-26T03:01:17.010219+0000","last_peered":"2024-03-26T03:01:17.010219+0000","last_clean":"2024-03-26T03:01:17.010219+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:01:17.010219+0000","last_undegraded":"2024-03-26T03:01:17.010219+0000","last_fullsized":"2024-03-26T03:01:17.010219+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.375080+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.471939+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236815+0000","last_change":"2024-03-26T03:00:29.282199+0000","last_active":"2024-03-26T03:01:14.236815+0000","last_peered":"2024-03-26T03:01:14.236815+0000","last_clean":"2024-03-26T03:01:14.236815+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:01:14.236815+0000","last_undegraded":"2024-03-26T03:01:14.236815+0000","last_fullsized":"2024-03-26T03:01:14.236815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:29.282115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:29.282115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.011941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"100'2550","reported_seq":2568,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.897312+0000","last_change":"2024-03-26T03:00:55.434714+0000","last_active":"2024-03-26T03:01:23.897312+0000","last_peered":"2024-03-26T03:01:23.897312+0000","last_clean":"2024-03-26T03:01:23.897312+0000","last_became_active":"2024-03-26T03:00:55.434404+0000","last_became_peered":"2024-03-26T03:00:55.434404+0000","last_unstale":"2024-03-26T03:01:23.897312+0000","last_undegraded":"2024-03-26T03:01:23.897312+0000","last_fullsized":"2024-03-26T03:01:23.897312+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:21.017560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167116800,"num_objects":2550,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":163200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234873+0000","last_change":"2024-03-26T03:00:41.268246+0000","last_active":"2024-03-26T03:01:14.234873+0000","last_peered":"2024-03-26T03:01:14.234873+0000","last_clean":"2024-03-26T03:01:14.234873+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:01:14.234873+0000","last_undegraded":"2024-03-26T03:01:14.234873+0000","last_fullsized":"2024-03-26T03:01:14.234873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.268157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:56.113846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":240,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:22.065388+0000","last_change":"2024-03-26T03:00:04.289538+0000","last_active":"2024-03-26T03:01:22.065388+0000","last_peered":"2024-03-26T03:01:22.065388+0000","last_clean":"2024-03-26T03:01:22.065388+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:01:22.065388+0000","last_undegraded":"2024-03-26T03:01:22.065388+0000","last_fullsized":"2024-03-26T03:01:22.065388+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:00:04.289452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.289452+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:25.471861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":133,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.235301+0000","last_change":"2024-03-26T03:00:23.413750+0000","last_active":"2024-03-26T03:01:14.235301+0000","last_peered":"2024-03-26T03:01:14.235301+0000","last_clean":"2024-03-26T03:01:14.235301+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:01:14.235301+0000","last_undegraded":"2024-03-26T03:01:14.235301+0000","last_fullsized":"2024-03-26T03:01:14.235301+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:23.413656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.413656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:37.034913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":127,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.230578+0000","last_change":"2024-03-26T03:00:23.305418+0000","last_active":"2024-03-26T03:01:14.230578+0000","last_peered":"2024-03-26T03:01:14.230578+0000","last_clean":"2024-03-26T03:01:14.230578+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:01:14.230578+0000","last_undegraded":"2024-03-26T03:01:14.230578+0000","last_fullsized":"2024-03-26T03:01:14.230578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:00:23.305307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.305307+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:52.977903+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.237111+0000","last_change":"2024-03-26T03:00:25.404962+0000","last_active":"2024-03-26T03:01:14.237111+0000","last_peered":"2024-03-26T03:01:14.237111+0000","last_clean":"2024-03-26T03:01:14.237111+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:01:14.237111+0000","last_undegraded":"2024-03-26T03:01:14.237111+0000","last_fullsized":"2024-03-26T03:01:14.237111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.404851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.404851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.989593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"100'2631","reported_seq":2649,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.925694+0000","last_change":"2024-03-26T03:00:55.439370+0000","last_active":"2024-03-26T03:01:23.925694+0000","last_peered":"2024-03-26T03:01:23.925694+0000","last_clean":"2024-03-26T03:01:23.925694+0000","last_became_active":"2024-03-26T03:00:55.438420+0000","last_became_peered":"2024-03-26T03:00:55.438420+0000","last_unstale":"2024-03-26T03:01:23.925694+0000","last_undegraded":"2024-03-26T03:01:23.925694+0000","last_fullsized":"2024-03-26T03:01:23.925694+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:12.861624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":168384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236825+0000","last_change":"2024-03-26T03:00:24.058533+0000","last_active":"2024-03-26T03:01:14.236825+0000","last_peered":"2024-03-26T03:01:14.236825+0000","last_clean":"2024-03-26T03:01:14.236825+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:01:14.236825+0000","last_undegraded":"2024-03-26T03:01:14.236825+0000","last_fullsized":"2024-03-26T03:01:14.236825+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:24.058394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:24.058394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.713360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":101,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.240430+0000","last_change":"2024-03-26T03:00:45.454030+0000","last_active":"2024-03-26T03:01:14.240430+0000","last_peered":"2024-03-26T03:01:14.240430+0000","last_clean":"2024-03-26T03:01:14.240430+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:01:14.240430+0000","last_undegraded":"2024-03-26T03:01:14.240430+0000","last_fullsized":"2024-03-26T03:01:14.240430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:00:45.453925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.453925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:12.344283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":97,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236656+0000","last_change":"2024-03-26T03:00:03.775166+0000","last_active":"2024-03-26T03:01:14.236656+0000","last_peered":"2024-03-26T03:01:14.236656+0000","last_clean":"2024-03-26T03:01:14.236656+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:01:14.236656+0000","last_undegraded":"2024-03-26T03:01:14.236656+0000","last_fullsized":"2024-03-26T03:01:14.236656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:03.775065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:03.775065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:17.102921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":331,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234313+0000","last_change":"2024-03-26T02:59:36.442802+0000","last_active":"2024-03-26T03:01:14.234313+0000","last_peered":"2024-03-26T03:01:14.234313+0000","last_clean":"2024-03-26T03:01:14.234313+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:01:14.234313+0000","last_undegraded":"2024-03-26T03:01:14.234313+0000","last_fullsized":"2024-03-26T03:01:14.234313+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-26T02:59:36.442703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T02:59:36.442703+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:15.193537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":140,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.010567+0000","last_change":"2024-03-26T03:00:28.559278+0000","last_active":"2024-03-26T03:01:17.010567+0000","last_peered":"2024-03-26T03:01:17.010567+0000","last_clean":"2024-03-26T03:01:17.010567+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:01:17.010567+0000","last_undegraded":"2024-03-26T03:01:17.010567+0000","last_fullsized":"2024-03-26T03:01:17.010567+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:00:28.559147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.559147+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:36.143298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234803+0000","last_change":"2024-03-26T03:00:15.331397+0000","last_active":"2024-03-26T03:01:14.234803+0000","last_peered":"2024-03-26T03:01:14.234803+0000","last_clean":"2024-03-26T03:01:14.234803+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:01:14.234803+0000","last_undegraded":"2024-03-26T03:01:14.234803+0000","last_fullsized":"2024-03-26T03:01:14.234803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.331210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.331210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.836621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"100'2562","reported_seq":2578,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.911756+0000","last_change":"2024-03-26T03:00:55.438582+0000","last_active":"2024-03-26T03:01:23.911756+0000","last_peered":"2024-03-26T03:01:23.911756+0000","last_clean":"2024-03-26T03:01:23.911756+0000","last_became_active":"2024-03-26T03:00:55.438295+0000","last_became_peered":"2024-03-26T03:00:55.438295+0000","last_unstale":"2024-03-26T03:01:23.911756+0000","last_undegraded":"2024-03-26T03:01:23.911756+0000","last_fullsized":"2024-03-26T03:01:23.911756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2562,"log_dups_size":0,"ondisk_log_size":2562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:19.892973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":7686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2562,"num_write_kb":163968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.230011+0000","last_change":"2024-03-26T03:00:35.391826+0000","last_active":"2024-03-26T03:01:14.230011+0000","last_peered":"2024-03-26T03:01:14.230011+0000","last_clean":"2024-03-26T03:01:14.230011+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:01:14.230011+0000","last_undegraded":"2024-03-26T03:01:14.230011+0000","last_fullsized":"2024-03-26T03:01:14.230011+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.391746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.391746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:47.437885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.235770+0000","last_change":"2024-03-26T03:00:32.315656+0000","last_active":"2024-03-26T03:01:14.235770+0000","last_peered":"2024-03-26T03:01:14.235770+0000","last_clean":"2024-03-26T03:01:14.235770+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:01:14.235770+0000","last_undegraded":"2024-03-26T03:01:14.235770+0000","last_fullsized":"2024-03-26T03:01:14.235770+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.315559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:52.465452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234192+0000","last_change":"2024-03-26T03:00:47.234857+0000","last_active":"2024-03-26T03:01:14.234192+0000","last_peered":"2024-03-26T03:01:14.234192+0000","last_clean":"2024-03-26T03:01:14.234192+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:01:14.234192+0000","last_undegraded":"2024-03-26T03:01:14.234192+0000","last_fullsized":"2024-03-26T03:01:14.234192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:47.234781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:47.234781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:51.672136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":106,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.239520+0000","last_change":"2024-03-26T02:59:58.957331+0000","last_active":"2024-03-26T03:01:14.239520+0000","last_peered":"2024-03-26T03:01:14.239520+0000","last_clean":"2024-03-26T03:01:14.239520+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:01:14.239520+0000","last_undegraded":"2024-03-26T03:01:14.239520+0000","last_fullsized":"2024-03-26T03:01:14.239520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T02:59:58.957239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.957239+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:16.565757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236811+0000","last_change":"2024-03-26T03:00:35.969971+0000","last_active":"2024-03-26T03:01:14.236811+0000","last_peered":"2024-03-26T03:01:14.236811+0000","last_clean":"2024-03-26T03:01:14.236811+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:01:14.236811+0000","last_undegraded":"2024-03-26T03:01:14.236811+0000","last_fullsized":"2024-03-26T03:01:14.236811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.969837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.969837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:52.221758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"100'2644","reported_seq":2660,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.909683+0000","last_change":"2024-03-26T03:00:55.440674+0000","last_active":"2024-03-26T03:01:23.909683+0000","last_peered":"2024-03-26T03:01:23.909683+0000","last_clean":"2024-03-26T03:01:23.909683+0000","last_became_active":"2024-03-26T03:00:55.440376+0000","last_became_peered":"2024-03-26T03:00:55.440376+0000","last_unstale":"2024-03-26T03:01:23.909683+0000","last_undegraded":"2024-03-26T03:01:23.909683+0000","last_fullsized":"2024-03-26T03:01:23.909683+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2644,"log_dups_size":0,"ondisk_log_size":2644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:09.184755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173277184,"num_objects":2644,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2644,"num_write_kb":169216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229468+0000","last_change":"2024-03-26T03:00:07.343102+0000","last_active":"2024-03-26T03:01:14.229468+0000","last_peered":"2024-03-26T03:01:14.229468+0000","last_clean":"2024-03-26T03:01:14.229468+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:01:14.229468+0000","last_undegraded":"2024-03-26T03:01:14.229468+0000","last_fullsized":"2024-03-26T03:01:14.229468+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.343020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.343020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:23.674172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234905+0000","last_change":"2024-03-26T03:00:49.223170+0000","last_active":"2024-03-26T03:01:14.234905+0000","last_peered":"2024-03-26T03:01:14.234905+0000","last_clean":"2024-03-26T03:01:14.234905+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:01:14.234905+0000","last_undegraded":"2024-03-26T03:01:14.234905+0000","last_fullsized":"2024-03-26T03:01:14.234905+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:49.223092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:49.223092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:04.304319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":136,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236321+0000","last_change":"2024-03-26T03:00:25.226087+0000","last_active":"2024-03-26T03:01:14.236321+0000","last_peered":"2024-03-26T03:01:14.236321+0000","last_clean":"2024-03-26T03:01:14.236321+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:01:14.236321+0000","last_undegraded":"2024-03-26T03:01:14.236321+0000","last_fullsized":"2024-03-26T03:01:14.236321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:00:25.225991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.225991+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:32.215639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225919+0000","last_change":"2024-03-26T03:00:10.472741+0000","last_active":"2024-03-26T03:01:14.225919+0000","last_peered":"2024-03-26T03:01:14.225919+0000","last_clean":"2024-03-26T03:01:14.225919+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:01:14.225919+0000","last_undegraded":"2024-03-26T03:01:14.225919+0000","last_fullsized":"2024-03-26T03:01:14.225919+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:10.472642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.472642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.985525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"100'2647","reported_seq":2665,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.925176+0000","last_change":"2024-03-26T03:00:55.438809+0000","last_active":"2024-03-26T03:01:23.925176+0000","last_peered":"2024-03-26T03:01:23.925176+0000","last_clean":"2024-03-26T03:01:23.925176+0000","last_became_active":"2024-03-26T03:00:55.438509+0000","last_became_peered":"2024-03-26T03:00:55.438509+0000","last_unstale":"2024-03-26T03:01:23.925176+0000","last_undegraded":"2024-03-26T03:01:23.925176+0000","last_fullsized":"2024-03-26T03:01:23.925176+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:06.130904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173473792,"num_objects":2647,"num_object_clones":0,"num_object_copies":7941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2647,"num_write_kb":169408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227440+0000","last_change":"2024-03-26T02:59:02.944368+0000","last_active":"2024-03-26T03:01:14.227440+0000","last_peered":"2024-03-26T03:01:14.227440+0000","last_clean":"2024-03-26T03:01:14.227440+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:01:14.227440+0000","last_undegraded":"2024-03-26T03:01:14.227440+0000","last_fullsized":"2024-03-26T03:01:14.227440+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.231377+0000","last_change":"2024-03-26T03:00:17.286899+0000","last_active":"2024-03-26T03:01:14.231377+0000","last_peered":"2024-03-26T03:01:14.231377+0000","last_clean":"2024-03-26T03:01:14.231377+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:01:14.231377+0000","last_undegraded":"2024-03-26T03:01:14.231377+0000","last_fullsized":"2024-03-26T03:01:14.231377+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.286796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:38.424244+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":156,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238177+0000","last_change":"2024-03-26T03:00:08.772624+0000","last_active":"2024-03-26T03:01:14.238177+0000","last_peered":"2024-03-26T03:01:14.238177+0000","last_clean":"2024-03-26T03:01:14.238177+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:01:14.238177+0000","last_undegraded":"2024-03-26T03:01:14.238177+0000","last_fullsized":"2024-03-26T03:01:14.238177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.772486+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:19.103774+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227394+0000","last_change":"2024-03-26T02:59:00.944090+0000","last_active":"2024-03-26T03:01:14.227394+0000","last_peered":"2024-03-26T03:01:14.227394+0000","last_clean":"2024-03-26T03:01:14.227394+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:01:14.227394+0000","last_undegraded":"2024-03-26T03:01:14.227394+0000","last_fullsized":"2024-03-26T03:01:14.227394+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"100'2573","reported_seq":2591,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.938280+0000","last_change":"2024-03-26T03:00:55.436650+0000","last_active":"2024-03-26T03:01:23.938280+0000","last_peered":"2024-03-26T03:01:23.938280+0000","last_clean":"2024-03-26T03:01:23.938280+0000","last_became_active":"2024-03-26T03:00:55.436340+0000","last_became_peered":"2024-03-26T03:00:55.436340+0000","last_unstale":"2024-03-26T03:01:23.938280+0000","last_undegraded":"2024-03-26T03:01:23.938280+0000","last_fullsized":"2024-03-26T03:01:23.938280+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2573,"log_dups_size":0,"ondisk_log_size":2573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:23.625707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":164672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234900+0000","last_change":"2024-03-26T03:00:48.212377+0000","last_active":"2024-03-26T03:01:14.234900+0000","last_peered":"2024-03-26T03:01:14.234900+0000","last_clean":"2024-03-26T03:01:14.234900+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:01:14.234900+0000","last_undegraded":"2024-03-26T03:01:14.234900+0000","last_fullsized":"2024-03-26T03:01:14.234900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:48.212296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:48.212296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:00.206723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225435+0000","last_change":"2024-03-26T03:00:33.621455+0000","last_active":"2024-03-26T03:01:14.225435+0000","last_peered":"2024-03-26T03:01:14.225435+0000","last_clean":"2024-03-26T03:01:14.225435+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:01:14.225435+0000","last_undegraded":"2024-03-26T03:01:14.225435+0000","last_fullsized":"2024-03-26T03:01:14.225435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:33.621377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:33.621377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:33.796224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":120,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227174+0000","last_change":"2024-03-26T03:00:07.927770+0000","last_active":"2024-03-26T03:01:14.227174+0000","last_peered":"2024-03-26T03:01:14.227174+0000","last_clean":"2024-03-26T03:01:14.227174+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:01:14.227174+0000","last_undegraded":"2024-03-26T03:01:14.227174+0000","last_fullsized":"2024-03-26T03:01:14.227174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:00:07.927648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.927648+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"91'9","reported_seq":123,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.081221+0000","last_change":"2024-03-26T03:00:37.456923+0000","last_active":"2024-03-26T03:01:17.081221+0000","last_peered":"2024-03-26T03:01:17.081221+0000","last_clean":"2024-03-26T03:01:17.081221+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:01:17.081221+0000","last_undegraded":"2024-03-26T03:01:17.081221+0000","last_fullsized":"2024-03-26T03:01:17.081221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:00:37.456826+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:57.760174+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"100'2656","reported_seq":2674,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.907699+0000","last_change":"2024-03-26T03:00:55.434339+0000","last_active":"2024-03-26T03:01:23.907699+0000","last_peered":"2024-03-26T03:01:23.907699+0000","last_clean":"2024-03-26T03:01:23.907699+0000","last_became_active":"2024-03-26T03:00:55.433417+0000","last_became_peered":"2024-03-26T03:00:55.433417+0000","last_unstale":"2024-03-26T03:01:23.907699+0000","last_undegraded":"2024-03-26T03:01:23.907699+0000","last_fullsized":"2024-03-26T03:01:23.907699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2656,"log_dups_size":0,"ondisk_log_size":2656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:58.168181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":169984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238408+0000","last_change":"2024-03-26T03:00:15.672174+0000","last_active":"2024-03-26T03:01:14.238408+0000","last_peered":"2024-03-26T03:01:14.238408+0000","last_clean":"2024-03-26T03:01:14.238408+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:01:14.238408+0000","last_undegraded":"2024-03-26T03:01:14.238408+0000","last_fullsized":"2024-03-26T03:01:14.238408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.672090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.667763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236774+0000","last_change":"2024-03-26T03:00:27.990812+0000","last_active":"2024-03-26T03:01:14.236774+0000","last_peered":"2024-03-26T03:01:14.236774+0000","last_clean":"2024-03-26T03:01:14.236774+0000","last_became_active":"2024-03-26T02:59:00.937450+0000","last_became_peered":"2024-03-26T02:59:00.937450+0000","last_unstale":"2024-03-26T03:01:14.236774+0000","last_undegraded":"2024-03-26T03:01:14.236774+0000","last_fullsized":"2024-03-26T03:01:14.236774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.990730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.990730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:42.775418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234347+0000","last_change":"2024-03-26T03:00:38.251869+0000","last_active":"2024-03-26T03:01:14.234347+0000","last_peered":"2024-03-26T03:01:14.234347+0000","last_clean":"2024-03-26T03:01:14.234347+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:01:14.234347+0000","last_undegraded":"2024-03-26T03:01:14.234347+0000","last_fullsized":"2024-03-26T03:01:14.234347+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:38.251782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:38.251782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:41.653096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":118,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227826+0000","last_change":"2024-03-26T03:00:05.879783+0000","last_active":"2024-03-26T03:01:14.227826+0000","last_peered":"2024-03-26T03:01:14.227826+0000","last_clean":"2024-03-26T03:01:14.227826+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:01:14.227826+0000","last_undegraded":"2024-03-26T03:01:14.227826+0000","last_fullsized":"2024-03-26T03:01:14.227826+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:05.879672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.879672+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:21.218794+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"100'2571","reported_seq":2588,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.913607+0000","last_change":"2024-03-26T03:00:55.436147+0000","last_active":"2024-03-26T03:01:23.913607+0000","last_peered":"2024-03-26T03:01:23.913607+0000","last_clean":"2024-03-26T03:01:23.913607+0000","last_became_active":"2024-03-26T03:00:55.435784+0000","last_became_peered":"2024-03-26T03:00:55.435784+0000","last_unstale":"2024-03-26T03:01:23.913607+0000","last_undegraded":"2024-03-26T03:01:23.913607+0000","last_fullsized":"2024-03-26T03:01:23.913607+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2571,"log_dups_size":0,"ondisk_log_size":2571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:03.444240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168493056,"num_objects":2571,"num_object_clones":0,"num_object_copies":7713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2571,"num_write_kb":164544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236420+0000","last_change":"2024-03-26T03:00:40.962821+0000","last_active":"2024-03-26T03:01:14.236420+0000","last_peered":"2024-03-26T03:01:14.236420+0000","last_clean":"2024-03-26T03:01:14.236420+0000","last_became_active":"2024-03-26T02:59:02.944279+0000","last_became_peered":"2024-03-26T02:59:02.944279+0000","last_unstale":"2024-03-26T03:01:14.236420+0000","last_undegraded":"2024-03-26T03:01:14.236420+0000","last_fullsized":"2024-03-26T03:01:14.236420+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.962737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:50.772443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":121,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:18.932418+0000","last_change":"2024-03-26T03:00:30.182479+0000","last_active":"2024-03-26T03:01:18.932418+0000","last_peered":"2024-03-26T03:01:18.932418+0000","last_clean":"2024-03-26T03:01:18.932418+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:01:18.932418+0000","last_undegraded":"2024-03-26T03:01:18.932418+0000","last_fullsized":"2024-03-26T03:01:18.932418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:30.182384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:30.182384+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:55.144083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229774+0000","last_change":"2024-03-26T03:00:52.439831+0000","last_active":"2024-03-26T03:01:14.229774+0000","last_peered":"2024-03-26T03:01:14.229774+0000","last_clean":"2024-03-26T03:01:14.229774+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:01:14.229774+0000","last_undegraded":"2024-03-26T03:01:14.229774+0000","last_fullsized":"2024-03-26T03:01:14.229774+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.439710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.439710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:04.532141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":113,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.231603+0000","last_change":"2024-03-26T03:00:28.371034+0000","last_active":"2024-03-26T03:01:14.231603+0000","last_peered":"2024-03-26T03:01:14.231603+0000","last_clean":"2024-03-26T03:01:14.231603+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:01:14.231603+0000","last_undegraded":"2024-03-26T03:01:14.231603+0000","last_fullsized":"2024-03-26T03:01:14.231603+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:28.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.370925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:35.020056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"100'2590","reported_seq":2606,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.927022+0000","last_change":"2024-03-26T03:00:55.439219+0000","last_active":"2024-03-26T03:01:23.927022+0000","last_peered":"2024-03-26T03:01:23.927022+0000","last_clean":"2024-03-26T03:01:23.927022+0000","last_became_active":"2024-03-26T03:00:55.438925+0000","last_became_peered":"2024-03-26T03:00:55.438925+0000","last_unstale":"2024-03-26T03:01:23.927022+0000","last_undegraded":"2024-03-26T03:01:23.927022+0000","last_fullsized":"2024-03-26T03:01:23.927022+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:00.888584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227659+0000","last_change":"2024-03-26T02:59:02.945294+0000","last_active":"2024-03-26T03:01:14.227659+0000","last_peered":"2024-03-26T03:01:14.227659+0000","last_clean":"2024-03-26T03:01:14.227659+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:01:14.227659+0000","last_undegraded":"2024-03-26T03:01:14.227659+0000","last_fullsized":"2024-03-26T03:01:14.227659+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.221580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":123,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:18.932476+0000","last_change":"2024-03-26T03:00:20.452771+0000","last_active":"2024-03-26T03:01:18.932476+0000","last_peered":"2024-03-26T03:01:18.932476+0000","last_clean":"2024-03-26T03:01:18.932476+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:01:18.932476+0000","last_undegraded":"2024-03-26T03:01:18.932476+0000","last_fullsized":"2024-03-26T03:01:18.932476+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.452648+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:45.711803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.230150+0000","last_change":"2024-03-26T02:58:56.930656+0000","last_active":"2024-03-26T03:01:14.230150+0000","last_peered":"2024-03-26T03:01:14.230150+0000","last_clean":"2024-03-26T03:01:14.230150+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:01:14.230150+0000","last_undegraded":"2024-03-26T03:01:14.230150+0000","last_fullsized":"2024-03-26T03:01:14.230150+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:58:55.892771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":102,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227514+0000","last_change":"2024-03-26T03:00:08.975032+0000","last_active":"2024-03-26T03:01:14.227514+0000","last_peered":"2024-03-26T03:01:14.227514+0000","last_clean":"2024-03-26T03:01:14.227514+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:01:14.227514+0000","last_undegraded":"2024-03-26T03:01:14.227514+0000","last_fullsized":"2024-03-26T03:01:14.227514+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:00:08.974874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.974874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:34.309201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"100'2550","reported_seq":2568,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.906003+0000","last_change":"2024-03-26T03:00:55.433696+0000","last_active":"2024-03-26T03:01:23.906003+0000","last_peered":"2024-03-26T03:01:23.906003+0000","last_clean":"2024-03-26T03:01:23.906003+0000","last_became_active":"2024-03-26T03:00:55.433381+0000","last_became_peered":"2024-03-26T03:00:55.433381+0000","last_unstale":"2024-03-26T03:01:23.906003+0000","last_undegraded":"2024-03-26T03:01:23.906003+0000","last_fullsized":"2024-03-26T03:01:23.906003+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:19.173957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167116800,"num_objects":2550,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":163200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236489+0000","last_change":"2024-03-26T03:00:37.323420+0000","last_active":"2024-03-26T03:01:14.236489+0000","last_peered":"2024-03-26T03:01:14.236489+0000","last_clean":"2024-03-26T03:01:14.236489+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:01:14.236489+0000","last_undegraded":"2024-03-26T03:01:14.236489+0000","last_fullsized":"2024-03-26T03:01:14.236489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:37.323337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:37.323337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:47.824493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229952+0000","last_change":"2024-03-26T03:00:45.397106+0000","last_active":"2024-03-26T03:01:14.229952+0000","last_peered":"2024-03-26T03:01:14.229952+0000","last_clean":"2024-03-26T03:01:14.229952+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:01:14.229952+0000","last_undegraded":"2024-03-26T03:01:14.229952+0000","last_fullsized":"2024-03-26T03:01:14.229952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.397023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:13.829539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234549+0000","last_change":"2024-03-26T03:00:00.526049+0000","last_active":"2024-03-26T03:01:14.234549+0000","last_peered":"2024-03-26T03:01:14.234549+0000","last_clean":"2024-03-26T03:01:14.234549+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:01:14.234549+0000","last_undegraded":"2024-03-26T03:01:14.234549+0000","last_fullsized":"2024-03-26T03:01:14.234549+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:00.525932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:00.525932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.138757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":115,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.227725+0000","last_change":"2024-03-26T03:00:52.286513+0000","last_active":"2024-03-26T03:01:14.227725+0000","last_peered":"2024-03-26T03:01:14.227725+0000","last_clean":"2024-03-26T03:01:14.227725+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:01:14.227725+0000","last_undegraded":"2024-03-26T03:01:14.227725+0000","last_fullsized":"2024-03-26T03:01:14.227725+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:52.286392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.286392+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:00.045442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"100'2520","reported_seq":2537,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.935998+0000","last_change":"2024-03-26T03:00:55.435579+0000","last_active":"2024-03-26T03:01:23.935998+0000","last_peered":"2024-03-26T03:01:23.935998+0000","last_clean":"2024-03-26T03:01:23.935998+0000","last_became_active":"2024-03-26T03:00:55.435266+0000","last_became_peered":"2024-03-26T03:00:55.435266+0000","last_unstale":"2024-03-26T03:01:23.935998+0000","last_undegraded":"2024-03-26T03:01:23.935998+0000","last_fullsized":"2024-03-26T03:01:23.935998+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2520,"log_dups_size":0,"ondisk_log_size":2520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:19.533380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":161280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.234196+0000","last_change":"2024-03-26T02:59:02.945728+0000","last_active":"2024-03-26T03:01:14.234196+0000","last_peered":"2024-03-26T03:01:14.234196+0000","last_clean":"2024-03-26T03:01:14.234196+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:01:14.234196+0000","last_undegraded":"2024-03-26T03:01:14.234196+0000","last_fullsized":"2024-03-26T03:01:14.234196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:15.193442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.235684+0000","last_change":"2024-03-26T03:00:26.426212+0000","last_active":"2024-03-26T03:01:14.235684+0000","last_peered":"2024-03-26T03:01:14.235684+0000","last_clean":"2024-03-26T03:01:14.235684+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:01:14.235684+0000","last_undegraded":"2024-03-26T03:01:14.235684+0000","last_fullsized":"2024-03-26T03:01:14.235684+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.426122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:26.618075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.235914+0000","last_change":"2024-03-26T02:59:58.637319+0000","last_active":"2024-03-26T03:01:14.235914+0000","last_peered":"2024-03-26T03:01:14.235914+0000","last_clean":"2024-03-26T03:01:14.235914+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:01:14.235914+0000","last_undegraded":"2024-03-26T03:01:14.235914+0000","last_fullsized":"2024-03-26T03:01:14.235914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:58.637207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T02:59:58.637207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:18.247095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":159,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.225811+0000","last_change":"2024-03-26T03:00:36.550373+0000","last_active":"2024-03-26T03:01:14.225811+0000","last_peered":"2024-03-26T03:01:14.225811+0000","last_clean":"2024-03-26T03:01:14.225811+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:01:14.225811+0000","last_undegraded":"2024-03-26T03:01:14.225811+0000","last_fullsized":"2024-03-26T03:01:14.225811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:00:36.550261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:36.550261+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:39.737738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"100'2666","reported_seq":2684,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.907733+0000","last_change":"2024-03-26T03:00:55.434949+0000","last_active":"2024-03-26T03:01:23.907733+0000","last_peered":"2024-03-26T03:01:23.907733+0000","last_clean":"2024-03-26T03:01:23.907733+0000","last_became_active":"2024-03-26T03:00:55.434651+0000","last_became_peered":"2024-03-26T03:00:55.434651+0000","last_unstale":"2024-03-26T03:01:23.907733+0000","last_undegraded":"2024-03-26T03:01:23.907733+0000","last_fullsized":"2024-03-26T03:01:23.907733+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2666,"log_dups_size":0,"ondisk_log_size":2666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:56.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174718976,"num_objects":2666,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":170624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":80,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236065+0000","last_change":"2024-03-26T03:00:48.304407+0000","last_active":"2024-03-26T03:01:14.236065+0000","last_peered":"2024-03-26T03:01:14.236065+0000","last_clean":"2024-03-26T03:01:14.236065+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:01:14.236065+0000","last_undegraded":"2024-03-26T03:01:14.236065+0000","last_fullsized":"2024-03-26T03:01:14.236065+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:00:48.304319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:59.635463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.238037+0000","last_change":"2024-03-26T03:00:40.307781+0000","last_active":"2024-03-26T03:01:14.238037+0000","last_peered":"2024-03-26T03:01:14.238037+0000","last_clean":"2024-03-26T03:01:14.238037+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:01:14.238037+0000","last_undegraded":"2024-03-26T03:01:14.238037+0000","last_fullsized":"2024-03-26T03:01:14.238037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:40.307697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.307697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:09.089384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.229898+0000","last_change":"2024-03-26T02:59:57.244851+0000","last_active":"2024-03-26T03:01:14.229898+0000","last_peered":"2024-03-26T03:01:14.229898+0000","last_clean":"2024-03-26T03:01:14.229898+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:01:14.229898+0000","last_undegraded":"2024-03-26T03:01:14.229898+0000","last_fullsized":"2024-03-26T03:01:14.229898+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_clean_scrub_stamp":"2024-03-26T02:59:57.244756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:14.738002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":121,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236915+0000","last_change":"2024-03-26T03:00:27.238978+0000","last_active":"2024-03-26T03:01:14.236915+0000","last_peered":"2024-03-26T03:01:14.236915+0000","last_clean":"2024-03-26T03:01:14.236915+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:01:14.236915+0000","last_undegraded":"2024-03-26T03:01:14.236915+0000","last_fullsized":"2024-03-26T03:01:14.236915+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.238859+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:48.977374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"100'2456","reported_seq":2474,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:23.907486+0000","last_change":"2024-03-26T03:00:55.435500+0000","last_active":"2024-03-26T03:01:23.907486+0000","last_peered":"2024-03-26T03:01:23.907486+0000","last_clean":"2024-03-26T03:01:23.907486+0000","last_became_active":"2024-03-26T03:00:55.435115+0000","last_became_peered":"2024-03-26T03:00:55.435115+0000","last_unstale":"2024-03-26T03:01:23.907486+0000","last_undegraded":"2024-03-26T03:01:23.907486+0000","last_fullsized":"2024-03-26T03:01:23.907486+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":2456,"log_dups_size":0,"ondisk_log_size":2456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:10.999417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160956416,"num_objects":2456,"num_object_clones":0,"num_object_copies":7368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2456,"num_write_kb":157184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":79,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236341+0000","last_change":"2024-03-26T03:00:39.931986+0000","last_active":"2024-03-26T03:01:14.236341+0000","last_peered":"2024-03-26T03:01:14.236341+0000","last_clean":"2024-03-26T03:01:14.236341+0000","last_became_active":"2024-03-26T02:59:03.206195+0000","last_became_peered":"2024-03-26T02:59:03.206195+0000","last_unstale":"2024-03-26T03:01:14.236341+0000","last_undegraded":"2024-03-26T03:01:14.236341+0000","last_fullsized":"2024-03-26T03:01:14.236341+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:39.931898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:39.931898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:02:06.683610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":92,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236871+0000","last_change":"2024-03-26T03:00:30.387386+0000","last_active":"2024-03-26T03:01:14.236871+0000","last_peered":"2024-03-26T03:01:14.236871+0000","last_clean":"2024-03-26T03:01:14.236871+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:01:14.236871+0000","last_undegraded":"2024-03-26T03:01:14.236871+0000","last_fullsized":"2024-03-26T03:01:14.236871+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:30.387306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:30.387306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:44.836336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":136,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:14.236152+0000","last_change":"2024-03-26T03:00:23.030834+0000","last_active":"2024-03-26T03:01:14.236152+0000","last_peered":"2024-03-26T03:01:14.236152+0000","last_clean":"2024-03-26T03:01:14.236152+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:01:14.236152+0000","last_undegraded":"2024-03-26T03:01:14.236152+0000","last_fullsized":"2024-03-26T03:01:14.236152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:00:23.030734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.030734+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:24.236468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"71'8","reported_seq":122,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-26T03:01:17.010552+0000","last_change":"2024-03-26T03:00:44.256721+0000","last_active":"2024-03-26T03:01:17.010552+0000","last_peered":"2024-03-26T03:01:17.010552+0000","last_clean":"2024-03-26T03:01:17.010552+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:01:17.010552+0000","last_undegraded":"2024-03-26T03:01:17.010552+0000","last_fullsized":"2024-03-26T03:01:17.010552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:44.256622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:44.256622+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:01:49.146703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2715877376,"num_objects":41441,"num_object_clones":0,"num_object_copies":124323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41441,"num_write_kb":2652224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8148877312,"data_stored":8148877312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41441,"ondisk_log_size":41441,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103202,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":834504,"kb_used_data":807488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92910648,"statfs":{"total":95995035648,"available":95140503552,"internally_reserved":0,"allocated":826867712,"data_stored":826489854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430246,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849536,"kb_used_data":822520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92895616,"statfs":{"total":95995035648,"available":95125110784,"internally_reserved":0,"allocated":842260480,"data_stored":841890196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017632,"kb_used_data":968408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92727520,"statfs":{"total":95995035648,"available":94952980480,"internally_reserved":0,"allocated":991649792,"data_stored":991248016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370320,"kb_used_data":1325256,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92374832,"statfs":{"total":95995035648,"available":94591827968,"internally_reserved":0,"allocated":1357062144,"data_stored":1356676747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006780,"kb_used_data":975604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92738372,"statfs":{"total":95995035648,"available":94964092928,"internally_reserved":0,"allocated":999018496,"data_stored":998652821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1535128,"kb_used_data":1490064,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92210024,"statfs":{"total":95995035648,"available":94423064576,"internally_reserved":0,"allocated":1525825536,"data_stored":1525430561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":832848,"kb_used_data":801672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92912304,"statfs":{"total":95995035648,"available":95142199296,"internally_reserved":0,"allocated":820912128,"data_stored":820525443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691884,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672832,"kb_used_data":637496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93072320,"statfs":{"total":95995035648,"available":95306055680,"internally_reserved":0,"allocated":652795904,"data_stored":652425154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1357512704,"data_stored":1357512704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1008926720,"data_stored":1008926720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1024065536,"data_stored":1024065536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1540161536,"data_stored":1540161536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":847642624,"data_stored":847642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":673185792,"data_stored":673185792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":844300288,"data_stored":844300288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":853082112,"data_stored":853082112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:01:34.410 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T03:01:34.410 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-03-26T03:01:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:34.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:34.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:01:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:01:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:01:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:01:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:01:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:01:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 6.6 scrub starts 2024-03-26T03:01:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:34 smithi192 ceph-mon[33110]: 6.6 scrub ok 2024-03-26T03:01:34.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:34.578 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:34.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:34.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:34.742 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:34.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:34.778 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:01:34.846 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:34.913 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:34.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:35.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:35.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:35.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:35.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:35.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 32 16 57973 57957 113.165 77.125 0.00764653 0.00883109 2024-03-26T03:01:35.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:35.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:35.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: pgmap v85: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 4.1c scrub starts 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 4.1c scrub ok 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 6.13 scrub starts 2024-03-26T03:01:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[29666]: 6.13 scrub ok 2024-03-26T03:01:35.360 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:35 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T03:01:35.360 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:35 smithi066 systemd[1]: Stopped Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:01:35.360 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:35 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.232s CPU time. 2024-03-26T03:01:35.360 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:35 smithi066 systemd[1]: Starting Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: pgmap v85: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:01:35.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:01:35.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:01:35.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:01:35.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 4.1c scrub starts 2024-03-26T03:01:35.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 4.1c scrub ok 2024-03-26T03:01:35.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 6.13 scrub starts 2024-03-26T03:01:35.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:35 smithi066 ceph-mon[37920]: 6.13 scrub ok 2024-03-26T03:01:35.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:35.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:01:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: pgmap v85: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 4.1c scrub starts 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 4.1c scrub ok 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 6.13 scrub starts 2024-03-26T03:01:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:35 smithi192 ceph-mon[33110]: 6.13 scrub ok 2024-03-26T03:01:35.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:35.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:35.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.752 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:35.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:35.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:35.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:35.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:35.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:36.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:36.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:36.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:36.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:36.259 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 33 16 59402 59386 112.442 89.3125 0.0135828 0.00888834 2024-03-26T03:01:36.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:36.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:01:36.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:01:36.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:01:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:01:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:01:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:01:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:01:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:36 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:01:36.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:36.426 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:36 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:01:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:36 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:01:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:36 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:01:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:36 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:01:36.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:36.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:36.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.711 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:36.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:36.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:36.841 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:36.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:36.944 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:36.965 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:37.019 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:37.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:37.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:37.227 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[29666]: pgmap v86: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T03:01:37.227 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:37.227 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:37.227 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[29666]: 4.0 scrub starts 2024-03-26T03:01:37.228 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[29666]: 4.0 scrub ok 2024-03-26T03:01:37.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[37920]: pgmap v86: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T03:01:37.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:37.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:37.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[37920]: 4.0 scrub starts 2024-03-26T03:01:37.228 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:37 smithi066 ceph-mon[37920]: 4.0 scrub ok 2024-03-26T03:01:37.228 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:37 smithi066 podman[80342]: 2024-03-26 03:01:36.972934211 +0000 UTC m=+0.696536880 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T03:01:37.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:37.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 34 16 60771 60755 111.651 85.5625 0.00388229 0.00894967 2024-03-26T03:01:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.302 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:37.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:37.403 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:37.449 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:37 smithi192 ceph-mon[33110]: pgmap v86: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T03:01:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:37 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:37 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:37 smithi192 ceph-mon[33110]: 4.0 scrub starts 2024-03-26T03:01:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:37 smithi192 ceph-mon[33110]: 4.0 scrub ok 2024-03-26T03:01:37.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:37.557 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:37 smithi066 podman[80342]: 2024-03-26 03:01:37.347446338 +0000 UTC m=+1.071048986 volume create 964aad9b7fb2373b952381fac051e75842355c30c924f9a724d89b5df5da6ac6 2024-03-26T03:01:37.569 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:37.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:37.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:37.699 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:01:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.741 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:37.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:37.742 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:01:37.786 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:01:37.829 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:01:37.829 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:37 smithi066 podman[80342]: 2024-03-26 03:01:37.683119568 +0000 UTC m=+1.406722215 container create 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:37.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:37.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:37.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:37.873 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:01:37.915 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:01:37.958 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:01:37.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:38.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:01:38.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:38.004 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:01:38.005 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:01:38.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:38.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[29666]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T03:01:38.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[29666]: 6.8 scrub starts 2024-03-26T03:01:38.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[29666]: 6.8 scrub ok 2024-03-26T03:01:38.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/826737340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:38.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[29666]: 4.10 scrub starts 2024-03-26T03:01:38.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[29666]: 4.10 scrub ok 2024-03-26T03:01:38.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[37920]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T03:01:38.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[37920]: 6.8 scrub starts 2024-03-26T03:01:38.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[37920]: 6.8 scrub ok 2024-03-26T03:01:38.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/826737340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:38.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[37920]: 4.10 scrub starts 2024-03-26T03:01:38.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-mon[37920]: 4.10 scrub ok 2024-03-26T03:01:38.148 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:38.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:38.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 35 16 62033 62017 110.715 78.875 0.00844042 0.00902674 2024-03-26T03:01:38.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:38.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:38.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:38 smithi192 ceph-mon[33110]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T03:01:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:38 smithi192 ceph-mon[33110]: 6.8 scrub starts 2024-03-26T03:01:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:38 smithi192 ceph-mon[33110]: 6.8 scrub ok 2024-03-26T03:01:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/826737340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:38 smithi192 ceph-mon[33110]: 4.10 scrub starts 2024-03-26T03:01:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:38 smithi192 ceph-mon[33110]: 4.10 scrub ok 2024-03-26T03:01:38.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:38.564 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:38.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:38.715 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:38.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:38.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 podman[80342]: 2024-03-26 03:01:38.56912989 +0000 UTC m=+2.292732543 container init 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 podman[80342]: 2024-03-26 03:01:38.574882624 +0000 UTC m=+2.298485271 container start 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.603Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.603Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.604Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.66 port=9094 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.606Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.652Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.653Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.663Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T03:01:38.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:38.663Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T03:01:38.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:38.997 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:38.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:39.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:39.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:39.227 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:39.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 36 16 63218 63202 109.696 74.0625 0.00634736 0.00910769 2024-03-26T03:01:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.271 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:39.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:39.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: pgmap v88: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 645 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: 5.17 scrub starts 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: 5.17 scrub ok 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:39.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:39.359 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 bash[80342]: 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b 2024-03-26T03:01:39.359 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:38 smithi066 systemd[1]: Started Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: pgmap v88: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 645 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: 5.17 scrub starts 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: 5.17 scrub ok 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:39.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:39.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:39.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:01:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: pgmap v88: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 645 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: 5.17 scrub starts 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: 5.17 scrub ok 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:39.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:39.515 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.559 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:39.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:39.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:39.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:39.814 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.855 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:39.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:39.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:39.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:39.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.14841.0:62858 7.b e105/96)", 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.901552+0000", 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.001697439, 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001723593, 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.054 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901552+0000", 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901552+0000", 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901553+0000", 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5900000000000005e-07 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901557+0000", 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3950000000000003e-06 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901565+0000", 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0139999999999998e-06 2024-03-26T03:01:40.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901572+0000", 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.105e-06 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.14841.0:62858 7.b e105/96)", 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.902666+0000", 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.00058390700000000005, 2024-03-26T03:01:40.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00068616, 2024-03-26T03:01:40.058 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.058 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.058 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:01:40.058 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.061 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902666+0000", 2024-03-26T03:01:40.061 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902666+0000", 2024-03-26T03:01:40.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902667+0000", 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.046e-06 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902671+0000", 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5469999999999998e-06 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902751+0000", 2024-03-26T03:01:40.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9887000000000003e-05 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902759+0000", 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4590000000000001e-06 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:62856 7.b 7:dd866162:::benchmark_data_smithi066_76917_object62855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.895048+0000", 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0082012360000000006, 2024-03-26T03:01:40.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0083703649999999994, 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62856 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895048+0000", 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895048+0000", 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895051+0000", 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.305e-06 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895078+0000", 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7508000000000002e-05 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.068 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895093+0000", 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4959e-05 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895102+0000", 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3030000000000005e-06 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895169+0000", 2024-03-26T03:01:40.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6639999999999999e-05 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895297+0000", 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000128107 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.895433+0000", 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000136277 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:01:40.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.897718+0000", 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022848230000000001 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.898812+0000", 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001094212 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:62858 7.b 7:d27bbe02:::benchmark_data_smithi066_76917_object62857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.899136+0000", 2024-03-26T03:01:40.071 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0041136030000000004, 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043994960000000001, 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62858 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899136+0000", 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899136+0000", 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899138+0000", 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3489999999999999e-06 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899164+0000", 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5826e-05 2024-03-26T03:01:40.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899182+0000", 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7445e-05 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899191+0000", 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2390000000000008e-06 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899253+0000", 2024-03-26T03:01:40.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2205999999999994e-05 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899412+0000", 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015908200000000001 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899908+0000", 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00049554799999999995 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:62860 7.b 7:dee19304:::benchmark_data_smithi066_76917_object62859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.899804+0000", 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.003445849, 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038942970000000001, 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62860 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899804+0000", 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899804+0000", 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899805+0000", 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5030000000000001e-06 2024-03-26T03:01:40.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899832+0000", 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7248000000000001e-05 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899849+0000", 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6334e-05 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.899858+0000", 2024-03-26T03:01:40.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8210000000000008e-06 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.902653+0000", 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0027954070000000002 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.903188+0000", 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00053457000000000005 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:01:40.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.903379+0000", 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000191545 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:62857 7.2 e105/96 7:4c8bb0c1:::benchmark_data_smithi066_76917_object62856:head v 105'3931, mlcod=105'3929)", 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.900041+0000", 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0032081610000000002, 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003744388, 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900041+0000", 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900041+0000", 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.086 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.086 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900043+0000", 2024-03-26T03:01:40.086 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2610000000000001e-06 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900065+0000", 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.19e-05 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900072+0000", 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.943e-06 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900079+0000", 2024-03-26T03:01:40.087 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3729999999999998e-06 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900505+0000", 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00042665699999999999 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.088 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900523+0000", 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7436999999999999e-05 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.14841.0:62856 7.b e105/96)", 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.900987+0000", 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0022626619999999999, 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002863427, 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.089 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900987+0000", 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900987+0000", 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.090 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900988+0000", 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.158e-06 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.900994+0000", 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5509999999999999e-06 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901003+0000", 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4879999999999996e-06 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901010+0000", 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7989999999999996e-06 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:62859 7.0 e105/96 7:0aa6c6d5:::benchmark_data_smithi066_76917_object62858:head v 105'3984, mlcod=105'3981)", 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:38.901021+0000", 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0022285450000000002, 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002895693, 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:01:40.092 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:01:40.093 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:01:40.093 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:01:40.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.093 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:01:40.093 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901021+0000", 2024-03-26T03:01:40.093 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901021+0000", 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901021+0000", 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7899999999999999e-07 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:01:40.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901054+0000", 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3099000000000003e-05 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901060+0000", 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7490000000000004e-06 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901065+0000", 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1229999999999999e-06 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901145+0000", 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9969000000000006e-05 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:38.901164+0000", 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.861e-05 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 8 2024-03-26T03:01:40.096 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:01:40.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.141 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:40.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:40.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:40.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 37 16 64309 64293 108.575 68.1875 0.00323077 0.00917354 2024-03-26T03:01:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.272 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:40.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:40.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:40.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[29666]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T03:01:40.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:01:40.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:01:40.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[29666]: Reconfiguring daemon prometheus.a on smithi192 2024-03-26T03:01:40.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:40.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[37920]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T03:01:40.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:01:40.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:01:40.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-mon[37920]: Reconfiguring daemon prometheus.a on smithi192 2024-03-26T03:01:40.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:40 smithi192 ceph-mon[33110]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T03:01:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:40 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:01:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:40 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:01:40.388 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:40 smithi192 ceph-mon[33110]: Reconfiguring daemon prometheus.a on smithi192 2024-03-26T03:01:40.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:40.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:40.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:40.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:40.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:40.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:40.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:40.858 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:40 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:40.607Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000438412s 2024-03-26T03:01:40.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:40.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:41.053 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:41.099 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:41.201 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: 6.16 scrub starts 2024-03-26T03:01:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: 6.16 scrub ok 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: pgmap v89: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:01:41.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:41 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:01:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.240 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:41.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:41.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 38 16 65514 65498 107.699 75.3125 0.0184419 0.00927938 2024-03-26T03:01:41.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: 6.16 scrub starts 2024-03-26T03:01:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: 6.16 scrub ok 2024-03-26T03:01:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: pgmap v89: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-26T03:01:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: 6.16 scrub starts 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: 6.16 scrub ok 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: pgmap v89: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:01:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:41 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:01:41.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:41.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:41.517 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:41.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:41.651 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:41.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:41.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:41.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:41.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:41.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:41.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:42.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:42.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:42.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:42.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:42.260 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 39 15 66780 66765 106.968 79.1875 0.00680698 0.00934375 2024-03-26T03:01:42.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:42.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:42.433 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.463 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:42.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:01:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:01:42.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:01:42.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:01:42.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T03:01:42.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: 5.1 scrub starts 2024-03-26T03:01:42.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:42 smithi192 ceph-mon[33110]: 5.1 scrub ok 2024-03-26T03:01:42.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:42.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:01:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:01:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: 5.1 scrub starts 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[29666]: 5.1 scrub ok 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T03:01:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: 5.1 scrub starts 2024-03-26T03:01:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:42 smithi066 ceph-mon[37920]: 5.1 scrub ok 2024-03-26T03:01:42.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:42.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:42.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:42.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:42.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:42.962 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:43.007 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:43.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:43.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:43.237 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 systemd[1]: Stopping Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: pgmap v91: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:01:43.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:01:43.248 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:43.261 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:01:43.261390+0000 min lat: 0.00124684 max lat: 0.19348 avg lat: 0.00941539 2024-03-26T03:01:43.261 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:01:43.261 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 40 16 67966 67950 106.144 74.0625 0.0123288 0.00941539 2024-03-26T03:01:43.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:43.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:43.445 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: pgmap v91: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:01:43.505 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:01:43.506 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:01:43.506 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:01:43.506 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:01:43.506 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:01:43.506 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: pgmap v91: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-26T03:01:43.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:01:43.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:01:43.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:01:43.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:01:43.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:01:43.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:43 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:01:43.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:43.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:43.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:43.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.494Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.495Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.495Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T03:01:43.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[61971]: ts=2024-03-26T03:01:43.495Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T03:01:43.814 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 podman[69048]: 2024-03-26 03:01:43.504240166 +0000 UTC m=+0.161956978 container died c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:43.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:43.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:43.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:43.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:43.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:44.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:44.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:44.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:44.129 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 podman[69048]: 2024-03-26 03:01:43.93649412 +0000 UTC m=+0.594210948 container cleanup c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:44.130 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:43 smithi192 bash[69048]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T03:01:44.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:01:44.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:01:44.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:01:44.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:01:44.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[29666]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T03:01:44.193 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:01:44.193 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:01:44.193 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:01:44.193 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:01:44.193 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:44 smithi066 ceph-mon[37920]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T03:01:44.215 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:44.239 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:01:44.262 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 41 16 69251 69235 105.511 80.3125 0.0114812 0.00947227 2024-03-26T03:01:44.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:44.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:44.373 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:44.431 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:44.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:44 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:01:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:44 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:01:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:44 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:01:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:44 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:01:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:44 smithi192 ceph-mon[33110]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T03:01:44.488 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:44 smithi192 podman[69059]: 2024-03-26 03:01:44.27184312 +0000 UTC m=+0.762835818 container remove c81b558debf54f675133fd3814d8f65387d6f55a8107775c7ae5a1840fa7fe9e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:44.532 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:44.588 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:44.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:44.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:44.745 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:44.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:44.829 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:44 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T03:01:44.830 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:44 smithi192 systemd[1]: Stopped Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:01:44.830 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:44 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.317s CPU time. 2024-03-26T03:01:44.830 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:44 smithi192 systemd[1]: Starting Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:01:44.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:44.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:44.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:44.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:45.012 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:45.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:45.130 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:44 smithi192 podman[69227]: 2024-03-26 03:01:44.734885855 +0000 UTC m=+0.025649611 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T03:01:45.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:45.223 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: pgmap v93: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T03:01:45.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:45.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:45.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:01:45.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:01:45.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:01:45.224 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: pgmap v93: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:01:45.226 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:45 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:01:45.240 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:45.262 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 42 16 70330 70314 104.605 67.4375 0.0122874 0.00955386 2024-03-26T03:01:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.302 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:45.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:45.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:01:45.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:45.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: pgmap v93: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T03:01:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:01:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:01:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:01:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:45 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:01:45.530 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.589 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:45.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:45.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:45.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:45.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:45.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:45.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:45.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:45.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:46.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:46.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:46.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:46.263 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 43 16 71339 71323 103.638 63.0625 0.0107457 0.00964309 2024-03-26T03:01:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.264 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:46.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:46.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-mon[33110]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T03:01:46.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-mon[33110]: 4.1b scrub starts 2024-03-26T03:01:46.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-mon[33110]: 4.1b scrub ok 2024-03-26T03:01:46.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-mon[33110]: 2.1 scrub starts 2024-03-26T03:01:46.335 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-mon[33110]: 2.1 scrub ok 2024-03-26T03:01:46.336 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 podman[69227]: 2024-03-26 03:01:46.057160282 +0000 UTC m=+1.347924028 container create 786874bf8abca698d9681bb9f305d45f8fa73f84f1f12704743c74265618151d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:46.336 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 podman[69227]: 2024-03-26 03:01:46.335240422 +0000 UTC m=+1.626004185 container init 786874bf8abca698d9681bb9f305d45f8fa73f84f1f12704743c74265618151d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:46.364 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:46.410 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:46.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:46.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:46.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[29666]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T03:01:46.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[29666]: 4.1b scrub starts 2024-03-26T03:01:46.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[29666]: 4.1b scrub ok 2024-03-26T03:01:46.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[29666]: 2.1 scrub starts 2024-03-26T03:01:46.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[29666]: 2.1 scrub ok 2024-03-26T03:01:46.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[37920]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T03:01:46.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[37920]: 4.1b scrub starts 2024-03-26T03:01:46.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[37920]: 4.1b scrub ok 2024-03-26T03:01:46.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[37920]: 2.1 scrub starts 2024-03-26T03:01:46.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-mon[37920]: 2.1 scrub ok 2024-03-26T03:01:46.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.698 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:46.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:46.738 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 podman[69227]: 2024-03-26 03:01:46.340076979 +0000 UTC m=+1.630840735 container start 786874bf8abca698d9681bb9f305d45f8fa73f84f1f12704743c74265618151d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:01:46.738 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.374Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.374Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.374Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi192 (none))" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.374Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.374Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.376Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.377Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.380Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.380Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.381Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.381Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.2µs 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.382Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.382Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T03:01:46.739 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.382Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.382Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=42.564µs wal_replay_duration=879.519µs wbl_replay_duration=125ns total_replay_duration=962.183µs 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.385Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.385Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.385Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.418Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=33.090344ms db_storage=1.157µs remote_storage=1.487µs web_handler=672ns query_engine=810ns scrape=897.079µs scrape_sd=186.493µs notify=16.236µs notify_sd=45.383µs rules=31.460389ms tracing=8.031µs 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.418Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:01:46.418Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 bash[69227]: 786874bf8abca698d9681bb9f305d45f8fa73f84f1f12704743c74265618151d 2024-03-26T03:01:46.740 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:01:46 smithi192 systemd[1]: Started Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:01:46.789 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:46.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:46.827 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STOPPING 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STOPPED 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STARTING 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Serving on http://:::9283 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STARTED 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STOPPING 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STOPPED 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STARTING 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Serving on http://:::9283 2024-03-26T03:01:46.828 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STARTED 2024-03-26T03:01:46.829 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STOPPING 2024-03-26T03:01:46.829 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:01:46.829 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STOPPED 2024-03-26T03:01:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:46.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:46.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:46.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:46.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:47.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:47.105 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STARTING 2024-03-26T03:01:47.105 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Serving on http://:::9283 2024-03-26T03:01:47.105 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:46 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:01:46] ENGINE Bus STARTED 2024-03-26T03:01:47.190 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:47.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:47.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 44 15 72414 72399 102.808 67.25 0.00718937 0.00972178 2024-03-26T03:01:47.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:47.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:47.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: pgmap v95: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi066.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:01:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: pgmap v95: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:01:47.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi066.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:01:47.362 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:47.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:47.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:47.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: pgmap v95: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi066.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:01:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:47.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:01:47.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:47.615 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:47.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:47.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:47.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:47.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:47.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:47.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:47.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:47.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:48.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:48.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:48.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:48.259 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:48.264 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 45 16 73498 73482 102.027 67.6875 0.0182829 0.00979469 2024-03-26T03:01:48.265 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 7 pg_num to 26 2024-03-26T03:01:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.311 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:48.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:48.412 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:48.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:48.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi066.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:01:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1577133459' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi066.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:01:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:01:48.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1577133459' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T03:01:48.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:48.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi066.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi192.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='mon.0 v1:172.21.15.66:0/2707218205' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi192.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:01:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:48 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1577133459' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-26T03:01:48.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:48.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:48.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:48.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:48.947 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:01:48 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:01:48.610Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004048569s 2024-03-26T03:01:48.983 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.032 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:49.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:49.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:49.175 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:49.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:01:49.264 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 46 16 74541 74525 101.226 65.1875 0.0207953 0.00987291 2024-03-26T03:01:49.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.328 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:49.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:49.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:49.477 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:49.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:01:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:01:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: pgmap v96: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 749 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1577133459' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: 2.e scrub starts 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[29666]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: pgmap v96: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 749 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-26T03:01:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:01:49.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1577133459' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-26T03:01:49.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:01:49.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T03:01:49.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: 2.e scrub starts 2024-03-26T03:01:49.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:49 smithi066 ceph-mon[37920]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T03:01:49.628 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:49.693 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:01:49.693 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: pgmap v96: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 749 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1577133459' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: 2.e scrub starts 2024-03-26T03:01:49.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:49 smithi192 ceph-mon[33110]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T03:01:49.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:49.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:49.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:49.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:49.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:49.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.038 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:50.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:50.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:50.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:50.264 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 47 16 75518 75502 100.372 61.0625 0.0157331 0.00995621 2024-03-26T03:01:50.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:50.284 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:01:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:50.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:50.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:50.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: 2.e scrub ok 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1488551364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: 2.1b scrub starts 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: 2.1b scrub ok 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T03:01:50.440 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: 2.e scrub ok 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1488551364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: 2.1b scrub starts 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: 2.1b scrub ok 2024-03-26T03:01:50.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:50 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T03:01:50.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:50.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:50.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.695 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:50.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: 2.e scrub ok 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1488551364' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: from='client.24787 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: 2.1b scrub starts 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: 2.1b scrub ok 2024-03-26T03:01:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:50 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-26T03:01:50.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:50.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:50.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:50.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:50.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:51.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:51.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:51.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:51.205 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:51.267 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 48 16 75543 75527 98.3137 1.5625 0.00410201 0.00997947 2024-03-26T03:01:51.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:51.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:51.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:51.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:51.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:51.594 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: pgmap v99: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 2.a scrub starts 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 2.a scrub ok 2024-03-26T03:01:51.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 4.17 scrub starts 2024-03-26T03:01:51.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[29666]: 4.17 scrub ok 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: pgmap v99: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T03:01:51.598 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T03:01:51.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:01:51.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:01:51.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 2.a scrub starts 2024-03-26T03:01:51.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 2.a scrub ok 2024-03-26T03:01:51.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 4.17 scrub starts 2024-03-26T03:01:51.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:51 smithi066 ceph-mon[37920]: 4.17 scrub ok 2024-03-26T03:01:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:51.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:51.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: pgmap v99: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: from='client.24787 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 2.a scrub starts 2024-03-26T03:01:51.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 2.a scrub ok 2024-03-26T03:01:51.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 4.17 scrub starts 2024-03-26T03:01:51.656 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:51 smithi192 ceph-mon[33110]: 4.17 scrub ok 2024-03-26T03:01:51.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:51.761 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:51.863 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:51.897 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:51.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:51.998 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:52.048 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:52.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:52.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:52.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 49 16 76577 76561 97.6262 64.625 0.011186 0.0102381 2024-03-26T03:01:52.279 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:52.328 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:52.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:52.456 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:52.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:52.590 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 5.9 deep-scrub starts 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 5.9 deep-scrub ok 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 7.1 scrub starts 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 7.0 scrub starts 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 7.1 scrub ok 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: 7.2 scrub starts 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:52.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 5.9 deep-scrub starts 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 5.9 deep-scrub ok 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 7.1 scrub starts 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 7.0 scrub starts 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 7.1 scrub ok 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: 7.2 scrub starts 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:52.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:52.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:52.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:01:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:01:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T03:01:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 5.9 deep-scrub starts 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 5.9 deep-scrub ok 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 7.1 scrub starts 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 7.0 scrub starts 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 7.1 scrub ok 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: 7.2 scrub starts 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:52.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:52.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:52.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:52.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:52.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:53.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:53.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:53.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:53.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:53.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:53.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 50 16 76630 76614 95.7402 3.3125 0.064666 0.0102414 2024-03-26T03:01:53.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:53.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:53.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:53.475 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:53.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:53.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.8 scrub starts 2024-03-26T03:01:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.9 scrub starts 2024-03-26T03:01:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: pgmap v102: 158 pgs: 10 unknown, 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.0 scrub ok 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.7 scrub starts 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.5 scrub starts 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.8 scrub ok 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.5 scrub ok 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.2 scrub ok 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: 7.3 scrub starts 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[37920]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.8 scrub starts 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.9 scrub starts 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: pgmap v102: 158 pgs: 10 unknown, 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.0 scrub ok 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.7 scrub starts 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.5 scrub starts 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.8 scrub ok 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.5 scrub ok 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.2 scrub ok 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: 7.3 scrub starts 2024-03-26T03:01:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:53.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:53.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-mon[29666]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T03:01:53.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.733 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:53.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.8 scrub starts 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.9 scrub starts 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: pgmap v102: 158 pgs: 10 unknown, 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.0 scrub ok 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.7 scrub starts 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-26T03:01:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.5 scrub starts 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.8 scrub ok 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.5 scrub ok 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.2 scrub ok 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: 7.3 scrub starts 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:01:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:53 smithi192 ceph-mon[33110]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T03:01:53.834 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:53.863 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:53.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:53.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:53.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:54.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:01:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:01:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:01:54.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:54.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:54.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:54.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:54.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:54.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 51 16 76630 76614 93.8632 0 - 0.0102414 2024-03-26T03:01:54.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:54.420 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:54.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:54.521 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:54.595 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:54.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:01:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:01:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 7.9 scrub ok 2024-03-26T03:01:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 7.7 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 7.4 scrub starts 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 7.4 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: 7.3 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[29666]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 7.9 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 7.7 scrub ok 2024-03-26T03:01:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 7.4 scrub starts 2024-03-26T03:01:54.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:01:54.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:01:54.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 7.4 scrub ok 2024-03-26T03:01:54.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: 7.3 scrub ok 2024-03-26T03:01:54.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:54 smithi066 ceph-mon[37920]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T03:01:54.697 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:01:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:01:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 7.9 scrub ok 2024-03-26T03:01:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 7.7 scrub ok 2024-03-26T03:01:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 7.4 scrub starts 2024-03-26T03:01:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:01:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:01:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 7.4 scrub ok 2024-03-26T03:01:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: 7.3 scrub ok 2024-03-26T03:01:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:54 smithi192 ceph-mon[33110]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T03:01:54.747 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:54.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:54.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:54.903 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:54.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:55.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:55.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:55.134 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:55.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:55.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 52 16 76803 76787 92.2664 5.40625 0.00618214 0.0106993 2024-03-26T03:01:55.279 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.324 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:55.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:55.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:55.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:55.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:55.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: 7.6 scrub starts 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: pgmap v105: 158 pgs: 2 remapped+peering, 148 active+clean, 8 unknown; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: 7.6 scrub ok 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:01:55.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: 7.6 scrub starts 2024-03-26T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: pgmap v105: 158 pgs: 2 remapped+peering, 148 active+clean, 8 unknown; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-26T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: 7.6 scrub ok 2024-03-26T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:01:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:55 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:01:55.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:55.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: 7.6 scrub starts 2024-03-26T03:01:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: pgmap v105: 158 pgs: 2 remapped+peering, 148 active+clean, 8 unknown; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-26T03:01:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:01:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:01:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: 7.6 scrub ok 2024-03-26T03:01:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:01:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:55 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:01:55.764 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:01:55.764 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:01:55.764 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:01:55.764 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:01:55.774 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:01:55.774 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:01:55.803 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:01:55.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:55.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:55.841 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:01:55.882 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:01:55.918 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:01:55.921 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:01:55.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:01:55.934 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:55.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:55.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:56.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:56.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:56.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.247 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:56.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:56.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 53 16 76803 76787 90.5258 0 - 0.0106993 2024-03-26T03:01:56.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:56.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:56.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:56.517 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1310652002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1310652002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:56.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:56.651 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:01:56.651 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:01:56.652 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-26T03:01:56.652 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:01:56.652 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:01:56.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-26T03:01:56.659 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1310652002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:01:56.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:56.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:56.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:56.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:01:56.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:56.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:57.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:57.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:57.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:57.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 54 16 76803 76787 88.8492 0 - 0.0106993 2024-03-26T03:01:57.269 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:57.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:57.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:57.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:01:57.538 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:01:57.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:57 smithi066 ceph-mon[37920]: pgmap v107: 158 pgs: 1 active+clean+scrubbing, 10 remapped+peering, 147 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 KiB/s, 0 objects/s recovering 2024-03-26T03:01:57.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:57 smithi066 ceph-mon[29666]: pgmap v107: 158 pgs: 1 active+clean+scrubbing, 10 remapped+peering, 147 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 KiB/s, 0 objects/s recovering 2024-03-26T03:01:57.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:57.658 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:57.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:57 smithi192 ceph-mon[33110]: pgmap v107: 158 pgs: 1 active+clean+scrubbing, 10 remapped+peering, 147 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 KiB/s, 0 objects/s recovering 2024-03-26T03:01:57.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:57.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:57.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:57.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:57.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:58.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:58.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:58.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:58.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:58.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 55 16 76803 76787 87.2341 0 - 0.0106993 2024-03-26T03:01:58.272 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.318 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:58.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:58.419 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.472 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:58.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:58.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:01:58.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:01:58.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:58 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:01:58.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:58 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:01:58.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:58 smithi066 ceph-mon[29666]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T03:01:58.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:58 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:01:58.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:58 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:01:58.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:58 smithi066 ceph-mon[37920]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T03:01:58.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:01:58.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:01:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:58 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:01:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:58 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:01:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:58 smithi192 ceph-mon[33110]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T03:01:58.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:58.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:58.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:01:58.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:58.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:01:59.041 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:01:59.099 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:59.166 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:59.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:59.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 56 16 77475 77459 86.4265 10.5 0.0397706 0.0115646 2024-03-26T03:01:59.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:59.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:59.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:01:59.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:01:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:01:59.380 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:01:59.399 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:01:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:59.470 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:59.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:01:59.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:01:59 smithi066 ceph-mon[37920]: pgmap v109: 158 pgs: 1 active+clean+scrubbing, 10 remapped+peering, 147 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 902 B/s rd, 0 op/s; 34 KiB/s, 0 objects/s recovering 2024-03-26T03:01:59.558 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:01:59 smithi066 ceph-mon[29666]: pgmap v109: 158 pgs: 1 active+clean+scrubbing, 10 remapped+peering, 147 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 902 B/s rd, 0 op/s; 34 KiB/s, 0 objects/s recovering 2024-03-26T03:01:59.571 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:01:59.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:01:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:59.648 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:59.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:01:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:01:59 smithi192 ceph-mon[33110]: pgmap v109: 158 pgs: 1 active+clean+scrubbing, 10 remapped+peering, 147 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 902 B/s rd, 0 op/s; 34 KiB/s, 0 objects/s recovering 2024-03-26T03:01:59.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:01:59.836 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:01:59.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:59.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:01:59.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:01:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:01:59.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:01:59.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:00.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:00.116 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:00.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:00.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:00.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 57 16 78208 78192 85.714 45.8125 0.0054907 0.01166 2024-03-26T03:02:00.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:00.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:00.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.547 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:00.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:00.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:00.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:00.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:00.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:00.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:00.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:00.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:00.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:00.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:01.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:01.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:01.181 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.255 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:01.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:01.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 58 16 79045 79029 85.1381 52.3125 0.0303182 0.0117397 2024-03-26T03:02:01.357 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:01.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:01.460 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:01.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:01.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:01.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:01 smithi066 ceph-mon[29666]: pgmap v110: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 14350/232536 objects degraded (6.171%); 28270/232536 objects misplaced (12.157%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:02:01.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:01 smithi066 ceph-mon[29666]: Health check failed: Degraded data redundancy: 14350/232536 objects degraded (6.171%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:01.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:01 smithi066 ceph-mon[37920]: pgmap v110: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 14350/232536 objects degraded (6.171%); 28270/232536 objects misplaced (12.157%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:02:01.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:01 smithi066 ceph-mon[37920]: Health check failed: Degraded data redundancy: 14350/232536 objects degraded (6.171%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:01.644 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:01.715 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:01 smithi192 ceph-mon[33110]: pgmap v110: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 14350/232536 objects degraded (6.171%); 28270/232536 objects misplaced (12.157%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:02:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:01 smithi192 ceph-mon[33110]: Health check failed: Degraded data redundancy: 14350/232536 objects degraded (6.171%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:01.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:01.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:01.845 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:02:01.845 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:02:01.846 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:02:01.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:01.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:01.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:02.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:02.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:02.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:02.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:02.264 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:02.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:02.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 59 16 79785 79769 84.479 46.25 0.00790837 0.0118307 2024-03-26T03:02:02.366 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:02.425 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:02.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:02.527 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:02.583 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:02 smithi066 ceph-mon[29666]: pgmap v111: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 14350/232536 objects degraded (6.171%); 28270/232536 objects misplaced (12.157%); 15 MiB/s, 241 objects/s recovering 2024-03-26T03:02:02.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:02 smithi066 ceph-mon[37920]: pgmap v111: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 14350/232536 objects degraded (6.171%); 28270/232536 objects misplaced (12.157%); 15 MiB/s, 241 objects/s recovering 2024-03-26T03:02:02.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:02.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:02.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:02 smithi192 ceph-mon[33110]: pgmap v111: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 14350/232536 objects degraded (6.171%); 28270/232536 objects misplaced (12.157%); 15 MiB/s, 241 objects/s recovering 2024-03-26T03:02:02.765 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:02.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:02.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:02.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:02.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:02.998 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:03.053 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:03.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:03.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:03.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:02:03.266437+0000 min lat: 0.00124684 max lat: 6.18392 avg lat: 0.0119122 2024-03-26T03:02:03.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:02:03.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 60 16 80586 80570 83.9054 50.0625 0.0140204 0.0119122 2024-03-26T03:02:03.289 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:03.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:03.442 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:03.494 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:03.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:03.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:03.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:03.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:03.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:03.886 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:03.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:03.930 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:02:03.931 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:02:03.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:04.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:02:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:02:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:02:04.117 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:04.168 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:04.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 61 16 81378 81362 83.3414 49.5 0.00795051 0.0119928 2024-03-26T03:02:04.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:04.321 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:04.386 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:04.423 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:04.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:04.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:04.615 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:04.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:04.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:04.848 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:04.895 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:04.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:04.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:05.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:05.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:05.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:05.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:04 smithi192 ceph-mon[33110]: pgmap v112: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 528 B/s rd, 0 op/s; 14350/235722 objects degraded (6.088%); 27933/235722 objects misplaced (11.850%); 15 MiB/s, 234 objects/s recovering 2024-03-26T03:02:05.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:05.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 62 15 81996 81981 82.6213 38.6875 0.0160982 0.0120977 2024-03-26T03:02:05.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:05.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:04 smithi066 ceph-mon[29666]: pgmap v112: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 528 B/s rd, 0 op/s; 14350/235722 objects degraded (6.088%); 27933/235722 objects misplaced (11.850%); 15 MiB/s, 234 objects/s recovering 2024-03-26T03:02:05.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:04 smithi066 ceph-mon[37920]: pgmap v112: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 528 B/s rd, 0 op/s; 14350/235722 objects degraded (6.088%); 27933/235722 objects misplaced (11.850%); 15 MiB/s, 234 objects/s recovering 2024-03-26T03:02:05.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:05.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:05.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:05.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:05.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:05.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:05.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:05.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:05.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:05.961 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:05.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:06.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:06.114 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:06.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:06.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:06.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 63 16 82673 82657 81.9805 42.25 0.0210873 0.0121919 2024-03-26T03:02:06.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:06 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14350/244236 objects degraded (5.875%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:06.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:06 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14350/244236 objects degraded (5.875%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:06.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:06.413 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:06.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:06 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14350/244236 objects degraded (5.875%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:06.515 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.572 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:06.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:06.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:06.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:06.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:06.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:06.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:06.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:06.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:07.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:07.118 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:07.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:07.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:07.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 64 16 83355 83339 81.3656 42.625 0.0036915 0.0122635 2024-03-26T03:02:07.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:07 smithi066 ceph-mon[29666]: pgmap v113: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 214 op/s; 14350/244236 objects degraded (5.875%); 26476/244236 objects misplaced (10.840%); 20 MiB/s, 317 objects/s recovering 2024-03-26T03:02:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:07 smithi066 ceph-mon[37920]: pgmap v113: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 214 op/s; 14350/244236 objects degraded (5.875%); 26476/244236 objects misplaced (10.840%); 20 MiB/s, 317 objects/s recovering 2024-03-26T03:02:07.377 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:07.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:07 smithi192 ceph-mon[33110]: pgmap v113: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 214 op/s; 14350/244236 objects degraded (5.875%); 26476/244236 objects misplaced (10.840%); 20 MiB/s, 317 objects/s recovering 2024-03-26T03:02:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:07.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:07.606 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:07.658 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:07.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:07.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:07.891 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:07.918 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:07.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:08.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:08.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:08.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:08.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:08.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 65 16 83944 83928 80.6802 36.8125 0.00759383 0.0123854 2024-03-26T03:02:08.279 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:08.310 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:08.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:08.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:08.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:08.569 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:08.671 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:08.723 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:08.825 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:08.861 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:08.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:08.963 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:09.017 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:09.119 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.148 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:09.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:09.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:09.266 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 66 16 84636 84620 80.1131 43.25 0.0230724 0.012477 2024-03-26T03:02:09.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:09.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:09.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:09.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:02:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-mon[33110]: pgmap v114: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 963 B/s rd, 13 MiB/s wr, 201 op/s; 14350/244236 objects degraded (5.875%); 26476/244236 objects misplaced (10.840%); 19 MiB/s, 299 objects/s recovering 2024-03-26T03:02:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:02:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:02:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:09 smithi192 ceph-mon[33110]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T03:02:09.530 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.575 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:09.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:09.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[37920]: pgmap v114: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 963 B/s rd, 13 MiB/s wr, 201 op/s; 14350/244236 objects degraded (5.875%); 26476/244236 objects misplaced (10.840%); 19 MiB/s, 299 objects/s recovering 2024-03-26T03:02:09.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:02:09.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[37920]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[29666]: pgmap v114: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 963 B/s rd, 13 MiB/s wr, 201 op/s; 14350/244236 objects degraded (5.875%); 26476/244236 objects misplaced (10.840%); 19 MiB/s, 299 objects/s recovering 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:09 smithi066 ceph-mon[29666]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T03:02:09.676 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:09.721 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:09.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:09.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:09.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:09.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:09.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:10.084 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:10.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:10.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:10.240 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:10.269 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 67 16 85400 85384 79.6267 47.75 0.0389829 0.0125526 2024-03-26T03:02:10.292 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:10.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:10.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:10.446 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:10.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:10.549 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:10.597 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:10.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:10.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:10.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:10.751 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:10.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:10.853 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:10.904 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:10.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:11.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:11.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:11.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:11.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:11.269 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 68 16 86108 86092 79.1065 44.25 0.0148272 0.0126361 2024-03-26T03:02:11.286 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:11.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:11.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:11.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:11 smithi192 ceph-mon[33110]: pgmap v116: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 721 op/s; 14350/254163 objects degraded (5.646%); 24802/254163 objects misplaced (9.758%); 15 MiB/s, 246 objects/s recovering 2024-03-26T03:02:11.572 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:11.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:11 smithi066 ceph-mon[29666]: pgmap v116: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 721 op/s; 14350/254163 objects degraded (5.646%); 24802/254163 objects misplaced (9.758%); 15 MiB/s, 246 objects/s recovering 2024-03-26T03:02:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:11 smithi066 ceph-mon[37920]: pgmap v116: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 721 op/s; 14350/254163 objects degraded (5.646%); 24802/254163 objects misplaced (9.758%); 15 MiB/s, 246 objects/s recovering 2024-03-26T03:02:11.710 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:11.754 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:11.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:11.908 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:11.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:12.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:12.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:12.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:12.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:12.270 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 69 16 86852 86836 78.634 46.5 0.0296212 0.0127105 2024-03-26T03:02:12.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.326 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:12.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:12.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:12.464 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:12.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:12.595 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:12.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.744 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:12.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:12.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:12.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:12.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:12.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:12.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:13.100 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:13.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:13.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:13.237 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.266 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:13.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:13.269 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 70 16 87262 87246 77.8768 25.625 0.181008 0.0128098 2024-03-26T03:02:13.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:13.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:13.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:13 smithi192 ceph-mon[33110]: pgmap v117: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 721 op/s; 14350/254163 objects degraded (5.646%); 24802/254163 objects misplaced (9.758%); 15 MiB/s, 246 objects/s recovering 2024-03-26T03:02:13.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:13.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:13.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:13 smithi066 ceph-mon[37920]: pgmap v117: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 721 op/s; 14350/254163 objects degraded (5.646%); 24802/254163 objects misplaced (9.758%); 15 MiB/s, 246 objects/s recovering 2024-03-26T03:02:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:13 smithi066 ceph-mon[29666]: pgmap v117: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 721 op/s; 14350/254163 objects degraded (5.646%); 24802/254163 objects misplaced (9.758%); 15 MiB/s, 246 objects/s recovering 2024-03-26T03:02:13.631 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:13.680 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:13.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:13.907 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:13.961 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:13.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:14.062 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:14.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:02:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:02:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:02:14.113 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:14.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:14.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:14.270 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 71 16 87860 87844 77.3064 37.375 0.00969276 0.0129083 2024-03-26T03:02:14.341 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:14.390 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14350/254163 objects degraded (5.646%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:14 smithi192 ceph-mon[33110]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T03:02:14.493 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:14.540 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:14.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14350/254163 objects degraded (5.646%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:14.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:14 smithi066 ceph-mon[29666]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T03:02:14.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14350/254163 objects degraded (5.646%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:14.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:14 smithi066 ceph-mon[37920]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T03:02:14.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:14.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:14.774 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.802 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:14.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:14.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:14.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:14.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:15.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:15.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:15.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:15.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:15.270 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 72 16 88562 88546 76.8421 43.875 0.00813331 0.0130084 2024-03-26T03:02:15.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:15.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:15.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:15.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:15.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:15 smithi192 ceph-mon[33110]: pgmap v119: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 529 op/s; 14350/256938 objects degraded (5.585%); 24356/256938 objects misplaced (9.479%); 13 MiB/s, 207 objects/s recovering 2024-03-26T03:02:15.583 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:15 smithi066 ceph-mon[29666]: pgmap v119: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 529 op/s; 14350/256938 objects degraded (5.585%); 24356/256938 objects misplaced (9.479%); 13 MiB/s, 207 objects/s recovering 2024-03-26T03:02:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:15 smithi066 ceph-mon[37920]: pgmap v119: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 529 op/s; 14350/256938 objects degraded (5.585%); 24356/256938 objects misplaced (9.479%); 13 MiB/s, 207 objects/s recovering 2024-03-26T03:02:15.631 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:15.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:15.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:15.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:15.916 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:15.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:16.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:16.044 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:16.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:16.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:16.270 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 73 16 89263 89247 76.3897 43.8125 0.0182018 0.0130853 2024-03-26T03:02:16.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:16.306 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:16.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:16.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:16.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:16.587 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:16.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:16.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:16.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:16.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:16.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:16.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:16.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:17.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.112 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:17.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:17.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.247 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:17.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:17.270 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 74 16 89935 89919 75.925 42 0.0318972 0.0131652 2024-03-26T03:02:17.348 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:17.400 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:17 smithi192 ceph-mon[33110]: pgmap v120: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 828 op/s; 14350/264090 objects degraded (5.434%); 23010/264090 objects misplaced (8.713%); 20 MiB/s, 316 objects/s recovering 2024-03-26T03:02:17.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:17.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:17 smithi066 ceph-mon[29666]: pgmap v120: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 828 op/s; 14350/264090 objects degraded (5.434%); 23010/264090 objects misplaced (8.713%); 20 MiB/s, 316 objects/s recovering 2024-03-26T03:02:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:17 smithi066 ceph-mon[37920]: pgmap v120: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 828 op/s; 14350/264090 objects degraded (5.434%); 23010/264090 objects misplaced (8.713%); 20 MiB/s, 316 objects/s recovering 2024-03-26T03:02:17.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:17.681 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:17.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:17.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:17.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:17.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:17.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:18.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:18.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:18.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:18.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:18.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:18.270 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 75 16 90707 90691 75.556 48.25 0.0195468 0.0132291 2024-03-26T03:02:18.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:18.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:18.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:18.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:18.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:18.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:18.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:18.742 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:18.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:18.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:18.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:18.974 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:19.024 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:19.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:19.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:19.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:19.271 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 76 16 91387 91371 75.1203 42.5 0.0161341 0.0133056 2024-03-26T03:02:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.312 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:19.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:19.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.461 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:19.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:19.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:19 smithi192 ceph-mon[33110]: pgmap v121: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 576 B/s rd, 28 MiB/s wr, 442 op/s; 14350/264090 objects degraded (5.434%); 23010/264090 objects misplaced (8.713%); 11 MiB/s, 179 objects/s recovering 2024-03-26T03:02:19.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14350/264090 objects degraded (5.434%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:19.490 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:02:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:02:19.562 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:19 smithi066 ceph-mon[29666]: pgmap v121: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 576 B/s rd, 28 MiB/s wr, 442 op/s; 14350/264090 objects degraded (5.434%); 23010/264090 objects misplaced (8.713%); 11 MiB/s, 179 objects/s recovering 2024-03-26T03:02:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14350/264090 objects degraded (5.434%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:19 smithi066 ceph-mon[37920]: pgmap v121: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 576 B/s rd, 28 MiB/s wr, 442 op/s; 14350/264090 objects degraded (5.434%); 23010/264090 objects misplaced (8.713%); 11 MiB/s, 179 objects/s recovering 2024-03-26T03:02:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14350/264090 objects degraded (5.434%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.614 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:19.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:19.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:19.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:19.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:19.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:19.970 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:19.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:19.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:20.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:20.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:20.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:20.271 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 77 16 91925 91909 74.5812 33.625 0.0177459 0.0134011 2024-03-26T03:02:20.278 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:20.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:20.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:20.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.560 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:20.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:20.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:20.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:20.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:20.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:20.834 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:20.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:20.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:20.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:21.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:21.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:21.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:21.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:21.271 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 78 16 92605 92589 74.1699 42.5 0.020441 0.0134755 2024-03-26T03:02:21.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:21.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:21.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:21.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:21 smithi192 ceph-mon[33110]: pgmap v122: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 43 MiB/s wr, 682 op/s; 14350/274632 objects degraded (5.225%); 20987/274632 objects misplaced (7.642%); 18 MiB/s, 283 objects/s recovering 2024-03-26T03:02:21.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:21.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:21 smithi066 ceph-mon[29666]: pgmap v122: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 43 MiB/s wr, 682 op/s; 14350/274632 objects degraded (5.225%); 20987/274632 objects misplaced (7.642%); 18 MiB/s, 283 objects/s recovering 2024-03-26T03:02:21.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:21 smithi066 ceph-mon[37920]: pgmap v122: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 43 MiB/s wr, 682 op/s; 14350/274632 objects degraded (5.225%); 20987/274632 objects misplaced (7.642%); 18 MiB/s, 283 objects/s recovering 2024-03-26T03:02:21.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:21.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:21.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:21.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:21.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:21.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:21.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:21.969 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:22.010 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:22.112 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:22.154 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:22.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:22.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 79 16 92958 92942 73.5104 22.0625 0.00483086 0.0134853 2024-03-26T03:02:22.287 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:22.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:22.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:22 smithi192 ceph-mon[33110]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T03:02:22.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:22.567 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:22.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:22 smithi066 ceph-mon[29666]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T03:02:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:22 smithi066 ceph-mon[37920]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T03:02:22.669 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:22.722 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:22.823 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:22.852 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:22.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:22.953 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:23.012 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:23.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:23.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:23.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:23.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:23.271 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:02:23.272229+0000 min lat: 0.00124684 max lat: 6.18392 avg lat: 0.0136728 2024-03-26T03:02:23.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:02:23.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 80 16 93586 93570 73.0821 39.25 0.0573797 0.0136728 2024-03-26T03:02:23.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:23.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:23.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-mon[29666]: pgmap v124: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s; 14350/274632 objects degraded (5.225%); 20987/274632 objects misplaced (7.642%); 22 MiB/s, 352 objects/s recovering 2024-03-26T03:02:23.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-mon[29666]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T03:02:23.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:23.493 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-mon[37920]: pgmap v124: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s; 14350/274632 objects degraded (5.225%); 20987/274632 objects misplaced (7.642%); 22 MiB/s, 352 objects/s recovering 2024-03-26T03:02:23.493 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-mon[37920]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T03:02:23.493 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:23.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:23.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:02:23.576 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:23.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:23.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:23 smithi192 ceph-mon[33110]: pgmap v124: 158 pgs: 6 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s; 14350/274632 objects degraded (5.225%); 20987/274632 objects misplaced (7.642%); 22 MiB/s, 352 objects/s recovering 2024-03-26T03:02:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:23 smithi192 ceph-mon[33110]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T03:02:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:23.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:23.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:23.938 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:23.996 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:23.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:24.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:24.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:02:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:02:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:02:24.150 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:24.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:24.252 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:24.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 81 16 94310 94294 72.7385 45.25 0.00777902 0.0137426 2024-03-26T03:02:24.299 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:24.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:24.401 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:24.452 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:24.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:24.554 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:24.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14350/274632 objects degraded (5.225%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:24.586 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14350/274632 objects degraded (5.225%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:24.602 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:24.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:24.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:24.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:24.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14350/274632 objects degraded (5.225%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:24.834 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:24.881 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:24.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:24.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:25.012 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:25.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.170 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:25.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:25.271 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:25.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 82 16 95071 95055 72.4315 47.5625 0.0196626 0.0137925 2024-03-26T03:02:25.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:25.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:25.459 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:25.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:25.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:25 smithi066 ceph-mon[29666]: pgmap v126: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 539 op/s; 14350/277035 objects degraded (5.180%); 20838/277035 objects misplaced (7.522%); 13 MiB/s, 207 objects/s recovering 2024-03-26T03:02:25.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:25 smithi066 ceph-mon[37920]: pgmap v126: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 539 op/s; 14350/277035 objects degraded (5.180%); 20838/277035 objects misplaced (7.522%); 13 MiB/s, 207 objects/s recovering 2024-03-26T03:02:25.687 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.722 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:25.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:25 smithi192 ceph-mon[33110]: pgmap v126: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 539 op/s; 14350/277035 objects degraded (5.180%); 20838/277035 objects misplaced (7.522%); 13 MiB/s, 207 objects/s recovering 2024-03-26T03:02:25.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:25.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:25.952 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:25.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:25.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:26.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:26.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:26.218 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.248 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:26.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:26.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 83 16 95665 95649 72.0062 37.125 0.010104 0.0138767 2024-03-26T03:02:26.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:26.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:26.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:26.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:26.604 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:26.649 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:26.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:26.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:26.876 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:26.919 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:26.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:27.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:27.044 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:27.146 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:27.170 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:02:27.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:02:27.171 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-26T03:02:27.171 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:02:27.171 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:02:27.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-26T03:02:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:27.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:27.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 84 16 96083 96067 71.46 26.125 0.0036651 0.0138837 2024-03-26T03:02:27.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:27.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:27.409 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.439 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:27.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:27.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:27.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:27 smithi066 ceph-mon[29666]: pgmap v127: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 804 op/s; 14350/283371 objects degraded (5.064%); 18984/283371 objects misplaced (6.699%); 23 MiB/s, 368 objects/s recovering 2024-03-26T03:02:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:27 smithi066 ceph-mon[29666]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T03:02:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:27 smithi066 ceph-mon[37920]: pgmap v127: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 804 op/s; 14350/283371 objects degraded (5.064%); 18984/283371 objects misplaced (6.699%); 23 MiB/s, 368 objects/s recovering 2024-03-26T03:02:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:27 smithi066 ceph-mon[37920]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T03:02:27.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:27.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:27 smithi192 ceph-mon[33110]: pgmap v127: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 804 op/s; 14350/283371 objects degraded (5.064%); 18984/283371 objects misplaced (6.699%); 23 MiB/s, 368 objects/s recovering 2024-03-26T03:02:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:27 smithi192 ceph-mon[33110]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T03:02:27.803 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:27.852 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:27.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:27.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.007 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:28.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:28.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:28.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:28.237 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:28.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 85 16 96762 96746 71.1186 42.4375 0.00609902 0.0140549 2024-03-26T03:02:28.282 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:28.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:28.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:28.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:28.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:28.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:28 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:02:28.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:28 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:02:28.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:28 smithi066 ceph-mon[29666]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T03:02:28.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:28 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:02:28.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:28 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:02:28.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:28 smithi066 ceph-mon[37920]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T03:02:28.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:28 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:02:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:28 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:02:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:28 smithi192 ceph-mon[33110]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T03:02:28.773 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:28.801 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:28.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:28.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:29.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:29.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:29.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:29.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:29.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 86 16 97351 97335 70.7197 36.8125 0.00630814 0.0141342 2024-03-26T03:02:29.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:29.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:29.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:29.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:02:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:02:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:29 smithi192 ceph-mon[33110]: pgmap v130: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 30 MiB/s wr, 486 op/s; 14350/283371 objects degraded (5.064%); 18984/283371 objects misplaced (6.699%); 15 MiB/s, 238 objects/s recovering 2024-03-26T03:02:29.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14350/283371 objects degraded (5.064%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:29.499 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:29.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:29 smithi066 ceph-mon[29666]: pgmap v130: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 30 MiB/s wr, 486 op/s; 14350/283371 objects degraded (5.064%); 18984/283371 objects misplaced (6.699%); 15 MiB/s, 238 objects/s recovering 2024-03-26T03:02:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14350/283371 objects degraded (5.064%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:29 smithi066 ceph-mon[37920]: pgmap v130: 158 pgs: 5 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 149 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 30 MiB/s wr, 486 op/s; 14350/283371 objects degraded (5.064%); 18984/283371 objects misplaced (6.699%); 15 MiB/s, 238 objects/s recovering 2024-03-26T03:02:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14350/283371 objects degraded (5.064%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:29.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:29.733 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:29.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:29.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:29.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:29.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:30.015 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:30.068 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:30.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:30.199 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:30.272 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 87 16 98091 98075 70.4384 46.25 0.0675503 0.0141872 2024-03-26T03:02:30.300 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:30.349 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:30.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:30.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:30.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:30.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:30.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:30.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:30.840 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:30.869 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:30.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:30.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:30.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:31.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:31.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:31.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:31.273 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 88 16 98769 98753 70.119 42.375 0.0374051 0.014256 2024-03-26T03:02:31.281 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:31.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:31.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:31.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:31.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:31 smithi066 ceph-mon[29666]: pgmap v131: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 48 MiB/s wr, 774 op/s; 12017/292566 objects degraded (4.107%); 19615/292566 objects misplaced (6.704%); 20 MiB/s, 319 objects/s recovering 2024-03-26T03:02:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:31 smithi066 ceph-mon[37920]: pgmap v131: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 48 MiB/s wr, 774 op/s; 12017/292566 objects degraded (4.107%); 19615/292566 objects misplaced (6.704%); 20 MiB/s, 319 objects/s recovering 2024-03-26T03:02:31.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:31.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:31 smithi192 ceph-mon[33110]: pgmap v131: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 48 MiB/s wr, 774 op/s; 12017/292566 objects degraded (4.107%); 19615/292566 objects misplaced (6.704%); 20 MiB/s, 319 objects/s recovering 2024-03-26T03:02:31.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.812 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:31.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:31.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:31.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:31.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:32.039 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.089 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:32.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:32.190 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:32.238 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:32.273 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 89 16 99450 99434 69.8094 42.5625 0.0409557 0.0143193 2024-03-26T03:02:32.340 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.368 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:32.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:32.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:32.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:32.599 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:32.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.657 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:32.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:32.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:32.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:32.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:32.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:32.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:33.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:33.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:33.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.179 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:33.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:33.273 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 90 16 100180 100164 69.5407 45.625 0.00689591 0.0143629 2024-03-26T03:02:33.280 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:33.321 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:33.423 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:33.460 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:33.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:33.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:33 smithi066 ceph-mon[29666]: pgmap v132: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 648 op/s; 12017/292566 objects degraded (4.107%); 19615/292566 objects misplaced (6.704%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:02:33.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:33 smithi066 ceph-mon[37920]: pgmap v132: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 648 op/s; 12017/292566 objects degraded (4.107%); 19615/292566 objects misplaced (6.704%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:02:33.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:33.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:33 smithi192 ceph-mon[33110]: pgmap v132: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 648 op/s; 12017/292566 objects degraded (4.107%); 19615/292566 objects misplaced (6.704%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:02:33.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:33.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:33.940 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:33.981 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:33.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:34.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:02:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:02:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:02:34.120 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:34.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:34.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:34.273 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total time run: 90.0554 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total writes made: 100180 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Write size: 65536 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object size: 65536 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Bandwidth (MB/sec): 69.5266 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Bandwidth: 42.6278 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max bandwidth (MB/sec): 191.625 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min bandwidth (MB/sec): 0 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average IOPS: 1112 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev IOPS: 682.054 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max IOPS: 3066 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min IOPS: 0 2024-03-26T03:02:34.274 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average Latency(s): 0.0143777 2024-03-26T03:02:34.275 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Latency(s): 0.062007 2024-03-26T03:02:34.275 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max latency(s): 6.18392 2024-03-26T03:02:34.275 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min latency(s): 0.00124684 2024-03-26T03:02:34.279 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:02:34.348 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:34.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:34.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:34.525 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 12017/292566 objects degraded (4.107%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 12017/292566 objects degraded (4.107%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:34.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:34.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 12017/292566 objects degraded (4.107%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:34.758 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:34.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:34.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:34.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:34.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:35.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.044 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:35.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:35.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:35.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:35.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:35.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:35.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:35.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:35.530 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:35.577 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:35 smithi066 ceph-mon[29666]: pgmap v133: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 506 op/s; 12017/295518 objects degraded (4.066%); 19615/295518 objects misplaced (6.637%); 8.1 MiB/s, 129 objects/s recovering 2024-03-26T03:02:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:35 smithi066 ceph-mon[37920]: pgmap v133: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 506 op/s; 12017/295518 objects degraded (4.066%); 19615/295518 objects misplaced (6.637%); 8.1 MiB/s, 129 objects/s recovering 2024-03-26T03:02:35.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:35.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:35 smithi192 ceph-mon[33110]: pgmap v133: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 506 op/s; 12017/295518 objects degraded (4.066%); 19615/295518 objects misplaced (6.637%); 8.1 MiB/s, 129 objects/s recovering 2024-03-26T03:02:35.809 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:35.858 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:35.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:35.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:35.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:36.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:36.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:36.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:36.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:36.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:36.364 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:02:36.420 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:36.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:36.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:36 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/650302089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:02:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:36 smithi066 ceph-mon[29666]: from='client.34757 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:02:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:36 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/650302089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:02:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:36 smithi066 ceph-mon[37920]: from='client.34757 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:02:36.651 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:36.708 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:36 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/650302089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:02:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:36 smithi192 ceph-mon[33110]: from='client.34757 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:02:36.810 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:36.859 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:36.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:36.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:36.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:37.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:37.141 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:37.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:37.243 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:37.289 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:37.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:37.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:37.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:37.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:37.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:37.615 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:37.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:37.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:37 smithi192 ceph-mon[33110]: pgmap v134: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 39 MiB/s wr, 617 op/s; 12016/301293 objects degraded (3.988%); 17474/301293 objects misplaced (5.800%); 14 MiB/s, 230 objects/s recovering 2024-03-26T03:02:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:37 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:02:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:37 smithi192 ceph-mon[33110]: from='client.34757 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:02:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:37 smithi192 ceph-mon[33110]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T03:02:37.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:37.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:37.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[29666]: pgmap v134: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 39 MiB/s wr, 617 op/s; 12016/301293 objects degraded (3.988%); 17474/301293 objects misplaced (5.800%); 14 MiB/s, 230 objects/s recovering 2024-03-26T03:02:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:02:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[29666]: from='client.34757 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:02:37.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[29666]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T03:02:37.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[37920]: pgmap v134: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 39 MiB/s wr, 617 op/s; 12016/301293 objects degraded (3.988%); 17474/301293 objects misplaced (5.800%); 14 MiB/s, 230 objects/s recovering 2024-03-26T03:02:37.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:02:37.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[37920]: from='client.34757 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:02:37.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:37 smithi066 ceph-mon[37920]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T03:02:37.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:37.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:37.977 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:38.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:38.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:38.199 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:38.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:38.324 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:38.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:38.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:38.432 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.519 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:38.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[29666]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[29666]: pgmap v137: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 23 MiB/s wr, 365 op/s; 12016/301293 objects degraded (3.988%); 17474/301293 objects misplaced (5.800%); 9.4 MiB/s, 150 objects/s recovering 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:02:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[37920]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T03:02:38.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[37920]: pgmap v137: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 23 MiB/s wr, 365 op/s; 12016/301293 objects degraded (3.988%); 17474/301293 objects misplaced (5.800%); 9.4 MiB/s, 150 objects/s recovering 2024-03-26T03:02:38.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:38.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:38.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:38.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:38 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:02:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:38 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:02:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:38 smithi192 ceph-mon[33110]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T03:02:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:38 smithi192 ceph-mon[33110]: pgmap v137: 158 pgs: 5 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 23 MiB/s wr, 365 op/s; 12016/301293 objects degraded (3.988%); 17474/301293 objects misplaced (5.800%); 9.4 MiB/s, 150 objects/s recovering 2024-03-26T03:02:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:38.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:38.792 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:38.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:38.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:38.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:39.056 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:39.118 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:39.166 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:02:39.220 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.281 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:39.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:39.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:39.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:02:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:02:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:39.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:39.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:39.640 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:39.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:39.714 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[29666]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T03:02:39.714 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 12016/301293 objects degraded (3.988%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:39.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[29666]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T03:02:39.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:39.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[37920]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T03:02:39.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 12016/301293 objects degraded (3.988%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:39.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[37920]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T03:02:39.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:39 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:39 smithi192 ceph-mon[33110]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T03:02:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 12016/301293 objects degraded (3.988%), 3 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:39 smithi192 ceph-mon[33110]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T03:02:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:39 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:39.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:39.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:39.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:39.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:39.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:40.054 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:40.121 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:40.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:40.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:40.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:40.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:40.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:40.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:40.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:40.477 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:40.567 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:40.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:40.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:40.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:40.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:40.726 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:02:40.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:02:40.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[29666]: pgmap v140: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 776 op/s; 7075/291981 objects degraded (2.423%); 19620/291981 objects misplaced (6.720%); 21 MiB/s, 342 objects/s recovering 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[37920]: pgmap v140: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 776 op/s; 7075/291981 objects degraded (2.423%); 19620/291981 objects misplaced (6.720%); 21 MiB/s, 342 objects/s recovering 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:40.727 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:40 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:02:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:40 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:02:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:40 smithi192 ceph-mon[33110]: pgmap v140: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 776 op/s; 7075/291981 objects degraded (2.423%); 19620/291981 objects misplaced (6.720%); 21 MiB/s, 342 objects/s recovering 2024-03-26T03:02:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:40.778 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:02:40.797 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:40.828 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:02:40.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:40.869 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:40.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:40.872 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:02:40.913 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:02:40.952 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:02:40.970 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:40.996 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:02:41.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:41.037 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:02:41.044 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:02:41.045 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:02:41.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:41.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:41.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:41.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:41.393 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.455 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:41.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:41.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:41.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:41.687 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:41 smithi192 ceph-mon[33110]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T03:02:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2005057669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:02:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:02:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:02:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.743 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:41.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:41.778 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[29666]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2005057669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[37920]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2005057669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:02:41.779 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:02:41.844 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:41.894 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:41.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:41.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:42.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:42.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.186 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:42.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:42.287 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.346 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:42.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:42.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.497 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:42.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:42.599 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:42.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:42 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:02:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:42 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:02:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:42 smithi192 ceph-mon[33110]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T03:02:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:42 smithi192 ceph-mon[33110]: pgmap v143: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 777 op/s; 7075/291981 objects degraded (2.423%); 19620/291981 objects misplaced (6.720%); 21 MiB/s, 342 objects/s recovering 2024-03-26T03:02:42.768 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[29666]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[29666]: pgmap v143: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 777 op/s; 7075/291981 objects degraded (2.423%); 19620/291981 objects misplaced (6.720%); 21 MiB/s, 342 objects/s recovering 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:02:42.774 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:02:42.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:02:42.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[37920]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T03:02:42.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:42 smithi066 ceph-mon[37920]: pgmap v143: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 777 op/s; 7075/291981 objects degraded (2.423%); 19620/291981 objects misplaced (6.720%); 21 MiB/s, 342 objects/s recovering 2024-03-26T03:02:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.823 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:42.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:42.924 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:42.972 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:42.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:43.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:43.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:43.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.253 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:43.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:43.354 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:43.375 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:43.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:43.462 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:02:43.462 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:02:43.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.462 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "MOSDPGPush(7.14 128/114 [PushOp(7:2b6b8819:::benchmark_data_smithi066_76917_object38046:head, version: 100'2340, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2b6b8819:::benchmark_data_smithi066_76917_object38046:head@100'2340, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.448626+0000", 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.010136236999999999, 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010190961, 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.448626+0000", 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.463 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.448626+0000", 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.448627+0000", 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7699999999999992e-07 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.448663+0000", 2024-03-26T03:02:43.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6494e-05 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.448705+0000", 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1865e-05 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.448725+0000", 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9762000000000002e-05 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:106464 7.14 e128/114 7:2a7f4aa9:::benchmark_data_smithi066_76917_object6281:head v 128'5638, mlcod=100'2334)", 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.452394+0000", 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0063682230000000001, 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0064797470000000001, 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452394+0000", 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452394+0000", 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452394+0000", 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8599999999999998e-07 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452401+0000", 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7859999999999999e-06 2024-03-26T03:02:43.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452411+0000", 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7429999999999997e-06 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452416+0000", 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0819999999999998e-06 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:02:43.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453660+0000", 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001243929 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453684+0000", 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4672999999999999e-05 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:106478 7.a e128/96 7:5185704b:::benchmark_data_smithi066_76917_object6295:head v 128'6645, mlcod=128'6643)", 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.452984+0000", 2024-03-26T03:02:43.469 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0057776670000000002, 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0059571709999999998, 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452984+0000", 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452984+0000", 2024-03-26T03:02:43.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452986+0000", 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.358e-06 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.452993+0000", 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7999999999999999e-06 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453004+0000", 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0549e-05 2024-03-26T03:02:43.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453012+0000", 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8380000000000002e-06 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453250+0000", 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000238705 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453275+0000", 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4700000000000001e-05 2024-03-26T03:02:43.472 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:106477 7.e 7:7f1cf1ff:::benchmark_data_smithi066_76917_object6294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.453044+0000", 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0057178460000000004, 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0059609169999999996, 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:02:43.473 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 106477 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453044+0000", 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453044+0000", 2024-03-26T03:02:43.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453045+0000", 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4950000000000001e-06 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453050+0000", 2024-03-26T03:02:43.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8339999999999998e-06 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453060+0000", 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5249999999999994e-06 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453068+0000", 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5250000000000005e-06 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:02:43.476 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453376+0000", 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00030724700000000002 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453508+0000", 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013232000000000001 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.453636+0000", 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012819 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.477 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:106470 7.19 e128/114 7:98c019fd:::benchmark_data_smithi066_76917_object6287:head v 128'5554, mlcod=99'406)", 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.455332+0000", 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0034293259999999999, 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0037499550000000001, 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455332+0000", 2024-03-26T03:02:43.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455332+0000", 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455334+0000", 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.269e-06 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455349+0000", 2024-03-26T03:02:43.479 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5664000000000001e-05 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455360+0000", 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0549e-05 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455368+0000", 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0099999999999995e-06 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456640+0000", 2024-03-26T03:02:43.480 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001271891 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456662+0000", 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2212999999999999e-05 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "MOSDPGPush(7.19 128/114 [PushOp(7:9831d407:::benchmark_data_smithi066_76917_object6490:head, version: 99'408, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:9831d407:::benchmark_data_smithi066_76917_object6490:head@99'408, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.455388+0000", 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0033734759999999998, 2024-03-26T03:02:43.481 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0037736969999999999, 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455388+0000", 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455388+0000", 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.482 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455389+0000", 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1799999999999995e-07 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455464+0000", 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4721000000000005e-05 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455481+0000", 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6875e-05 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.483 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.455495+0000", 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4419e-05 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.14841.0:106477 7.e e128/96)", 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.456617+0000", 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0021447060000000001, 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0025894590000000001, 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.484 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456617+0000", 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456617+0000", 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.485 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456619+0000", 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.443e-06 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456624+0000", 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6570000000000002e-06 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456635+0000", 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0423e-05 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.486 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456642+0000", 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6079999999999998e-06 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:106479 7.e 7.eb51a67e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.456876+0000", 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.001885598, 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00237799, 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.487 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 106479 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456876+0000", 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.488 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456876+0000", 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456878+0000", 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.142e-06 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456885+0000", 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.703e-06 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456903+0000", 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8414e-05 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456912+0000", 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3250000000000008e-06 2024-03-26T03:02:43.490 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:106483 7.e 7.71cd932e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.456923+0000", 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.001838405, 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002390133, 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:02:43.491 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 106483 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456923+0000", 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456923+0000", 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456924+0000", 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8800000000000002e-07 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456928+0000", 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3900000000000002e-06 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456934+0000", 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9689999999999999e-06 2024-03-26T03:02:43.493 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456940+0000", 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2400000000000004e-06 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "MOSDPGPushReply(7.18 128/114 [PushReplyOp(7:1feb0609:::benchmark_data_smithi066_76917_object15940:head)])", 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.456978+0000", 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0017837269999999999, 2024-03-26T03:02:43.494 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0023963280000000001, 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456978+0000", 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456978+0000", 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.495 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456979+0000", 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.17e-06 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456984+0000", 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0329999999999999e-06 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.456994+0000", 2024-03-26T03:02:43.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5699999999999999e-06 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.457001+0000", 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2819999999999999e-06 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.457074+0000", 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3325999999999996e-05 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:106465 7.e 7:71bf4c0a:::benchmark_data_smithi066_76917_object6282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:43.437001+0000", 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.021760998, 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.022442997999999999, 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 106465 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.498 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437001+0000", 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437001+0000", 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437003+0000", 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3779999999999999e-06 2024-03-26T03:02:43.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437010+0000", 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3389999999999998e-06 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437022+0000", 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2244e-05 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437031+0000", 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8489999999999995e-06 2024-03-26T03:02:43.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437092+0000", 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1146e-05 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437234+0000", 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014197999999999999 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.437356+0000", 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012246199999999999 2024-03-26T03:02:43.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.442644+0000", 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0052875300000000004 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:43.445628+0000", 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002984511 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:02:43.502 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:02:43.503 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 11 2024-03-26T03:02:43.503 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:02:43.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:43.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:43.642 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:43.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:43.790 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:43.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:43.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:43.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:43.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:44.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:44.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:02:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:02:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:02:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:44.152 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:44.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:44.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:02:44.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:44.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:44.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:44.381 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:44.453 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:44.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:44.554 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:44.613 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:44.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:44.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:44.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:44.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:44.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:44.898 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:44.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:44.976 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:02:45.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:45.079 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:45.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:45.181 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 7075/291981 objects degraded (2.423%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[29666]: pgmap v144: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 183 B/s wr, 794 op/s; 7075/288039 objects degraded (2.456%); 19620/288039 objects misplaced (6.812%); 15 MiB/s, 245 objects/s recovering 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 7075/291981 objects degraded (2.423%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[37920]: pgmap v144: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 183 B/s wr, 794 op/s; 7075/288039 objects degraded (2.456%); 19620/288039 objects misplaced (6.812%); 15 MiB/s, 245 objects/s recovering 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:02:45.188 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:44 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:02:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 7075/291981 objects degraded (2.423%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:44 smithi192 ceph-mon[33110]: pgmap v144: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 183 B/s wr, 794 op/s; 7075/288039 objects degraded (2.456%); 19620/288039 objects misplaced (6.812%); 15 MiB/s, 245 objects/s recovering 2024-03-26T03:02:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:44 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:02:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:44 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:02:45.306 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:02:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:02:45.306 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:02:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:45.310 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:45.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:45.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:45.477 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:45.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:45.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:45.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:45.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:45.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:45.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:45.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:45.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:45.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:45.972 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.015 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:46.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:46.019 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:45 smithi066 ceph-mon[29666]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T03:02:46.020 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:45 smithi066 ceph-mon[37920]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T03:02:46.117 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:46.172 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:46.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:45 smithi192 ceph-mon[33110]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T03:02:46.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:46.324 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:46.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:46.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:46.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:46.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:46.682 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:46.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:46.764 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:02:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.793 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:46.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:46.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:46.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:46.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:47.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:47.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:47.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:46 smithi066 ceph-mon[29666]: pgmap v146: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 616 op/s; 6839/280890 objects degraded (2.435%); 17355/280890 objects misplaced (6.179%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:02:47.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:46 smithi066 ceph-mon[37920]: pgmap v146: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 616 op/s; 6839/280890 objects degraded (2.435%); 17355/280890 objects misplaced (6.179%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:02:47.151 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:47.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:47.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:46 smithi192 ceph-mon[33110]: pgmap v146: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 616 op/s; 6839/280890 objects degraded (2.435%); 17355/280890 objects misplaced (6.179%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:02:47.281 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:47.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:47.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:47.465 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:47.567 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:47.613 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:47.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:47.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:47.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:47.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:47.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:47.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.022 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:48.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:48.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:48.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:48.248 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:48.296 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:48.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:48.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:48.430 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:48.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:48.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:48.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:48.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:48.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:48.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:48.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:48.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:48.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:49.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.095 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:49.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:49.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:49.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:49.323 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:49 smithi066 ceph-mon[29666]: pgmap v147: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 489 op/s; 6839/280890 objects degraded (2.435%); 17355/280890 objects misplaced (6.179%); 8.8 MiB/s, 140 objects/s recovering 2024-03-26T03:02:49.323 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 6839/280890 objects degraded (2.435%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:49.323 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:49 smithi066 ceph-mon[29666]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T03:02:49.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:49 smithi066 ceph-mon[37920]: pgmap v147: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 489 op/s; 6839/280890 objects degraded (2.435%); 17355/280890 objects misplaced (6.179%); 8.8 MiB/s, 140 objects/s recovering 2024-03-26T03:02:49.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 6839/280890 objects degraded (2.435%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:49.324 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:49 smithi066 ceph-mon[37920]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T03:02:49.325 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:49.375 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:49.477 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:49.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:02:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:02:49.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:49 smithi192 ceph-mon[33110]: pgmap v147: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 489 op/s; 6839/280890 objects degraded (2.435%); 17355/280890 objects misplaced (6.179%); 8.8 MiB/s, 140 objects/s recovering 2024-03-26T03:02:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 6839/280890 objects degraded (2.435%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:49 smithi192 ceph-mon[33110]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T03:02:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.524 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:49.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:49.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:49.672 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:49.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:49.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:49.904 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:49.954 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:49.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:50.056 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:50.109 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:50.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:50.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:50.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:50.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:50 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:50.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:50 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:50.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:50.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:50 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:02:50.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:50.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:50.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:50.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:50.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:50.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:50.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:50.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:50.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:51.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:51.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:02:51.076 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:02:51.076 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T03:02:51.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:02:51.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:51.201 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:51.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:51.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:51.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:51 smithi066 ceph-mon[29666]: pgmap v149: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 6639/270153 objects degraded (2.457%); 15001/270153 objects misplaced (5.553%); 17 MiB/s, 270 objects/s recovering 2024-03-26T03:02:51.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:51 smithi066 ceph-mon[37920]: pgmap v149: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 6639/270153 objects degraded (2.457%); 15001/270153 objects misplaced (5.553%); 17 MiB/s, 270 objects/s recovering 2024-03-26T03:02:51.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:51 smithi192 ceph-mon[33110]: pgmap v149: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 6639/270153 objects degraded (2.457%); 15001/270153 objects misplaced (5.553%); 17 MiB/s, 270 objects/s recovering 2024-03-26T03:02:51.500 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:51.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:51.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:51.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:51.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:51.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:51.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:51.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:51.989 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:52.037 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:52.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:52.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:52.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:52.291 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:52.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:52.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:52 smithi192 ceph-mon[33110]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T03:02:52.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:52.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:52.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:52 smithi066 ceph-mon[29666]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T03:02:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:52 smithi066 ceph-mon[37920]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T03:02:52.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:52.683 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:52.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:52.831 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:52.933 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:52.961 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:52.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:53.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:53.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:53.190 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.219 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:53.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:53.320 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:53 smithi066 ceph-mon[29666]: pgmap v151: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 725 B/s rd, 0 B/s wr, 507 op/s; 6639/270153 objects degraded (2.457%); 15001/270153 objects misplaced (5.553%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:02:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:53 smithi066 ceph-mon[29666]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T03:02:53.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:53 smithi066 ceph-mon[37920]: pgmap v151: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 725 B/s rd, 0 B/s wr, 507 op/s; 6639/270153 objects degraded (2.457%); 15001/270153 objects misplaced (5.553%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:02:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:53 smithi066 ceph-mon[37920]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T03:02:53.376 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:53.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:53 smithi192 ceph-mon[33110]: pgmap v151: 158 pgs: 5 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 151 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 725 B/s rd, 0 B/s wr, 507 op/s; 6639/270153 objects degraded (2.457%); 15001/270153 objects misplaced (5.553%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:02:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:53 smithi192 ceph-mon[33110]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T03:02:53.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:53.610 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:53.661 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:53.763 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:53.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:53.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:53.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:53.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:53.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:54.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:54.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:02:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:02:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:02:54.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:54.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:54.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:54.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:54.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:54.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:54 smithi192 ceph-mon[33110]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T03:02:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 6639/270153 objects degraded (2.457%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:54 smithi192 ceph-mon[33110]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T03:02:54.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:54.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[29666]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T03:02:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 6639/270153 objects degraded (2.457%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[29666]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T03:02:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[37920]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T03:02:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:02:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 6639/270153 objects degraded (2.457%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:02:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:54 smithi066 ceph-mon[37920]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T03:02:54.694 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:54.743 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:54.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:54.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:54.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:54.975 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.003 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:55.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:55.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:55.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:55.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:55.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:55.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:55.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:55 smithi192 ceph-mon[33110]: pgmap v155: 158 pgs: 4 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 305 op/s; 6639/266481 objects degraded (2.491%); 14821/266481 objects misplaced (5.562%); 1.4 MiB/s, 22 objects/s recovering 2024-03-26T03:02:55.491 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:55.539 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:55.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:55 smithi066 ceph-mon[29666]: pgmap v155: 158 pgs: 4 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 305 op/s; 6639/266481 objects degraded (2.491%); 14821/266481 objects misplaced (5.562%); 1.4 MiB/s, 22 objects/s recovering 2024-03-26T03:02:55.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:55 smithi066 ceph-mon[37920]: pgmap v155: 158 pgs: 4 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 152 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 305 op/s; 6639/266481 objects degraded (2.491%); 14821/266481 objects misplaced (5.562%); 1.4 MiB/s, 22 objects/s recovering 2024-03-26T03:02:55.641 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:55.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:55.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:55.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:55.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:55.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:55.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:56.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:56.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:56.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:56.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:56.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:56.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:56.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:56.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:56.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:56.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:56.720 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:56.764 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:56.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:56.907 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:56.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:57.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:57.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:57.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:02:57.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:57.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:02:57.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:57.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:57.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:57.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:57 smithi192 ceph-mon[33110]: pgmap v156: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 683 op/s; 4357/260220 objects degraded (1.674%); 15196/260220 objects misplaced (5.840%); 12 MiB/s, 189 objects/s recovering 2024-03-26T03:02:57.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:57.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:57.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:57 smithi066 ceph-mon[29666]: pgmap v156: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 683 op/s; 4357/260220 objects degraded (1.674%); 15196/260220 objects misplaced (5.840%); 12 MiB/s, 189 objects/s recovering 2024-03-26T03:02:57.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:57 smithi066 ceph-mon[37920]: pgmap v156: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 683 op/s; 4357/260220 objects degraded (1.674%); 15196/260220 objects misplaced (5.840%); 12 MiB/s, 189 objects/s recovering 2024-03-26T03:02:57.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:57.706 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:57.808 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:57.850 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:57.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:57.952 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:57.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:58.100 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:58.152 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:58.254 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:58.323 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:58.425 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:58.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:58.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:58.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:58.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:02:58.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:02:58.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:58.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:58.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:58.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:02:59.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:02:59.102 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:02:59.153 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:02:59.255 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:02:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.306 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:59.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:02:59.347 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:59 smithi066 ceph-mon[29666]: pgmap v157: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 551 op/s; 4357/260220 objects degraded (1.674%); 15196/260220 objects misplaced (5.840%); 9.5 MiB/s, 152 objects/s recovering 2024-03-26T03:02:59.348 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:02:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 4357/260220 objects degraded (1.674%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:02:59.348 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:59 smithi066 ceph-mon[37920]: pgmap v157: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 551 op/s; 4357/260220 objects degraded (1.674%); 15196/260220 objects misplaced (5.840%); 9.5 MiB/s, 152 objects/s recovering 2024-03-26T03:02:59.348 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:02:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 4357/260220 objects degraded (1.674%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:02:59.408 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.479 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:59.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:02:59.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:59 smithi192 ceph-mon[33110]: pgmap v157: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 551 op/s; 4357/260220 objects degraded (1.674%); 15196/260220 objects misplaced (5.840%); 9.5 MiB/s, 152 objects/s recovering 2024-03-26T03:02:59.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:02:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 4357/260220 objects degraded (1.674%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:02:59.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:02:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:02:59.580 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:02:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:59.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:02:59.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:02:59.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:02:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:02:59.867 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:02:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:02:59.921 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:02:59.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:00.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:00.072 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:00.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:00.196 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:03:00.202 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:00.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.333 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:00.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2227719715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:00 smithi066 ceph-mon[29666]: from='client.34766 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2227719715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:00 smithi066 ceph-mon[37920]: from='client.34766 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:00.434 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:00.483 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2227719715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:00 smithi192 ceph-mon[33110]: from='client.34766 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:00.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:00.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:00.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:00.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:00.763 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:00.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:00.916 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:00.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:01.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:01.089 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:01.191 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:01.243 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:01.343 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[29666]: pgmap v158: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 946 op/s; 4082/247875 objects degraded (1.647%); 13664/247875 objects misplaced (5.512%); 13 MiB/s, 215 objects/s recovering 2024-03-26T03:03:01.343 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:03:01.343 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[29666]: from='client.34766 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:03:01.344 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[29666]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T03:03:01.344 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[37920]: pgmap v158: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 946 op/s; 4082/247875 objects degraded (1.647%); 13664/247875 objects misplaced (5.512%); 13 MiB/s, 215 objects/s recovering 2024-03-26T03:03:01.344 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:03:01.344 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[37920]: from='client.34766 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:03:01.344 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:01 smithi066 ceph-mon[37920]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T03:03:01.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:01.394 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:01 smithi192 ceph-mon[33110]: pgmap v158: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 946 op/s; 4082/247875 objects degraded (1.647%); 13664/247875 objects misplaced (5.512%); 13 MiB/s, 215 objects/s recovering 2024-03-26T03:03:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:01 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:03:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:01 smithi192 ceph-mon[33110]: from='client.34766 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:03:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:01 smithi192 ceph-mon[33110]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T03:03:01.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:01.554 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:01.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:01.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:01.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:01.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:01.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:01.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:01.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:02.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:02.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:02.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:02.196 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:02.248 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:02.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:02.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:02.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:02 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3938611687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:03:02.489 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:03:02.490 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:03:02.491 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:02 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3938611687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:02.491 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:02 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3938611687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:02.524 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:02.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:02.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:02.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:02.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:02.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:02.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:02.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:03.010 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:03.067 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:03.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:03.169 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:03.201 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:03.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:03.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:03.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:03.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:03 smithi192 ceph-mon[33110]: pgmap v160: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 4082/247875 objects degraded (1.647%); 13664/247875 objects misplaced (5.512%); 13 MiB/s, 209 objects/s recovering 2024-03-26T03:03:03.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:03.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:03 smithi066 ceph-mon[29666]: pgmap v160: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 4082/247875 objects degraded (1.647%); 13664/247875 objects misplaced (5.512%); 13 MiB/s, 209 objects/s recovering 2024-03-26T03:03:03.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:03 smithi066 ceph-mon[37920]: pgmap v160: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 4082/247875 objects degraded (1.647%); 13664/247875 objects misplaced (5.512%); 13 MiB/s, 209 objects/s recovering 2024-03-26T03:03:03.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:03.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:03.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:03.871 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:03.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:03.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:04.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:04.102 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:03:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:03:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:03:04.153 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:04.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:04.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:04.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:04.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 4082/247875 objects degraded (1.647%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:04 smithi192 ceph-mon[33110]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T03:03:04.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:04.567 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:04.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 4082/247875 objects degraded (1.647%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:04.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:04 smithi066 ceph-mon[29666]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T03:03:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 4082/247875 objects degraded (1.647%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:04 smithi066 ceph-mon[37920]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T03:03:04.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:04.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:04.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:04.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:04.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:04.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:04.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:05.052 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.102 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:05.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:05.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:05.255 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:05.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:05.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:05 smithi192 ceph-mon[33110]: pgmap v162: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 695 op/s; 4082/243531 objects degraded (1.676%); 13664/243531 objects misplaced (5.611%); 6.1 MiB/s, 96 objects/s recovering 2024-03-26T03:03:05.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:05.516 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:05.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:05 smithi066 ceph-mon[29666]: pgmap v162: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 695 op/s; 4082/243531 objects degraded (1.676%); 13664/243531 objects misplaced (5.611%); 6.1 MiB/s, 96 objects/s recovering 2024-03-26T03:03:05.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:05 smithi066 ceph-mon[37920]: pgmap v162: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 695 op/s; 4082/243531 objects degraded (1.676%); 13664/243531 objects misplaced (5.611%); 6.1 MiB/s, 96 objects/s recovering 2024-03-26T03:03:05.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:05.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:05.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:05.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:05.879 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:05.923 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:05.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:06.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:06.071 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:06.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:06.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:06.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:06.352 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:06.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:06.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:06.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:06.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:06.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:06.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:06.860 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:06.905 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:06.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:07.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:07.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:07.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:07.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:07.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:07.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:07.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:07.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:07 smithi192 ceph-mon[33110]: pgmap v163: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 3826/236127 objects degraded (1.620%); 12177/236127 objects misplaced (5.157%); 12 MiB/s, 188 objects/s recovering 2024-03-26T03:03:07.521 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:07.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:07.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:07 smithi066 ceph-mon[29666]: pgmap v163: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 3826/236127 objects degraded (1.620%); 12177/236127 objects misplaced (5.157%); 12 MiB/s, 188 objects/s recovering 2024-03-26T03:03:07.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:07 smithi066 ceph-mon[37920]: pgmap v163: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 3826/236127 objects degraded (1.620%); 12177/236127 objects misplaced (5.157%); 12 MiB/s, 188 objects/s recovering 2024-03-26T03:03:07.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:07.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:07.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:07.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:07.927 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:07.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:07.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:08.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:08.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:08.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:08.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:08.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:08.482 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:08.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:08.616 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:08.665 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:08.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:08.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:08.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:08.949 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:08.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:09.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:09.068 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:03:09.068 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:03:09.068 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:03:09.068 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:03:09.068 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:03:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:09.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:09.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:09.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:09.332 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:09 smithi066 ceph-mon[29666]: pgmap v164: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 489 op/s; 3826/236127 objects degraded (1.620%); 12177/236127 objects misplaced (5.157%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T03:03:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 3826/236127 objects degraded (1.620%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:09 smithi066 ceph-mon[37920]: pgmap v164: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 489 op/s; 3826/236127 objects degraded (1.620%); 12177/236127 objects misplaced (5.157%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T03:03:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 3826/236127 objects degraded (1.620%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.374 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:09.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:09.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:09.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:03:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:03:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:09 smithi192 ceph-mon[33110]: pgmap v164: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 489 op/s; 3826/236127 objects degraded (1.620%); 12177/236127 objects misplaced (5.157%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T03:03:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 3826/236127 objects degraded (1.620%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:09.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:09.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:09.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:09.737 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:09.782 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:09.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:09.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:09.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:10.015 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:10.064 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:10.166 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:10.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.194 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:10.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:10.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:10.444 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:10 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:03:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:10 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:03:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:10 smithi192 ceph-mon[33110]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T03:03:10.493 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:10.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:03:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:03:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[29666]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T03:03:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:03:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:03:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:10 smithi066 ceph-mon[37920]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T03:03:10.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:10.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:10.796 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:10.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:10.945 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:10.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:11.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:11.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:11.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:11.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:11.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:11.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:11.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:11 smithi066 ceph-mon[29666]: pgmap v166: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s; 3550/223971 objects degraded (1.585%); 10489/223971 objects misplaced (4.683%); 12 MiB/s, 195 objects/s recovering 2024-03-26T03:03:11.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:11 smithi066 ceph-mon[37920]: pgmap v166: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s; 3550/223971 objects degraded (1.585%); 10489/223971 objects misplaced (4.683%); 12 MiB/s, 195 objects/s recovering 2024-03-26T03:03:11.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:11.488 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:11.590 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:11.632 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:11.734 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:11 smithi192 ceph-mon[33110]: pgmap v166: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 996 op/s; 3550/223971 objects degraded (1.585%); 10489/223971 objects misplaced (4.683%); 12 MiB/s, 195 objects/s recovering 2024-03-26T03:03:11.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:11.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:11.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:11.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:12.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:12.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:12.132 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.183 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:12.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:12.284 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:12.329 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:03:12.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:12.440 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:12.488 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:12.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:12.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:12.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:12.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:12.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:12.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:12.895 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:12.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:12.997 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:13.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:13.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:13.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:13.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.335 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:13.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:13.405 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:03:13.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:13.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:13.567 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:13.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:13 smithi066 ceph-mon[29666]: pgmap v167: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 987 op/s; 3550/223971 objects degraded (1.585%); 10489/223971 objects misplaced (4.683%); 12 MiB/s, 193 objects/s recovering 2024-03-26T03:03:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:13 smithi066 ceph-mon[37920]: pgmap v167: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 987 op/s; 3550/223971 objects degraded (1.585%); 10489/223971 objects misplaced (4.683%); 12 MiB/s, 193 objects/s recovering 2024-03-26T03:03:13.628 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:13.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:13 smithi192 ceph-mon[33110]: pgmap v167: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 987 op/s; 3550/223971 objects degraded (1.585%); 10489/223971 objects misplaced (4.683%); 12 MiB/s, 193 objects/s recovering 2024-03-26T03:03:13.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:13.827 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:13.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:13.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:13.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:13.983 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:03:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:03:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:03:13.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:14.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:14.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:14.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:14.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:14.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:14.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:14.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:14.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:14.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:14.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:03:14.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 3550/223971 objects degraded (1.585%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:14.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 3550/223971 objects degraded (1.585%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:14.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.693 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:14.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 3550/223971 objects degraded (1.585%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:14.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:14.863 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:14.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:14.964 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:15.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:15.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:15.152 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:15.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:15.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:15.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:15 smithi066 ceph-mon[29666]: pgmap v168: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 3550/219195 objects degraded (1.620%); 10489/219195 objects misplaced (4.785%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:03:15.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:15 smithi066 ceph-mon[37920]: pgmap v168: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 3550/219195 objects degraded (1.620%); 10489/219195 objects misplaced (4.785%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:03:15.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:15.416 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:15.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:15.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:15.649 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:15.703 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:15 smithi192 ceph-mon[33110]: pgmap v168: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 3550/219195 objects degraded (1.620%); 10489/219195 objects misplaced (4.785%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:03:15.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.839 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:15.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:15.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:15.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:15.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:16.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:16.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:16.195 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:16.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:16.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:16.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:16.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:16.524 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:16.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:16.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:16.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:16.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:16.889 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:16.926 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:16.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:17.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:17.053 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:17.155 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:17.188 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:17.290 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:17.332 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:17.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:17.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:17.565 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:17 smithi066 ceph-mon[29666]: pgmap v169: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s; 3313/211290 objects degraded (1.568%); 8595/211290 objects misplaced (4.068%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:03:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:17 smithi066 ceph-mon[29666]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T03:03:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:17 smithi066 ceph-mon[37920]: pgmap v169: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s; 3313/211290 objects degraded (1.568%); 8595/211290 objects misplaced (4.068%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:03:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:17 smithi066 ceph-mon[37920]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T03:03:17.611 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:17.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:17 smithi192 ceph-mon[33110]: pgmap v169: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s; 3313/211290 objects degraded (1.568%); 8595/211290 objects misplaced (4.068%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:03:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:17 smithi192 ceph-mon[33110]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T03:03:17.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:17.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:17.923 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:17.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:18.024 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:18.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:18.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:18.204 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:18.306 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.353 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:18.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:18.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:18.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:18.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:18 smithi066 ceph-mon[29666]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T03:03:18.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:18 smithi066 ceph-mon[37920]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T03:03:18.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:18.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:18.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:18 smithi192 ceph-mon[33110]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T03:03:18.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:18.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:18.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:18.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:18.976 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:19.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:19.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:19.180 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:19.282 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:19.343 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:19.344 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:19 smithi066 ceph-mon[29666]: pgmap v172: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 528 op/s; 3313/211290 objects degraded (1.568%); 8595/211290 objects misplaced (4.068%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T03:03:19.344 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 3313/211290 objects degraded (1.568%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:19.344 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:19 smithi066 ceph-mon[37920]: pgmap v172: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 528 op/s; 3313/211290 objects degraded (1.568%); 8595/211290 objects misplaced (4.068%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T03:03:19.344 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 3313/211290 objects degraded (1.568%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:19.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:19.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:19 smithi192 ceph-mon[33110]: pgmap v172: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 153 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 528 op/s; 3313/211290 objects degraded (1.568%); 8595/211290 objects misplaced (4.068%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T03:03:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 3313/211290 objects degraded (1.568%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:19.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:03:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:03:19.516 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:19.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:19.617 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.652 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:19.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:19.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:19.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:19.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:19.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:19.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:20.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:20.028 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:03:20.028 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:03:20.028 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 139, 2024-03-26T03:03:20.028 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:03:17.277593+0000", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T02:58:43.841339+0000", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5806080, 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "nodeep-scrub", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "noscrub", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:03:20.029 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 7, 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:03:20.030 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:03:20.031 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 1, 2024-03-26T03:03:20.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 1, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 1, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 1, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 1, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.033 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "22", 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.034 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.035 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.036 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.037 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 7.8899998664855957, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 7.8899998664855957, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 3, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 3, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.038 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.039 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:03:20.040 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:03:20.041 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.043 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:03:20.044 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.5, 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.5, 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.5, 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.5, 2024-03-26T03:03:20.045 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.046 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:03:20.047 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.048 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.049 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.050 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.051 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6500000953674316, 2024-03-26T03:03:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6500000953674316, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:03:20.053 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.054 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.055 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.056 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.057 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.058 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.059 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.25, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.25, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.25, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.25, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.060 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.061 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:03:20.062 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:03:20.063 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.064 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.065 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:03:20.066 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.25, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.25, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.25, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.25, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.067 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.068 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:03:20.068 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:03:20.068 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:03:20.069 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:03:20.069 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:03:20.069 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.069 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.069 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:03:20.070 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:03:20.071 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.072 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.074 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.75, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.75, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:03:20.075 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 7, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_0", 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:00:53.479167+0000", 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:03:20.076 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 26, 2024-03-26T03:03:20.077 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 26, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 26, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 26, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 26, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:03:20.078 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "113", 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "111", 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:03:20.079 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:03:20.080 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:03:20.081 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:03:20.082 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.1500000953674316, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.8500000238418579, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.1500000953674316, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.8500000238418579, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 1, 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:20.083 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 137, 2024-03-26T03:03:20.084 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.085 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.086 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:03:20.087 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 130, 2024-03-26T03:03:20.089 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.090 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.092 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.093 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 131, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.094 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:03:20.096 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:03:20.097 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 138, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.099 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.100 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.102 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.103 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.104 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 125, 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.105 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.106 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.107 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.108 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 111, 2024-03-26T03:03:20.109 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.110 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.111 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:03:20.112 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:03:20.112 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.112 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.112 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.112 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.112 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.113 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:20.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:20.113 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:03:20.114 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.115 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 131, 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.116 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.117 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.118 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.119 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 121, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:03:20.120 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.121 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:03:20.122 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.123 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.124 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:02:08.084205+0000", 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.125 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:02:37.301346+0000", 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.126 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:02:39.489448+0000", 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:03:08.469232+0000", 2024-03-26T03:03:20.127 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:02:27.234014+0000", 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.128 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:02:44.394806+0000", 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:03:20.129 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:01:56.633805+0000", 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:03:20.130 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:02:41.184682+0000", 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [], 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.131 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.132 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:03:20.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [ 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "7.10", 2024-03-26T03:03:20.134 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: 3, 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: 1 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "7.17", 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: 2 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.135 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "7.18", 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: 2, 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "7.19", 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: 2 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.136 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:03:20.137 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:03:20.138 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:03:20.139 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:03:20.140 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:03:20.140 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:20.140 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:03:20.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:20.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:20.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:20.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:20.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:20.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:20.555 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:20.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:20 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1132986287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:03:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:20 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1132986287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:03:20.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:20.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:20 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1132986287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:03:20.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:20.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:20.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:20.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:20.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:21.044 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:21.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:21.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:21.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:21.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:21.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:21.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:21.494 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:21.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:21.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:21 smithi066 ceph-mon[29666]: pgmap v173: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 332/199893 objects degraded (0.166%); 9672/199893 objects misplaced (4.839%); 14 MiB/s, 221 objects/s recovering 2024-03-26T03:03:21.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:21 smithi066 ceph-mon[37920]: pgmap v173: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 332/199893 objects degraded (0.166%); 9672/199893 objects misplaced (4.839%); 14 MiB/s, 221 objects/s recovering 2024-03-26T03:03:21.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:21.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:21 smithi192 ceph-mon[33110]: pgmap v173: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 332/199893 objects degraded (0.166%); 9672/199893 objects misplaced (4.839%); 14 MiB/s, 221 objects/s recovering 2024-03-26T03:03:21.778 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:21.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:21.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:21.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:22.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:22.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:22.144 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:22.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:22.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.328 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:22.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:22.363 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:03:22.364 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:03:22.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:22.491 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:22.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:22.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:22.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:22.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:22.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:22.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:22.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:22.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:23.041 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:23.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:23.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:23.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:23.325 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:23.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.474 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:23.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:23.511 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:23 smithi066 ceph-mon[29666]: pgmap v174: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 332/199893 objects degraded (0.166%); 9672/199893 objects misplaced (4.839%); 14 MiB/s, 220 objects/s recovering 2024-03-26T03:03:23.512 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:23.512 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:23 smithi066 ceph-mon[37920]: pgmap v174: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 332/199893 objects degraded (0.166%); 9672/199893 objects misplaced (4.839%); 14 MiB/s, 220 objects/s recovering 2024-03-26T03:03:23.512 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:23.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:23.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:23.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:23.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:23 smithi192 ceph-mon[33110]: pgmap v174: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 332/199893 objects degraded (0.166%); 9672/199893 objects misplaced (4.839%); 14 MiB/s, 220 objects/s recovering 2024-03-26T03:03:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:23.835 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:23.876 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:23.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:23.980 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.029 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:24.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:24.070 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:03:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:03:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:03:24.130 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:24.175 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:24.276 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:24.326 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:24.430 INFO:tasks.thrashosds.thrasher:j is {'epoch': 139, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:03:17.277593+0000', 'last_up_change': '2024-03-26T02:58:43.841339+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-26T03:00:53.479167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '111', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1500000953674316, 'score_stable': 1.850000023841858, 'optimal_score': 1, 'raw_score_acting': 2.1500000953674316, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 137, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 131, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 125, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 131, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:02:08.084205+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:02:37.301346+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:02:39.489448+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:03:08.469232+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:02:27.234014+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:02:44.394806+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:01:56.633805+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:02:41.184682+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.10', 'osds': [3, 1]}, {'pgid': '7.17', 'osds': [0, 2]}, {'pgid': '7.18', 'osds': [2, 7]}, {'pgid': '7.19', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:03:24.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:03:24.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:24.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:24.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:24.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:24.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 332/199893 objects degraded (0.166%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:24.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 332/199893 objects degraded (0.166%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:24.692 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 332/199893 objects degraded (0.166%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:24.756 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:24.858 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:24.928 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:24.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:25.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:25.059 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:25.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:25.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:25.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:03:25.289 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:25.353 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:25 smithi066 ceph-mon[29666]: pgmap v175: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 677 op/s; 332/195036 objects degraded (0.170%); 9672/195036 objects misplaced (4.959%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T03:03:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1408389150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:25.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:25 smithi066 ceph-mon[37920]: pgmap v175: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 677 op/s; 332/195036 objects degraded (0.170%); 9672/195036 objects misplaced (4.959%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T03:03:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1408389150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:25.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:25.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:25.496 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:03:25.496 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:03:25.497 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:03:25.498 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:03:25.498 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:03:25.498 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:03:25.583 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:25.646 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:25 smithi192 ceph-mon[33110]: pgmap v175: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 677 op/s; 332/195036 objects degraded (0.170%); 9672/195036 objects misplaced (4.959%); 6.9 MiB/s, 110 objects/s recovering 2024-03-26T03:03:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1408389150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:25.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:25.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:25.876 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:25.928 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:25.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:26.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:26.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:26.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.214 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:26.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:26.228 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:26.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:26.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:26.369 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:03:26.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:26 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3335338241' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:03:26.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:26 smithi066 ceph-mon[29666]: from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:03:26.422 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:26 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3335338241' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:03:26.422 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:26 smithi066 ceph-mon[37920]: from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:03:26.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:26.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:26.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:26.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:26.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:26 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3335338241' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:03:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:26 smithi192 ceph-mon[33110]: from='client.34781 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:03:26.756 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:26.858 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:26.913 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:26.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:27.015 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:27.064 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:27.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:27.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:27.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:27.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:27.446 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:27.497 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:27.598 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:03:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:03:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[29666]: pgmap v176: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 845 op/s; 332/186648 objects degraded (0.178%); 8140/186648 objects misplaced (4.361%); 10 MiB/s, 163 objects/s recovering 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[29666]: from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[29666]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[37920]: pgmap v176: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 845 op/s; 332/186648 objects degraded (0.178%); 8140/186648 objects misplaced (4.361%); 10 MiB/s, 163 objects/s recovering 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[37920]: from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:03:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:27 smithi066 ceph-mon[37920]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T03:03:27.639 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:27 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:03:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:27 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:03:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:27 smithi192 ceph-mon[33110]: pgmap v176: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 845 op/s; 332/186648 objects degraded (0.178%); 8140/186648 objects misplaced (4.361%); 10 MiB/s, 163 objects/s recovering 2024-03-26T03:03:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:27 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:03:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:27 smithi192 ceph-mon[33110]: from='client.34781 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:03:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:27 smithi192 ceph-mon[33110]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T03:03:27.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:27.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:27.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:27.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:27.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:28.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:28.050 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:28.151 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:28.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:28.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:03:28.308 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.350 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:28.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:28.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:28.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:28.578 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:28 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:03:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:28 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:03:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:28 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:03:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:28 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:03:28.629 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:28.731 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:28 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:03:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:28 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:03:28.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:28.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:28.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:28.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:29.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:29.057 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:29.159 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:29.198 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:29.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:29.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:29.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:29.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:29 smithi192 ceph-mon[33110]: pgmap v178: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s; 332/186648 objects degraded (0.178%); 8140/186648 objects misplaced (4.361%); 9.9 MiB/s, 159 objects/s recovering 2024-03-26T03:03:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 332/186648 objects degraded (0.178%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:29.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:03:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:03:29.565 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:29.606 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:29 smithi066 ceph-mon[29666]: pgmap v178: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s; 332/186648 objects degraded (0.178%); 8140/186648 objects misplaced (4.361%); 9.9 MiB/s, 159 objects/s recovering 2024-03-26T03:03:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 332/186648 objects degraded (0.178%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:29 smithi066 ceph-mon[37920]: pgmap v178: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s; 332/186648 objects degraded (0.178%); 8140/186648 objects misplaced (4.361%); 9.9 MiB/s, 159 objects/s recovering 2024-03-26T03:03:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 332/186648 objects degraded (0.178%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:29.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:29.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:29.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:29.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:29.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:29.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:29.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:30.094 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:30.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:30.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:30.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:30.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:30.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:30.540 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:30.643 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:30.674 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:30 smithi192 ceph-mon[33110]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T03:03:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:30 smithi192 ceph-mon[33110]: 6.2 scrub starts 2024-03-26T03:03:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:30 smithi192 ceph-mon[33110]: 6.2 scrub ok 2024-03-26T03:03:30.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:30.781 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:30.781 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[29666]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T03:03:30.781 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[29666]: 6.2 scrub starts 2024-03-26T03:03:30.781 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[29666]: 6.2 scrub ok 2024-03-26T03:03:30.782 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:30.782 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[37920]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T03:03:30.782 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[37920]: 6.2 scrub starts 2024-03-26T03:03:30.782 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:30 smithi066 ceph-mon[37920]: 6.2 scrub ok 2024-03-26T03:03:30.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:30.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:30.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:30.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:31.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:31.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:31.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:31.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:31.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:31.351 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:31.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:31.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:31.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:31.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:31.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:31 smithi066 ceph-mon[37920]: pgmap v180: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 332/173949 objects degraded (0.191%); 6390/173949 objects misplaced (3.673%); 12 MiB/s, 191 objects/s recovering 2024-03-26T03:03:31.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:31 smithi066 ceph-mon[37920]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T03:03:31.660 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:31 smithi066 ceph-mon[29666]: pgmap v180: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 332/173949 objects degraded (0.191%); 6390/173949 objects misplaced (3.673%); 12 MiB/s, 191 objects/s recovering 2024-03-26T03:03:31.660 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:31 smithi066 ceph-mon[29666]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T03:03:31.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:31 smithi192 ceph-mon[33110]: pgmap v180: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 332/173949 objects degraded (0.191%); 6390/173949 objects misplaced (3.673%); 12 MiB/s, 191 objects/s recovering 2024-03-26T03:03:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:31 smithi192 ceph-mon[33110]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T03:03:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:31.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:31.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:31.909 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:31.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:32.010 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:32.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:32.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:32.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:32.286 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:32.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:32.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:32.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:32.570 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:32.614 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:32.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:32 smithi192 ceph-mon[33110]: 7.16 scrub starts 2024-03-26T03:03:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:32 smithi192 ceph-mon[33110]: 7.16 scrub ok 2024-03-26T03:03:32.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:32.848 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:32 smithi066 ceph-mon[29666]: 7.16 scrub starts 2024-03-26T03:03:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:32 smithi066 ceph-mon[29666]: 7.16 scrub ok 2024-03-26T03:03:32.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:32 smithi066 ceph-mon[37920]: 7.16 scrub starts 2024-03-26T03:03:32.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:32 smithi066 ceph-mon[37920]: 7.16 scrub ok 2024-03-26T03:03:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:32.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:32.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:32.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:33.044 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:33.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:33.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:33.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:33.320 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:33.422 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:33.466 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:33.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:33.620 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:33.721 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:33 smithi192 ceph-mon[33110]: pgmap v182: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 706 op/s; 332/173949 objects degraded (0.191%); 6390/173949 objects misplaced (3.673%); 8.6 MiB/s, 136 objects/s recovering 2024-03-26T03:03:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:33 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:03:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:33 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:03:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.771 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:33.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-mon[29666]: pgmap v182: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 706 op/s; 332/173949 objects degraded (0.191%); 6390/173949 objects misplaced (3.673%); 8.6 MiB/s, 136 objects/s recovering 2024-03-26T03:03:33.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:03:33.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:03:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-mon[37920]: pgmap v182: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 706 op/s; 332/173949 objects degraded (0.191%); 6390/173949 objects misplaced (3.673%); 8.6 MiB/s, 136 objects/s recovering 2024-03-26T03:03:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:03:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:03:33.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:33.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:33.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:34.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:34.063 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:03:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:03:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:03:34.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:34.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:34.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:34.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:34.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.474 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:34.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:34.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:34.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:34 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:03:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:34 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:03:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:34 smithi192 ceph-mon[33110]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T03:03:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 332/173949 objects degraded (0.191%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:34 smithi192 ceph-mon[33110]: pgmap v184: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 195 op/s; 332/169542 objects degraded (0.196%); 6390/169542 objects misplaced (3.769%) 2024-03-26T03:03:34.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:34.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:34.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:03:34.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[29666]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 332/173949 objects degraded (0.191%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[29666]: pgmap v184: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 195 op/s; 332/169542 objects degraded (0.196%); 6390/169542 objects misplaced (3.769%) 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[37920]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 332/173949 objects degraded (0.191%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:34 smithi066 ceph-mon[37920]: pgmap v184: 158 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 195 op/s; 332/169542 objects degraded (0.196%); 6390/169542 objects misplaced (3.769%) 2024-03-26T03:03:34.908 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:34.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:35.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:35.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:35.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:35.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:35.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:35.293 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:35.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:35.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:35.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:35.471 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:35 smithi066 ceph-mon[29666]: 4.1a scrub starts 2024-03-26T03:03:35.471 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:35 smithi066 ceph-mon[37920]: 4.1a scrub starts 2024-03-26T03:03:35.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:35.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:35.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:35.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:35 smithi192 ceph-mon[33110]: 4.1a scrub starts 2024-03-26T03:03:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:35 smithi192 ceph-mon[33110]: 4.1a scrub ok 2024-03-26T03:03:35.814 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:35.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:35 smithi066 ceph-mon[29666]: 4.1a scrub ok 2024-03-26T03:03:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:35 smithi066 ceph-mon[37920]: 4.1a scrub ok 2024-03-26T03:03:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.861 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:35.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:35.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:35.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:35.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:36.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:36.140 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:36.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:36.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:36.290 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:36.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:36.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:36.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:36.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:36.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:36.525 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:36.572 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:36.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:36 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:03:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:36 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:03:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:36 smithi066 ceph-mon[29666]: pgmap v185: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 769 B/s rd, 0 B/s wr, 209 op/s; 332/161721 objects degraded (0.205%); 4884/161721 objects misplaced (3.020%) 2024-03-26T03:03:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:36 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:03:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:36 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:03:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:36 smithi066 ceph-mon[37920]: pgmap v185: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 769 B/s rd, 0 B/s wr, 209 op/s; 332/161721 objects degraded (0.205%); 4884/161721 objects misplaced (3.020%) 2024-03-26T03:03:36.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:36.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:36.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:36 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:03:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:36 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:03:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:36 smithi192 ceph-mon[33110]: pgmap v185: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 769 B/s rd, 0 B/s wr, 209 op/s; 332/161721 objects degraded (0.205%); 4884/161721 objects misplaced (3.020%) 2024-03-26T03:03:36.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:36.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:36.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:36.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.006 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:37.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:37.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:37.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:37.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:37.238 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:03:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:37.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:37.366 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:37.424 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:37.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:37.583 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:37.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:37.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:37.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:37.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:37.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:37.953 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:37.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:37.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:38.052 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:37 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:03:38.052 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:37 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:03:38.052 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:37 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:03:38.053 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:37 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:03:38.077 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:38.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:38.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:38.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:37 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:03:38.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:37 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:03:38.281 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:38.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:38.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:38.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:38.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:38.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:38.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:38.777 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:38.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:38.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:38.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:38.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:39.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:39.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:38 smithi066 ceph-mon[29666]: pgmap v186: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 332/161721 objects degraded (0.205%); 4884/161721 objects misplaced (3.020%) 2024-03-26T03:03:39.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:39.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:38 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 332/161721 objects degraded (0.205%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:39.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:38 smithi066 ceph-mon[37920]: pgmap v186: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 332/161721 objects degraded (0.205%); 4884/161721 objects misplaced (3.020%) 2024-03-26T03:03:39.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:39.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:38 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 332/161721 objects degraded (0.205%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:39.120 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:39.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:39.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:03:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:03:39.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:38 smithi192 ceph-mon[33110]: pgmap v186: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 332/161721 objects degraded (0.205%); 4884/161721 objects misplaced (3.020%) 2024-03-26T03:03:39.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:39.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:38 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 332/161721 objects degraded (0.205%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:39.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:39.309 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:03:39.313 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:03:39.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:39.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:39.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:39.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:39.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:39.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:39.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:39.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:39.888 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:39.932 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:39.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:40.034 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:40.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:40.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:40.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:39 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:39 smithi192 ceph-mon[33110]: from='client.14928 v1:172.21.15.66:0/2349726469' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:03:40.267 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:39 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:40.267 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:39 smithi066 ceph-mon[29666]: from='client.14928 v1:172.21.15.66:0/2349726469' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:03:40.267 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:39 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:40.267 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:39 smithi066 ceph-mon[37920]: from='client.14928 v1:172.21.15.66:0/2349726469' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:03:40.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:40.323 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-03-26T03:03:38.008010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3518208448,"num_objects":53907,"num_object_clones":0,"num_object_copies":161721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":332,"num_objects_misplaced":4884,"num_objects_unfound":0,"num_objects_dirty":53907,"num_whiteouts":0,"num_read":1491,"num_read_kb":1305,"num_write":119115,"num_write_kb":4684222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19342,"num_bytes_recovered":1267599274,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167126,"ondisk_log_size":167126,"up":474,"acting":471,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":488,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12589416,"kb_used_data":11542800,"kb_used_omap":12,"kb_used_meta":1046195,"kb_avail":737371800,"statfs":{"total":767960285184,"available":755068723200,"internally_reserved":0,"allocated":11819827200,"data_stored":11816759284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1071304240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,9,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-267091968,"num_objects":-4076,"num_object_clones":0,"num_object_copies":-12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1506,"num_objects_unfound":0,"num_objects_dirty":-4076,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-460,"num_write_kb":-202144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1679,"num_bytes_recovered":-110034944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003563"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399160+0000","last_change":"2024-03-26T03:00:51.383778+0000","last_active":"2024-03-26T03:03:33.399160+0000","last_peered":"2024-03-26T03:03:33.399160+0000","last_clean":"2024-03-26T03:03:33.399160+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:03:33.399160+0000","last_undegraded":"2024-03-26T03:03:33.399160+0000","last_fullsized":"2024-03-26T03:03:33.399160+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.383695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.383695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":177,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391541+0000","last_change":"2024-03-26T03:03:27.420047+0000","last_active":"2024-03-26T03:03:33.391541+0000","last_peered":"2024-03-26T03:03:33.391541+0000","last_clean":"2024-03-26T03:03:33.391541+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:03:33.391541+0000","last_undegraded":"2024-03-26T03:03:33.391541+0000","last_fullsized":"2024-03-26T03:03:33.391541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:27.419925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:27.419925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:56.331025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408939+0000","last_change":"2024-03-26T03:01:49.986072+0000","last_active":"2024-03-26T03:03:33.408939+0000","last_peered":"2024-03-26T03:03:33.408939+0000","last_clean":"2024-03-26T03:03:33.408939+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:03:33.408939+0000","last_undegraded":"2024-03-26T03:03:33.408939+0000","last_fullsized":"2024-03-26T03:03:33.408939+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.985144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.985144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":248,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398142+0000","last_change":"2024-03-26T03:00:50.380208+0000","last_active":"2024-03-26T03:03:33.398142+0000","last_peered":"2024-03-26T03:03:33.398142+0000","last_clean":"2024-03-26T03:03:33.398142+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:03:33.398142+0000","last_undegraded":"2024-03-26T03:03:33.398142+0000","last_fullsized":"2024-03-26T03:03:33.398142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:00:50.380106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:50.380106+0000","objects_scrubbed":11,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":165,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391260+0000","last_change":"2024-03-26T03:00:51.327781+0000","last_active":"2024-03-26T03:03:33.391260+0000","last_peered":"2024-03-26T03:03:33.391260+0000","last_clean":"2024-03-26T03:03:33.391260+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:03:33.391260+0000","last_undegraded":"2024-03-26T03:03:33.391260+0000","last_fullsized":"2024-03-26T03:03:33.391260+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.327687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.327687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":162,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402896+0000","last_change":"2024-03-26T02:59:00.932288+0000","last_active":"2024-03-26T03:03:33.402896+0000","last_peered":"2024-03-26T03:03:33.402896+0000","last_clean":"2024-03-26T03:03:33.402896+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:03:33.402896+0000","last_undegraded":"2024-03-26T03:03:33.402896+0000","last_fullsized":"2024-03-26T03:03:33.402896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397050+0000","last_change":"2024-03-26T03:00:41.464704+0000","last_active":"2024-03-26T03:03:33.397050+0000","last_peered":"2024-03-26T03:03:33.397050+0000","last_clean":"2024-03-26T03:03:33.397050+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:03:33.397050+0000","last_undegraded":"2024-03-26T03:03:33.397050+0000","last_fullsized":"2024-03-26T03:03:33.397050+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:41.464590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.464590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":201,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391227+0000","last_change":"2024-03-26T03:00:41.276506+0000","last_active":"2024-03-26T03:03:33.391227+0000","last_peered":"2024-03-26T03:03:33.391227+0000","last_clean":"2024-03-26T03:03:33.391227+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:03:33.391227+0000","last_undegraded":"2024-03-26T03:03:33.391227+0000","last_fullsized":"2024-03-26T03:03:33.391227+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:41.276382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.276382+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"143'6983","reported_seq":14111,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.654270+0000","last_change":"2024-03-26T03:03:30.405471+0000","last_active":"2024-03-26T03:03:34.654270+0000","last_peered":"2024-03-26T03:03:34.654270+0000","last_clean":"2024-03-26T03:03:34.654270+0000","last_became_active":"2024-03-26T03:03:30.396029+0000","last_became_peered":"2024-03-26T03:03:30.396029+0000","last_unstale":"2024-03-26T03:03:34.654270+0000","last_undegraded":"2024-03-26T03:03:34.654270+0000","last_fullsized":"2024-03-26T03:03:34.654270+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:35.867239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110198784,"num_objects":1682,"num_object_clones":0,"num_object_copies":5046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4608,"num_write_kb":201248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2334,"num_bytes_recovered":152961024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":165,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.407778+0000","last_change":"2024-03-26T03:00:25.999429+0000","last_active":"2024-03-26T03:03:33.407778+0000","last_peered":"2024-03-26T03:03:33.407778+0000","last_clean":"2024-03-26T03:03:33.407778+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:03:33.407778+0000","last_undegraded":"2024-03-26T03:03:33.407778+0000","last_fullsized":"2024-03-26T03:03:33.407778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:25.999320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.999320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397713+0000","last_change":"2024-03-26T03:01:39.191861+0000","last_active":"2024-03-26T03:03:33.397713+0000","last_peered":"2024-03-26T03:03:33.397713+0000","last_clean":"2024-03-26T03:03:33.397713+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:03:33.397713+0000","last_undegraded":"2024-03-26T03:03:33.397713+0000","last_fullsized":"2024-03-26T03:03:33.397713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:39.191709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:39.191709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398382+0000","last_change":"2024-03-26T03:00:51.169262+0000","last_active":"2024-03-26T03:03:33.398382+0000","last_peered":"2024-03-26T03:03:33.398382+0000","last_clean":"2024-03-26T03:03:33.398382+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:03:33.398382+0000","last_undegraded":"2024-03-26T03:03:33.398382+0000","last_fullsized":"2024-03-26T03:03:33.398382+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.169167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.169167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":195,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392540+0000","last_change":"2024-03-26T03:01:45.305022+0000","last_active":"2024-03-26T03:03:33.392540+0000","last_peered":"2024-03-26T03:03:33.392540+0000","last_clean":"2024-03-26T03:03:33.392540+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:03:33.392540+0000","last_undegraded":"2024-03-26T03:03:33.392540+0000","last_fullsized":"2024-03-26T03:03:33.392540+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:01:45.304877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:45.304877+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"143'6797","reported_seq":13865,"reported_epoch":143,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-26T03:03:34.934064+0000","last_change":"2024-03-26T03:02:38.161933+0000","last_active":"2024-03-26T03:03:34.934064+0000","last_peered":"2024-03-26T03:03:34.934064+0000","last_clean":"2024-03-26T03:01:52.264965+0000","last_became_active":"2024-03-26T03:01:54.659973+0000","last_became_peered":"2024-03-26T03:01:54.659973+0000","last_unstale":"2024-03-26T03:03:34.934064+0000","last_undegraded":"2024-03-26T03:03:34.934064+0000","last_fullsized":"2024-03-26T03:01:54.281915+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6797,"log_dups_size":0,"ondisk_log_size":6797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105152512,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1866,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4485,"num_write_kb":194848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1738,"num_bytes_recovered":113901568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[0,2],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":622},{"shards":"0,2","objects":983}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.404362+0000","last_change":"2024-03-26T03:00:33.322325+0000","last_active":"2024-03-26T03:03:33.404362+0000","last_peered":"2024-03-26T03:03:33.404362+0000","last_clean":"2024-03-26T03:03:33.404362+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:03:33.404362+0000","last_undegraded":"2024-03-26T03:03:33.404362+0000","last_fullsized":"2024-03-26T03:03:33.404362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:33.322244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:33.322244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":177,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408065+0000","last_change":"2024-03-26T03:01:42.231002+0000","last_active":"2024-03-26T03:03:33.408065+0000","last_peered":"2024-03-26T03:03:33.408065+0000","last_clean":"2024-03-26T03:03:33.408065+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:03:33.408065+0000","last_undegraded":"2024-03-26T03:03:33.408065+0000","last_fullsized":"2024-03-26T03:03:33.408065+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:42.230855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.230855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":186,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397885+0000","last_change":"2024-03-26T03:01:50.265230+0000","last_active":"2024-03-26T03:03:33.397885+0000","last_peered":"2024-03-26T03:03:33.397885+0000","last_clean":"2024-03-26T03:03:33.397885+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:03:33.397885+0000","last_undegraded":"2024-03-26T03:03:33.397885+0000","last_fullsized":"2024-03-26T03:03:33.397885+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:50.265114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.265114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":213,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391384+0000","last_change":"2024-03-26T03:00:06.927566+0000","last_active":"2024-03-26T03:03:33.391384+0000","last_peered":"2024-03-26T03:03:33.391384+0000","last_clean":"2024-03-26T03:03:33.391384+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:03:33.391384+0000","last_undegraded":"2024-03-26T03:03:33.391384+0000","last_fullsized":"2024-03-26T03:03:33.391384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:06.927439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:06.927439+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.400279+0000","last_change":"2024-03-26T03:00:52.383684+0000","last_active":"2024-03-26T03:03:33.400279+0000","last_peered":"2024-03-26T03:03:33.400279+0000","last_clean":"2024-03-26T03:03:33.400279+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:03:33.400279+0000","last_undegraded":"2024-03-26T03:03:33.400279+0000","last_fullsized":"2024-03-26T03:03:33.400279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.383607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.383607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392796+0000","last_change":"2024-03-26T03:00:46.268131+0000","last_active":"2024-03-26T03:03:33.392796+0000","last_peered":"2024-03-26T03:03:33.392796+0000","last_clean":"2024-03-26T03:03:33.392796+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:03:33.392796+0000","last_undegraded":"2024-03-26T03:03:33.392796+0000","last_fullsized":"2024-03-26T03:03:33.392796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:46.268052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:46.268052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":186,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399253+0000","last_change":"2024-03-26T03:01:49.896560+0000","last_active":"2024-03-26T03:03:33.399253+0000","last_peered":"2024-03-26T03:03:33.399253+0000","last_clean":"2024-03-26T03:03:33.399253+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:03:33.399253+0000","last_undegraded":"2024-03-26T03:03:33.399253+0000","last_fullsized":"2024-03-26T03:03:33.399253+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":209,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397509+0000","last_change":"2024-03-26T03:00:25.578239+0000","last_active":"2024-03-26T03:03:33.397509+0000","last_peered":"2024-03-26T03:03:33.397509+0000","last_clean":"2024-03-26T03:03:33.397509+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:03:33.397509+0000","last_undegraded":"2024-03-26T03:03:33.397509+0000","last_fullsized":"2024-03-26T03:03:33.397509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:00:25.578139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.578139+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391798+0000","last_change":"2024-03-26T03:00:27.177539+0000","last_active":"2024-03-26T03:03:33.391798+0000","last_peered":"2024-03-26T03:03:33.391798+0000","last_clean":"2024-03-26T03:03:33.391798+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:03:33.391798+0000","last_undegraded":"2024-03-26T03:03:33.391798+0000","last_fullsized":"2024-03-26T03:03:33.391798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.177404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.177404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":170,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402769+0000","last_change":"2024-03-26T03:00:18.300021+0000","last_active":"2024-03-26T03:03:33.402769+0000","last_peered":"2024-03-26T03:03:33.402769+0000","last_clean":"2024-03-26T03:03:33.402769+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:03:33.402769+0000","last_undegraded":"2024-03-26T03:03:33.402769+0000","last_fullsized":"2024-03-26T03:03:33.402769+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:18.299882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:18.299882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":186,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396514+0000","last_change":"2024-03-26T03:01:34.889068+0000","last_active":"2024-03-26T03:03:33.396514+0000","last_peered":"2024-03-26T03:03:33.396514+0000","last_clean":"2024-03-26T03:03:33.396514+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:03:33.396514+0000","last_undegraded":"2024-03-26T03:03:33.396514+0000","last_fullsized":"2024-03-26T03:03:33.396514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.888991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.888991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":207,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.401668+0000","last_change":"2024-03-26T03:01:34.386450+0000","last_active":"2024-03-26T03:03:33.401668+0000","last_peered":"2024-03-26T03:03:33.401668+0000","last_clean":"2024-03-26T03:03:33.401668+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:03:33.401668+0000","last_undegraded":"2024-03-26T03:03:33.401668+0000","last_fullsized":"2024-03-26T03:03:33.401668+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:01:34.386302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.386302+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":158,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402205+0000","last_change":"2024-03-26T02:59:02.938768+0000","last_active":"2024-03-26T03:03:33.402205+0000","last_peered":"2024-03-26T03:03:33.402205+0000","last_clean":"2024-03-26T03:03:33.402205+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:03:33.402205+0000","last_undegraded":"2024-03-26T03:03:33.402205+0000","last_fullsized":"2024-03-26T03:03:33.402205+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T02:59:01.918211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.404653+0000","last_change":"2024-03-26T03:01:41.862376+0000","last_active":"2024-03-26T03:03:33.404653+0000","last_peered":"2024-03-26T03:03:33.404653+0000","last_clean":"2024-03-26T03:03:33.404653+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:03:33.404653+0000","last_undegraded":"2024-03-26T03:03:33.404653+0000","last_fullsized":"2024-03-26T03:03:33.404653+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:41.862250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:01:41.862250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398297+0000","last_change":"2024-03-26T03:00:49.420677+0000","last_active":"2024-03-26T03:03:33.398297+0000","last_peered":"2024-03-26T03:03:33.398297+0000","last_clean":"2024-03-26T03:03:33.398297+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:03:33.398297+0000","last_undegraded":"2024-03-26T03:03:33.398297+0000","last_fullsized":"2024-03-26T03:03:33.398297+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:00:49.420583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:49.420583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":206,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397703+0000","last_change":"2024-03-26T03:00:30.623048+0000","last_active":"2024-03-26T03:03:33.397703+0000","last_peered":"2024-03-26T03:03:33.397703+0000","last_clean":"2024-03-26T03:03:33.397703+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:03:33.397703+0000","last_undegraded":"2024-03-26T03:03:33.397703+0000","last_fullsized":"2024-03-26T03:03:33.397703+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:30.622926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:30.622926+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":175,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398401+0000","last_change":"2024-03-26T03:01:46.312995+0000","last_active":"2024-03-26T03:03:33.398401+0000","last_peered":"2024-03-26T03:03:33.398401+0000","last_clean":"2024-03-26T03:03:33.398401+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:03:33.398401+0000","last_undegraded":"2024-03-26T03:03:33.398401+0000","last_fullsized":"2024-03-26T03:03:33.398401+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:01:46.312845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:46.312845+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":238,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397610+0000","last_change":"2024-03-26T03:00:25.586479+0000","last_active":"2024-03-26T03:03:33.397610+0000","last_peered":"2024-03-26T03:03:33.397610+0000","last_clean":"2024-03-26T03:03:33.397610+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:03:33.397610+0000","last_undegraded":"2024-03-26T03:03:33.397610+0000","last_fullsized":"2024-03-26T03:03:33.397610+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:25.586362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.586362+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":185,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.407763+0000","last_change":"2024-03-26T03:01:54.322573+0000","last_active":"2024-03-26T03:03:33.407763+0000","last_peered":"2024-03-26T03:03:33.407763+0000","last_clean":"2024-03-26T03:03:33.407763+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:03:33.407763+0000","last_undegraded":"2024-03-26T03:03:33.407763+0000","last_fullsized":"2024-03-26T03:03:33.407763+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:54.322450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.322450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":202,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.404621+0000","last_change":"2024-03-26T03:00:36.337159+0000","last_active":"2024-03-26T03:03:33.404621+0000","last_peered":"2024-03-26T03:03:33.404621+0000","last_clean":"2024-03-26T03:03:33.404621+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:03:33.404621+0000","last_undegraded":"2024-03-26T03:03:33.404621+0000","last_fullsized":"2024-03-26T03:03:33.404621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:00:36.337006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:36.337006+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"143'6972","reported_seq":14560,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.912362+0000","last_change":"2024-03-26T03:02:52.293628+0000","last_active":"2024-03-26T03:03:33.912362+0000","last_peered":"2024-03-26T03:03:33.912362+0000","last_clean":"2024-03-26T03:03:33.912362+0000","last_became_active":"2024-03-26T03:02:52.287145+0000","last_became_peered":"2024-03-26T03:02:52.287145+0000","last_unstale":"2024-03-26T03:03:33.912362+0000","last_undegraded":"2024-03-26T03:03:33.912362+0000","last_fullsized":"2024-03-26T03:03:33.912362+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:36.863203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110100480,"num_objects":1680,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4602,"num_write_kb":201024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2488,"num_bytes_recovered":163053568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399752+0000","last_change":"2024-03-26T03:01:34.770522+0000","last_active":"2024-03-26T03:03:33.399752+0000","last_peered":"2024-03-26T03:03:33.399752+0000","last_clean":"2024-03-26T03:03:33.399752+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:03:33.399752+0000","last_undegraded":"2024-03-26T03:03:33.399752+0000","last_fullsized":"2024-03-26T03:03:33.399752+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.770057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.770057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399574+0000","last_change":"2024-03-26T03:01:32.215005+0000","last_active":"2024-03-26T03:03:33.399574+0000","last_peered":"2024-03-26T03:03:33.399574+0000","last_clean":"2024-03-26T03:03:33.399574+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:03:33.399574+0000","last_undegraded":"2024-03-26T03:03:33.399574+0000","last_fullsized":"2024-03-26T03:03:33.399574+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.214884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.214884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397440+0000","last_change":"2024-03-26T03:00:29.597498+0000","last_active":"2024-03-26T03:03:33.397440+0000","last_peered":"2024-03-26T03:03:33.397440+0000","last_clean":"2024-03-26T03:03:33.397440+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:03:33.397440+0000","last_undegraded":"2024-03-26T03:03:33.397440+0000","last_fullsized":"2024-03-26T03:03:33.397440+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:29.597404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:29.597404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"71'17","reported_seq":205,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398608+0000","last_change":"2024-03-26T03:00:36.458059+0000","last_active":"2024-03-26T03:03:33.398608+0000","last_peered":"2024-03-26T03:03:33.398608+0000","last_clean":"2024-03-26T03:03:33.398608+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:03:33.398608+0000","last_undegraded":"2024-03-26T03:03:33.398608+0000","last_fullsized":"2024-03-26T03:03:33.398608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:36.457970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:36.457970+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"143'6995","reported_seq":14162,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.880176+0000","last_change":"2024-03-26T03:02:53.171830+0000","last_active":"2024-03-26T03:03:34.880176+0000","last_peered":"2024-03-26T03:03:34.880176+0000","last_clean":"2024-03-26T03:03:34.880176+0000","last_became_active":"2024-03-26T03:02:53.163455+0000","last_became_peered":"2024-03-26T03:02:53.163455+0000","last_unstale":"2024-03-26T03:03:34.880176+0000","last_undegraded":"2024-03-26T03:03:34.880176+0000","last_fullsized":"2024-03-26T03:03:34.880176+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:35.867216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108724224,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":201472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":153944064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":174,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397399+0000","last_change":"2024-03-26T03:01:44.279749+0000","last_active":"2024-03-26T03:03:33.397399+0000","last_peered":"2024-03-26T03:03:33.397399+0000","last_clean":"2024-03-26T03:03:33.397399+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:03:33.397399+0000","last_undegraded":"2024-03-26T03:03:33.397399+0000","last_fullsized":"2024-03-26T03:03:33.397399+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:44.279630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:44.279630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":170,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396482+0000","last_change":"2024-03-26T03:00:21.571415+0000","last_active":"2024-03-26T03:03:33.396482+0000","last_peered":"2024-03-26T03:03:33.396482+0000","last_clean":"2024-03-26T03:03:33.396482+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:03:33.396482+0000","last_undegraded":"2024-03-26T03:03:33.396482+0000","last_fullsized":"2024-03-26T03:03:33.396482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:21.571273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:21.571273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":185,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408918+0000","last_change":"2024-03-26T03:01:33.180277+0000","last_active":"2024-03-26T03:03:33.408918+0000","last_peered":"2024-03-26T03:03:33.408918+0000","last_clean":"2024-03-26T03:03:33.408918+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:03:33.408918+0000","last_undegraded":"2024-03-26T03:03:33.408918+0000","last_fullsized":"2024-03-26T03:03:33.408918+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.180027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.180027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399273+0000","last_change":"2024-03-26T03:01:37.776671+0000","last_active":"2024-03-26T03:03:33.399273+0000","last_peered":"2024-03-26T03:03:33.399273+0000","last_clean":"2024-03-26T03:03:33.399273+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:03:33.399273+0000","last_undegraded":"2024-03-26T03:03:33.399273+0000","last_fullsized":"2024-03-26T03:03:33.399273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"143'6994","reported_seq":10700,"reported_epoch":143,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:03:34.569810+0000","last_change":"2024-03-26T03:02:50.195053+0000","last_active":"2024-03-26T03:03:34.569810+0000","last_peered":"2024-03-26T03:03:34.569810+0000","last_clean":"2024-03-26T03:01:52.256933+0000","last_became_active":"2024-03-26T03:01:58.390747+0000","last_became_peered":"2024-03-26T03:01:58.390747+0000","last_unstale":"2024-03-26T03:03:34.569810+0000","last_undegraded":"2024-03-26T03:01:57.306815+0000","last_fullsized":"2024-03-26T03:01:54.278118+0000","mapping_epoch":114,"log_start":"99'788","ondisk_log_start":"99'788","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6206,"log_dups_size":0,"ondisk_log_size":6206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":332,"num_objects_misplaced":291,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4614,"num_write_kb":200576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":117637120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,1],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3","objects":1031},{"shards":"1,3,5","objects":291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398037+0000","last_change":"2024-03-26T03:00:34.438665+0000","last_active":"2024-03-26T03:03:33.398037+0000","last_peered":"2024-03-26T03:03:33.398037+0000","last_clean":"2024-03-26T03:03:33.398037+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:03:33.398037+0000","last_undegraded":"2024-03-26T03:03:33.398037+0000","last_fullsized":"2024-03-26T03:03:33.398037+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:34.438582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:34.438582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":162,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402878+0000","last_change":"2024-03-26T02:59:00.936834+0000","last_active":"2024-03-26T03:03:33.402878+0000","last_peered":"2024-03-26T03:03:33.402878+0000","last_clean":"2024-03-26T03:03:33.402878+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:03:33.402878+0000","last_undegraded":"2024-03-26T03:03:33.402878+0000","last_fullsized":"2024-03-26T03:03:33.402878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T02:58:59.914157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402435+0000","last_change":"2024-03-26T03:00:02.224750+0000","last_active":"2024-03-26T03:03:33.402435+0000","last_peered":"2024-03-26T03:03:33.402435+0000","last_clean":"2024-03-26T03:03:33.402435+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:03:33.402435+0000","last_undegraded":"2024-03-26T03:03:33.402435+0000","last_fullsized":"2024-03-26T03:03:33.402435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:02.224639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:02.224639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":192,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392410+0000","last_change":"2024-03-26T03:01:42.243460+0000","last_active":"2024-03-26T03:03:33.392410+0000","last_peered":"2024-03-26T03:03:33.392410+0000","last_clean":"2024-03-26T03:03:33.392410+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:03:33.392410+0000","last_undegraded":"2024-03-26T03:03:33.392410+0000","last_fullsized":"2024-03-26T03:03:33.392410+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:01:42.243318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.243318+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"143'6902","reported_seq":11725,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.871248+0000","last_change":"2024-03-26T03:02:27.288770+0000","last_active":"2024-03-26T03:03:33.871248+0000","last_peered":"2024-03-26T03:03:33.871248+0000","last_clean":"2024-03-26T03:03:33.871248+0000","last_became_active":"2024-03-26T03:02:27.284219+0000","last_became_peered":"2024-03-26T03:02:27.284219+0000","last_unstale":"2024-03-26T03:03:33.871248+0000","last_undegraded":"2024-03-26T03:03:33.871248+0000","last_fullsized":"2024-03-26T03:03:33.871248+0000","mapping_epoch":121,"log_start":"99'182","ondisk_log_start":"99'182","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:37.658775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115474432,"num_objects":1762,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":201920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2350,"num_bytes_recovered":154009600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403956+0000","last_change":"2024-03-26T03:00:44.275620+0000","last_active":"2024-03-26T03:03:33.403956+0000","last_peered":"2024-03-26T03:03:33.403956+0000","last_clean":"2024-03-26T03:03:33.403956+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:03:33.403956+0000","last_undegraded":"2024-03-26T03:03:33.403956+0000","last_fullsized":"2024-03-26T03:03:33.403956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:44.275533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:44.275533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":170,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399469+0000","last_change":"2024-03-26T03:00:17.616155+0000","last_active":"2024-03-26T03:03:33.399469+0000","last_peered":"2024-03-26T03:03:33.399469+0000","last_clean":"2024-03-26T03:03:33.399469+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:03:33.399469+0000","last_undegraded":"2024-03-26T03:03:33.399469+0000","last_fullsized":"2024-03-26T03:03:33.399469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:17.616013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.616013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":186,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396680+0000","last_change":"2024-03-26T03:01:33.891331+0000","last_active":"2024-03-26T03:03:33.396680+0000","last_peered":"2024-03-26T03:03:33.396680+0000","last_clean":"2024-03-26T03:03:33.396680+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:03:33.396680+0000","last_undegraded":"2024-03-26T03:03:33.396680+0000","last_fullsized":"2024-03-26T03:03:33.396680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.891247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.891247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":190,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402661+0000","last_change":"2024-03-26T03:01:40.604091+0000","last_active":"2024-03-26T03:03:33.402661+0000","last_peered":"2024-03-26T03:03:33.402661+0000","last_clean":"2024-03-26T03:03:33.402661+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:03:33.402661+0000","last_undegraded":"2024-03-26T03:03:33.402661+0000","last_fullsized":"2024-03-26T03:03:33.402661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-26T03:01:40.603834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.603834+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"143'6986","reported_seq":12014,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.774976+0000","last_change":"2024-03-26T03:03:31.833200+0000","last_active":"2024-03-26T03:03:33.774976+0000","last_peered":"2024-03-26T03:03:33.774976+0000","last_clean":"2024-03-26T03:03:33.774976+0000","last_became_active":"2024-03-26T03:02:38.934091+0000","last_became_peered":"2024-03-26T03:02:38.934091+0000","last_unstale":"2024-03-26T03:03:33.774976+0000","last_undegraded":"2024-03-26T03:03:33.774976+0000","last_fullsized":"2024-03-26T03:03:33.774976+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"142'6947","last_scrub_stamp":"2024-03-26T03:03:31.832793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:31.832793+0000","objects_scrubbed":1782,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:59.087361+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113836032,"num_objects":1737,"num_object_clones":0,"num_object_copies":5211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392353+0000","last_change":"2024-03-26T03:01:53.315527+0000","last_active":"2024-03-26T03:03:33.392353+0000","last_peered":"2024-03-26T03:03:33.392353+0000","last_clean":"2024-03-26T03:03:33.392353+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:03:33.392353+0000","last_undegraded":"2024-03-26T03:03:33.392353+0000","last_fullsized":"2024-03-26T03:03:33.392353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:01:53.315442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.315442+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":234,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398189+0000","last_change":"2024-03-26T03:00:32.418258+0000","last_active":"2024-03-26T03:03:33.398189+0000","last_peered":"2024-03-26T03:03:33.398189+0000","last_clean":"2024-03-26T03:03:33.398189+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:03:33.398189+0000","last_undegraded":"2024-03-26T03:03:33.398189+0000","last_fullsized":"2024-03-26T03:03:33.398189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:32.418162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.418162+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408216+0000","last_change":"2024-03-26T03:00:45.281332+0000","last_active":"2024-03-26T03:03:33.408216+0000","last_peered":"2024-03-26T03:03:33.408216+0000","last_clean":"2024-03-26T03:03:33.408216+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:03:33.408216+0000","last_undegraded":"2024-03-26T03:03:33.408216+0000","last_fullsized":"2024-03-26T03:03:33.408216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.281248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.281248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":201,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408331+0000","last_change":"2024-03-26T03:00:31.064878+0000","last_active":"2024-03-26T03:03:33.408331+0000","last_peered":"2024-03-26T03:03:33.408331+0000","last_clean":"2024-03-26T03:03:33.408331+0000","last_became_active":"2024-03-26T02:58:58.945477+0000","last_became_peered":"2024-03-26T02:58:58.945477+0000","last_unstale":"2024-03-26T03:03:33.408331+0000","last_undegraded":"2024-03-26T03:03:33.408331+0000","last_fullsized":"2024-03-26T03:03:33.408331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:31.064716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:31.064716+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.17","version":"143'7098","reported_seq":13345,"reported_epoch":143,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-26T03:03:34.914036+0000","last_change":"2024-03-26T03:03:15.542853+0000","last_active":"2024-03-26T03:03:34.914036+0000","last_peered":"2024-03-26T03:03:34.914036+0000","last_clean":"2024-03-26T03:01:52.271240+0000","last_became_active":"2024-03-26T03:01:54.698047+0000","last_became_peered":"2024-03-26T03:01:54.698047+0000","last_unstale":"2024-03-26T03:03:34.914036+0000","last_undegraded":"2024-03-26T03:03:34.914036+0000","last_fullsized":"2024-03-26T03:01:54.283032+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":7098,"log_dups_size":0,"ondisk_log_size":7098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111869952,"num_objects":1707,"num_object_clones":0,"num_object_copies":5121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2727,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4705,"num_write_kb":205184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1534,"num_bytes_recovered":100532224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[0,2],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,2","objects":798},{"shards":"0,3,6","objects":909}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":174,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403874+0000","last_change":"2024-03-26T03:01:39.883186+0000","last_active":"2024-03-26T03:03:33.403874+0000","last_peered":"2024-03-26T03:03:33.403874+0000","last_clean":"2024-03-26T03:03:33.403874+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:03:33.403874+0000","last_undegraded":"2024-03-26T03:03:33.403874+0000","last_fullsized":"2024-03-26T03:03:33.403874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:39.883059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:39.883059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":233,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408192+0000","last_change":"2024-03-26T03:00:10.855859+0000","last_active":"2024-03-26T03:03:33.408192+0000","last_peered":"2024-03-26T03:03:33.408192+0000","last_clean":"2024-03-26T03:03:33.408192+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:03:33.408192+0000","last_undegraded":"2024-03-26T03:03:33.408192+0000","last_fullsized":"2024-03-26T03:03:33.408192+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:10.855698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.855698+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":185,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.409260+0000","last_change":"2024-03-26T03:01:49.448360+0000","last_active":"2024-03-26T03:03:33.409260+0000","last_peered":"2024-03-26T03:03:33.409260+0000","last_clean":"2024-03-26T03:03:33.409260+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:03:33.409260+0000","last_undegraded":"2024-03-26T03:03:33.409260+0000","last_fullsized":"2024-03-26T03:03:33.409260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.448224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.448224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"71'22","reported_seq":213,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398368+0000","last_change":"2024-03-26T03:00:26.545331+0000","last_active":"2024-03-26T03:03:33.398368+0000","last_peered":"2024-03-26T03:03:33.398368+0000","last_clean":"2024-03-26T03:03:33.398368+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:03:33.398368+0000","last_undegraded":"2024-03-26T03:03:33.398368+0000","last_fullsized":"2024-03-26T03:03:33.398368+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'22","last_scrub_stamp":"2024-03-26T03:00:26.545179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:26.545179+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"143'6894","reported_seq":16442,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.564247+0000","last_change":"2024-03-26T03:03:17.309547+0000","last_active":"2024-03-26T03:03:34.564247+0000","last_peered":"2024-03-26T03:03:34.564247+0000","last_clean":"2024-03-26T03:03:34.564247+0000","last_became_active":"2024-03-26T03:03:17.295355+0000","last_became_peered":"2024-03-26T03:03:17.295355+0000","last_unstale":"2024-03-26T03:03:34.564247+0000","last_undegraded":"2024-03-26T03:03:34.564247+0000","last_fullsized":"2024-03-26T03:03:34.564247+0000","mapping_epoch":138,"log_start":"99'692","ondisk_log_start":"99'692","created":111,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:39.388188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108691456,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4579,"num_write_kb":199584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2356,"num_bytes_recovered":154402816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398285+0000","last_change":"2024-03-26T03:01:34.228140+0000","last_active":"2024-03-26T03:03:33.398285+0000","last_peered":"2024-03-26T03:03:33.398285+0000","last_clean":"2024-03-26T03:03:33.398285+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:03:33.398285+0000","last_undegraded":"2024-03-26T03:03:33.398285+0000","last_fullsized":"2024-03-26T03:03:33.398285+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.227979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.227979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408804+0000","last_change":"2024-03-26T03:00:20.999242+0000","last_active":"2024-03-26T03:03:33.408804+0000","last_peered":"2024-03-26T03:03:33.408804+0000","last_clean":"2024-03-26T03:03:33.408804+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:03:33.408804+0000","last_undegraded":"2024-03-26T03:03:33.408804+0000","last_fullsized":"2024-03-26T03:03:33.408804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:20.999149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:20.999149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":186,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396570+0000","last_change":"2024-03-26T03:01:32.933502+0000","last_active":"2024-03-26T03:03:33.396570+0000","last_peered":"2024-03-26T03:03:33.396570+0000","last_clean":"2024-03-26T03:03:33.396570+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:03:33.396570+0000","last_undegraded":"2024-03-26T03:03:33.396570+0000","last_fullsized":"2024-03-26T03:03:33.396570+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.933389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.933389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":219,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.400057+0000","last_change":"2024-03-26T03:01:50.876168+0000","last_active":"2024-03-26T03:03:33.400057+0000","last_peered":"2024-03-26T03:03:33.400057+0000","last_clean":"2024-03-26T03:03:33.400057+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:03:33.400057+0000","last_undegraded":"2024-03-26T03:03:33.400057+0000","last_fullsized":"2024-03-26T03:03:33.400057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"143'6899","reported_seq":11765,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.805789+0000","last_change":"2024-03-26T03:02:22.279568+0000","last_active":"2024-03-26T03:03:33.805789+0000","last_peered":"2024-03-26T03:03:33.805789+0000","last_clean":"2024-03-26T03:03:33.805789+0000","last_became_active":"2024-03-26T03:02:22.277522+0000","last_became_peered":"2024-03-26T03:02:22.277522+0000","last_unstale":"2024-03-26T03:03:33.805789+0000","last_undegraded":"2024-03-26T03:03:33.805789+0000","last_fullsized":"2024-03-26T03:03:33.805789+0000","mapping_epoch":119,"log_start":"99'272","ondisk_log_start":"99'272","created":111,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":6627,"log_dups_size":0,"ondisk_log_size":6627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:37.658745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":5226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":201280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":174,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397671+0000","last_change":"2024-03-26T03:01:52.467228+0000","last_active":"2024-03-26T03:03:33.397671+0000","last_peered":"2024-03-26T03:03:33.397671+0000","last_clean":"2024-03-26T03:03:33.397671+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:03:33.397671+0000","last_undegraded":"2024-03-26T03:03:33.397671+0000","last_fullsized":"2024-03-26T03:03:33.397671+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:52.467134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.467134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.400414+0000","last_change":"2024-03-26T03:01:38.802880+0000","last_active":"2024-03-26T03:03:33.400414+0000","last_peered":"2024-03-26T03:03:33.400414+0000","last_clean":"2024-03-26T03:03:33.400414+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:03:33.400414+0000","last_undegraded":"2024-03-26T03:03:33.400414+0000","last_fullsized":"2024-03-26T03:03:33.400414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:38.802745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:38.802745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":192,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398213+0000","last_change":"2024-03-26T03:01:46.408351+0000","last_active":"2024-03-26T03:03:33.398213+0000","last_peered":"2024-03-26T03:03:33.398213+0000","last_clean":"2024-03-26T03:03:33.398213+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:03:33.398213+0000","last_undegraded":"2024-03-26T03:03:33.398213+0000","last_fullsized":"2024-03-26T03:03:33.398213+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:01:46.408202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:46.408202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":214,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.404223+0000","last_change":"2024-03-26T03:00:40.380209+0000","last_active":"2024-03-26T03:03:33.404223+0000","last_peered":"2024-03-26T03:03:33.404223+0000","last_clean":"2024-03-26T03:03:33.404223+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:03:33.404223+0000","last_undegraded":"2024-03-26T03:03:33.404223+0000","last_fullsized":"2024-03-26T03:03:33.404223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:00:40.380114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.380114+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"143'9233","reported_seq":9341,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.556213+0000","last_change":"2024-03-26T03:00:55.439899+0000","last_active":"2024-03-26T03:03:34.556213+0000","last_peered":"2024-03-26T03:03:34.556213+0000","last_clean":"2024-03-26T03:03:34.556213+0000","last_became_active":"2024-03-26T03:00:55.439557+0000","last_became_peered":"2024-03-26T03:00:55.439557+0000","last_unstale":"2024-03-26T03:03:34.556213+0000","last_undegraded":"2024-03-26T03:03:34.556213+0000","last_fullsized":"2024-03-26T03:03:34.556213+0000","mapping_epoch":96,"log_start":"116'4923","ondisk_log_start":"116'4923","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215941120,"num_objects":3295,"num_object_clones":0,"num_object_copies":9885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9233,"num_write_kb":400896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399604+0000","last_change":"2024-03-26T03:00:35.424045+0000","last_active":"2024-03-26T03:03:33.399604+0000","last_peered":"2024-03-26T03:03:33.399604+0000","last_clean":"2024-03-26T03:03:33.399604+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:03:33.399604+0000","last_undegraded":"2024-03-26T03:03:33.399604+0000","last_fullsized":"2024-03-26T03:03:33.399604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.423904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.423904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":196,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392211+0000","last_change":"2024-03-26T03:01:43.256768+0000","last_active":"2024-03-26T03:03:33.392211+0000","last_peered":"2024-03-26T03:03:33.392211+0000","last_clean":"2024-03-26T03:03:33.392211+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:03:33.392211+0000","last_undegraded":"2024-03-26T03:03:33.392211+0000","last_fullsized":"2024-03-26T03:03:33.392211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:01:43.256605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:43.256605+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":243,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.394094+0000","last_change":"2024-03-26T03:00:25.142453+0000","last_active":"2024-03-26T03:03:33.394094+0000","last_peered":"2024-03-26T03:03:33.394094+0000","last_clean":"2024-03-26T03:03:33.394094+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:03:33.394094+0000","last_undegraded":"2024-03-26T03:03:33.394094+0000","last_fullsized":"2024-03-26T03:03:33.394094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:00:25.142334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:25.142334+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398316+0000","last_change":"2024-03-26T03:01:35.410181+0000","last_active":"2024-03-26T03:03:33.398316+0000","last_peered":"2024-03-26T03:03:33.398316+0000","last_clean":"2024-03-26T03:03:33.398316+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:03:33.398316+0000","last_undegraded":"2024-03-26T03:03:33.398316+0000","last_fullsized":"2024-03-26T03:03:33.398316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:35.410060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:35.410060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"143'6909","reported_seq":7055,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.871268+0000","last_change":"2024-03-26T03:01:51.817423+0000","last_active":"2024-03-26T03:03:33.871268+0000","last_peered":"2024-03-26T03:03:33.871268+0000","last_clean":"2024-03-26T03:03:33.871268+0000","last_became_active":"2024-03-26T03:01:51.288990+0000","last_became_peered":"2024-03-26T03:01:51.288990+0000","last_unstale":"2024-03-26T03:03:33.871268+0000","last_undegraded":"2024-03-26T03:03:33.871268+0000","last_fullsized":"2024-03-26T03:03:33.871268+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4725","last_scrub_stamp":"2024-03-26T03:01:51.817210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:51.817210+0000","objects_scrubbed":2411,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112066560,"num_objects":1710,"num_object_clones":0,"num_object_copies":5130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":47616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.404012+0000","last_change":"2024-03-26T03:00:11.023432+0000","last_active":"2024-03-26T03:03:33.404012+0000","last_peered":"2024-03-26T03:03:33.404012+0000","last_clean":"2024-03-26T03:03:33.404012+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:03:33.404012+0000","last_undegraded":"2024-03-26T03:03:33.404012+0000","last_fullsized":"2024-03-26T03:03:33.404012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:11.023351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:11.023351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403186+0000","last_change":"2024-03-26T03:00:27.327214+0000","last_active":"2024-03-26T03:03:33.403186+0000","last_peered":"2024-03-26T03:03:33.403186+0000","last_clean":"2024-03-26T03:03:33.403186+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:03:33.403186+0000","last_undegraded":"2024-03-26T03:03:33.403186+0000","last_fullsized":"2024-03-26T03:03:33.403186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.327116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.327116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"92'22","reported_seq":208,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402776+0000","last_change":"2024-03-26T03:00:42.399658+0000","last_active":"2024-03-26T03:03:33.402776+0000","last_peered":"2024-03-26T03:03:33.402776+0000","last_clean":"2024-03-26T03:03:33.402776+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:03:33.402776+0000","last_undegraded":"2024-03-26T03:03:33.402776+0000","last_fullsized":"2024-03-26T03:03:33.402776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:00:42.399566+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":246,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408442+0000","last_change":"2024-03-26T03:01:40.859597+0000","last_active":"2024-03-26T03:03:33.408442+0000","last_peered":"2024-03-26T03:03:33.408442+0000","last_clean":"2024-03-26T03:03:33.408442+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:03:33.408442+0000","last_undegraded":"2024-03-26T03:03:33.408442+0000","last_fullsized":"2024-03-26T03:03:33.408442+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:01:40.859466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.859466+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"143'7041","reported_seq":7191,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.853330+0000","last_change":"2024-03-26T03:01:52.740255+0000","last_active":"2024-03-26T03:03:33.853330+0000","last_peered":"2024-03-26T03:03:33.853330+0000","last_clean":"2024-03-26T03:03:33.853330+0000","last_became_active":"2024-03-26T03:01:51.303857+0000","last_became_peered":"2024-03-26T03:01:51.303857+0000","last_unstale":"2024-03-26T03:03:33.853330+0000","last_undegraded":"2024-03-26T03:03:33.853330+0000","last_fullsized":"2024-03-26T03:03:33.853330+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4820","last_scrub_stamp":"2024-03-26T03:01:52.740121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.740121+0000","objects_scrubbed":2415,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108199936,"num_objects":1651,"num_object_clones":0,"num_object_copies":4953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":46656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396772+0000","last_change":"2024-03-26T03:01:33.720224+0000","last_active":"2024-03-26T03:03:33.396772+0000","last_peered":"2024-03-26T03:03:33.396772+0000","last_clean":"2024-03-26T03:03:33.396772+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:03:33.396772+0000","last_undegraded":"2024-03-26T03:03:33.396772+0000","last_fullsized":"2024-03-26T03:03:33.396772+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.719795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.719795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":178,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.409264+0000","last_change":"2024-03-26T03:00:32.016983+0000","last_active":"2024-03-26T03:03:33.409264+0000","last_peered":"2024-03-26T03:03:33.409264+0000","last_clean":"2024-03-26T03:03:33.409264+0000","last_became_active":"2024-03-26T02:58:56.923317+0000","last_became_peered":"2024-03-26T02:58:56.923317+0000","last_unstale":"2024-03-26T03:03:33.409264+0000","last_undegraded":"2024-03-26T03:03:33.409264+0000","last_fullsized":"2024-03-26T03:03:33.409264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:32.016849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.016849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":302,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402946+0000","last_change":"2024-03-26T03:00:13.375248+0000","last_active":"2024-03-26T03:03:33.402946+0000","last_peered":"2024-03-26T03:03:33.402946+0000","last_clean":"2024-03-26T03:03:33.402946+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:03:33.402946+0000","last_undegraded":"2024-03-26T03:03:33.402946+0000","last_fullsized":"2024-03-26T03:03:33.402946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:00:13.375080+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.404137+0000","last_change":"2024-03-26T03:00:29.282199+0000","last_active":"2024-03-26T03:03:33.404137+0000","last_peered":"2024-03-26T03:03:33.404137+0000","last_clean":"2024-03-26T03:03:33.404137+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:03:33.404137+0000","last_undegraded":"2024-03-26T03:03:33.404137+0000","last_fullsized":"2024-03-26T03:03:33.404137+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:29.282115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:29.282115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"143'6802","reported_seq":6950,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.923339+0000","last_change":"2024-03-26T03:01:53.378883+0000","last_active":"2024-03-26T03:03:34.923339+0000","last_peered":"2024-03-26T03:03:34.923339+0000","last_clean":"2024-03-26T03:03:34.923339+0000","last_became_active":"2024-03-26T03:01:51.301399+0000","last_became_peered":"2024-03-26T03:01:51.301399+0000","last_unstale":"2024-03-26T03:03:34.923339+0000","last_undegraded":"2024-03-26T03:03:34.923339+0000","last_fullsized":"2024-03-26T03:03:34.923339+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4665","last_scrub_stamp":"2024-03-26T03:01:53.378673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.378673+0000","objects_scrubbed":2342,"log_size":6802,"log_dups_size":0,"ondisk_log_size":6802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2137,"num_write_kb":45568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398537+0000","last_change":"2024-03-26T03:00:41.268246+0000","last_active":"2024-03-26T03:03:33.398537+0000","last_peered":"2024-03-26T03:03:33.398537+0000","last_clean":"2024-03-26T03:03:33.398537+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:03:33.398537+0000","last_undegraded":"2024-03-26T03:03:33.398537+0000","last_fullsized":"2024-03-26T03:03:33.398537+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.268157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":457,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403139+0000","last_change":"2024-03-26T03:00:04.289538+0000","last_active":"2024-03-26T03:03:33.403139+0000","last_peered":"2024-03-26T03:03:33.403139+0000","last_clean":"2024-03-26T03:03:33.403139+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:03:33.403139+0000","last_undegraded":"2024-03-26T03:03:33.403139+0000","last_fullsized":"2024-03-26T03:03:33.403139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:00:04.289452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:04.289452+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":268,"num_read_kb":263,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398661+0000","last_change":"2024-03-26T03:01:45.357919+0000","last_active":"2024-03-26T03:03:33.398661+0000","last_peered":"2024-03-26T03:03:33.398661+0000","last_clean":"2024-03-26T03:03:33.398661+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:03:33.398661+0000","last_undegraded":"2024-03-26T03:03:33.398661+0000","last_fullsized":"2024-03-26T03:03:33.398661+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:45.357787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:45.357787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":214,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402213+0000","last_change":"2024-03-26T03:00:23.305418+0000","last_active":"2024-03-26T03:03:33.402213+0000","last_peered":"2024-03-26T03:03:33.402213+0000","last_clean":"2024-03-26T03:03:33.402213+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:03:33.402213+0000","last_undegraded":"2024-03-26T03:03:33.402213+0000","last_fullsized":"2024-03-26T03:03:33.402213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:00:23.305307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.305307+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.401111+0000","last_change":"2024-03-26T03:01:44.370403+0000","last_active":"2024-03-26T03:03:33.401111+0000","last_peered":"2024-03-26T03:03:33.401111+0000","last_clean":"2024-03-26T03:03:33.401111+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:03:33.401111+0000","last_undegraded":"2024-03-26T03:03:33.401111+0000","last_fullsized":"2024-03-26T03:03:33.401111+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:44.370278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:44.370278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"143'6969","reported_seq":7113,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.819743+0000","last_change":"2024-03-26T03:01:54.158951+0000","last_active":"2024-03-26T03:03:34.819743+0000","last_peered":"2024-03-26T03:03:34.819743+0000","last_clean":"2024-03-26T03:03:34.819743+0000","last_became_active":"2024-03-26T03:01:51.328380+0000","last_became_peered":"2024-03-26T03:01:51.328380+0000","last_unstale":"2024-03-26T03:03:34.819743+0000","last_undegraded":"2024-03-26T03:03:34.819743+0000","last_fullsized":"2024-03-26T03:03:34.819743+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4780","last_scrub_stamp":"2024-03-26T03:01:54.158811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.158811+0000","objects_scrubbed":2374,"log_size":6969,"log_dups_size":0,"ondisk_log_size":6969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2189,"num_write_kb":45440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":174,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408666+0000","last_change":"2024-03-26T03:01:47.363671+0000","last_active":"2024-03-26T03:03:33.408666+0000","last_peered":"2024-03-26T03:03:33.408666+0000","last_clean":"2024-03-26T03:03:33.408666+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:03:33.408666+0000","last_undegraded":"2024-03-26T03:03:33.408666+0000","last_fullsized":"2024-03-26T03:03:33.408666+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:47.363550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.363550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":188,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397406+0000","last_change":"2024-03-26T03:00:45.454030+0000","last_active":"2024-03-26T03:03:33.397406+0000","last_peered":"2024-03-26T03:03:33.397406+0000","last_clean":"2024-03-26T03:03:33.397406+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:03:33.397406+0000","last_undegraded":"2024-03-26T03:03:33.397406+0000","last_fullsized":"2024-03-26T03:03:33.397406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:00:45.453925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.453925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":192,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408175+0000","last_change":"2024-03-26T03:01:32.148305+0000","last_active":"2024-03-26T03:03:33.408175+0000","last_peered":"2024-03-26T03:03:33.408175+0000","last_clean":"2024-03-26T03:03:33.408175+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:03:33.408175+0000","last_undegraded":"2024-03-26T03:03:33.408175+0000","last_fullsized":"2024-03-26T03:03:33.408175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.148136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.148136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":426,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399388+0000","last_change":"2024-03-26T03:01:38.142321+0000","last_active":"2024-03-26T03:03:33.399388+0000","last_peered":"2024-03-26T03:03:33.399388+0000","last_clean":"2024-03-26T03:03:33.399388+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:03:33.399388+0000","last_undegraded":"2024-03-26T03:03:33.399388+0000","last_fullsized":"2024-03-26T03:03:33.399388+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:01:38.142194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:01:38.142194+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":261,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397024+0000","last_change":"2024-03-26T03:01:42.020043+0000","last_active":"2024-03-26T03:03:33.397024+0000","last_peered":"2024-03-26T03:03:33.397024+0000","last_clean":"2024-03-26T03:03:33.397024+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:03:33.397024+0000","last_undegraded":"2024-03-26T03:03:33.397024+0000","last_fullsized":"2024-03-26T03:03:33.397024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:01:42.019820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.019820+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398335+0000","last_change":"2024-03-26T03:01:48.249675+0000","last_active":"2024-03-26T03:03:33.398335+0000","last_peered":"2024-03-26T03:03:33.398335+0000","last_clean":"2024-03-26T03:03:33.398335+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:03:33.398335+0000","last_undegraded":"2024-03-26T03:03:33.398335+0000","last_fullsized":"2024-03-26T03:03:33.398335+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:48.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:48.249567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"143'6888","reported_seq":7032,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.889640+0000","last_change":"2024-03-26T03:01:52.635320+0000","last_active":"2024-03-26T03:03:33.889640+0000","last_peered":"2024-03-26T03:03:33.889640+0000","last_clean":"2024-03-26T03:03:33.889640+0000","last_became_active":"2024-03-26T03:01:51.316580+0000","last_became_peered":"2024-03-26T03:01:51.316580+0000","last_unstale":"2024-03-26T03:03:33.889640+0000","last_undegraded":"2024-03-26T03:03:33.889640+0000","last_fullsized":"2024-03-26T03:03:33.889640+0000","mapping_epoch":96,"log_start":"99'253","ondisk_log_start":"99'253","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4738","last_scrub_stamp":"2024-03-26T03:01:52.635130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.635130+0000","objects_scrubbed":2393,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105709568,"num_objects":1613,"num_object_clones":0,"num_object_copies":4839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2150,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403481+0000","last_change":"2024-03-26T03:00:35.391826+0000","last_active":"2024-03-26T03:03:33.403481+0000","last_peered":"2024-03-26T03:03:33.403481+0000","last_clean":"2024-03-26T03:03:33.403481+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:03:33.403481+0000","last_undegraded":"2024-03-26T03:03:33.403481+0000","last_fullsized":"2024-03-26T03:03:33.403481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.391746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.391746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.410669+0000","last_change":"2024-03-26T03:00:32.315656+0000","last_active":"2024-03-26T03:03:33.410669+0000","last_peered":"2024-03-26T03:03:33.410669+0000","last_clean":"2024-03-26T03:03:33.410669+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:03:33.410669+0000","last_undegraded":"2024-03-26T03:03:33.410669+0000","last_fullsized":"2024-03-26T03:03:33.410669+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.315559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399358+0000","last_change":"2024-03-26T03:00:47.234857+0000","last_active":"2024-03-26T03:03:33.399358+0000","last_peered":"2024-03-26T03:03:33.399358+0000","last_clean":"2024-03-26T03:03:33.399358+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:03:33.399358+0000","last_undegraded":"2024-03-26T03:03:33.399358+0000","last_fullsized":"2024-03-26T03:03:33.399358+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:47.234781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:47.234781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":201,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403998+0000","last_change":"2024-03-26T03:01:33.900763+0000","last_active":"2024-03-26T03:03:33.403998+0000","last_peered":"2024-03-26T03:03:33.403998+0000","last_clean":"2024-03-26T03:03:33.403998+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:03:33.403998+0000","last_undegraded":"2024-03-26T03:03:33.403998+0000","last_fullsized":"2024-03-26T03:03:33.403998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:33.900630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.900630+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":170,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408616+0000","last_change":"2024-03-26T03:00:35.969971+0000","last_active":"2024-03-26T03:03:33.408616+0000","last_peered":"2024-03-26T03:03:33.408616+0000","last_clean":"2024-03-26T03:03:33.408616+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:03:33.408616+0000","last_undegraded":"2024-03-26T03:03:33.408616+0000","last_fullsized":"2024-03-26T03:03:33.408616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.969837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.969837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"143'7010","reported_seq":7157,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.870903+0000","last_change":"2024-03-26T03:01:51.998675+0000","last_active":"2024-03-26T03:03:33.870903+0000","last_peered":"2024-03-26T03:03:33.870903+0000","last_clean":"2024-03-26T03:03:33.870903+0000","last_became_active":"2024-03-26T03:01:51.311443+0000","last_became_peered":"2024-03-26T03:01:51.311443+0000","last_unstale":"2024-03-26T03:03:33.870903+0000","last_undegraded":"2024-03-26T03:03:33.870903+0000","last_fullsized":"2024-03-26T03:03:33.870903+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4793","last_scrub_stamp":"2024-03-26T03:01:51.998539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:51.998539+0000","objects_scrubbed":2401,"log_size":7010,"log_dups_size":0,"ondisk_log_size":7010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2230,"num_write_kb":47488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402993+0000","last_change":"2024-03-26T03:00:07.343102+0000","last_active":"2024-03-26T03:03:33.402993+0000","last_peered":"2024-03-26T03:03:33.402993+0000","last_clean":"2024-03-26T03:03:33.402993+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:03:33.402993+0000","last_undegraded":"2024-03-26T03:03:33.402993+0000","last_fullsized":"2024-03-26T03:03:33.402993+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:07.343020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.343020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398490+0000","last_change":"2024-03-26T03:00:49.223170+0000","last_active":"2024-03-26T03:03:33.398490+0000","last_peered":"2024-03-26T03:03:33.398490+0000","last_clean":"2024-03-26T03:03:33.398490+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:03:33.398490+0000","last_undegraded":"2024-03-26T03:03:33.398490+0000","last_fullsized":"2024-03-26T03:03:33.398490+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:49.223092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:49.223092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":231,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408118+0000","last_change":"2024-03-26T03:01:47.898281+0000","last_active":"2024-03-26T03:03:33.408118+0000","last_peered":"2024-03-26T03:03:33.408118+0000","last_clean":"2024-03-26T03:03:33.408118+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:03:33.408118+0000","last_undegraded":"2024-03-26T03:03:33.408118+0000","last_fullsized":"2024-03-26T03:03:33.408118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:01:47.898135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.898135+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396570+0000","last_change":"2024-03-26T03:00:10.472741+0000","last_active":"2024-03-26T03:03:33.396570+0000","last_peered":"2024-03-26T03:03:33.396570+0000","last_clean":"2024-03-26T03:03:33.396570+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:03:33.396570+0000","last_undegraded":"2024-03-26T03:03:33.396570+0000","last_fullsized":"2024-03-26T03:03:33.396570+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:10.472642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:10.472642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"143'6925","reported_seq":7075,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.548031+0000","last_change":"2024-03-26T03:01:53.411487+0000","last_active":"2024-03-26T03:03:34.548031+0000","last_peered":"2024-03-26T03:03:34.548031+0000","last_clean":"2024-03-26T03:03:34.548031+0000","last_became_active":"2024-03-26T03:01:51.345923+0000","last_became_peered":"2024-03-26T03:01:51.345923+0000","last_unstale":"2024-03-26T03:03:34.548031+0000","last_undegraded":"2024-03-26T03:03:34.548031+0000","last_fullsized":"2024-03-26T03:03:34.548031+0000","mapping_epoch":96,"log_start":"99'773","ondisk_log_start":"99'773","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4741","last_scrub_stamp":"2024-03-26T03:01:53.411373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.411373+0000","objects_scrubbed":2420,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":45120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391561+0000","last_change":"2024-03-26T03:03:29.432831+0000","last_active":"2024-03-26T03:03:33.391561+0000","last_peered":"2024-03-26T03:03:33.391561+0000","last_clean":"2024-03-26T03:03:33.391561+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:03:33.391561+0000","last_undegraded":"2024-03-26T03:03:33.391561+0000","last_fullsized":"2024-03-26T03:03:33.391561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:29.432698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:29.432698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:38.308985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":181,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402417+0000","last_change":"2024-03-26T03:00:17.286899+0000","last_active":"2024-03-26T03:03:33.402417+0000","last_peered":"2024-03-26T03:03:33.402417+0000","last_clean":"2024-03-26T03:03:33.402417+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:03:33.402417+0000","last_undegraded":"2024-03-26T03:03:33.402417+0000","last_fullsized":"2024-03-26T03:03:33.402417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.286796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":251,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402678+0000","last_change":"2024-03-26T03:01:36.769019+0000","last_active":"2024-03-26T03:03:33.402678+0000","last_peered":"2024-03-26T03:03:33.402678+0000","last_clean":"2024-03-26T03:03:33.402678+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:03:33.402678+0000","last_undegraded":"2024-03-26T03:03:33.402678+0000","last_fullsized":"2024-03-26T03:03:33.402678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":170,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.394176+0000","last_change":"2024-03-26T03:01:41.237613+0000","last_active":"2024-03-26T03:03:33.394176+0000","last_peered":"2024-03-26T03:03:33.394176+0000","last_clean":"2024-03-26T03:03:33.394176+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:03:33.394176+0000","last_undegraded":"2024-03-26T03:03:33.394176+0000","last_fullsized":"2024-03-26T03:03:33.394176+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:41.236598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:41.236598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"143'6974","reported_seq":7118,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.570575+0000","last_change":"2024-03-26T03:01:52.734222+0000","last_active":"2024-03-26T03:03:34.570575+0000","last_peered":"2024-03-26T03:03:34.570575+0000","last_clean":"2024-03-26T03:03:34.570575+0000","last_became_active":"2024-03-26T03:01:51.309762+0000","last_became_peered":"2024-03-26T03:01:51.309762+0000","last_unstale":"2024-03-26T03:03:34.570575+0000","last_undegraded":"2024-03-26T03:03:34.570575+0000","last_fullsized":"2024-03-26T03:03:34.570575+0000","mapping_epoch":96,"log_start":"99'634","ondisk_log_start":"99'634","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4791","last_scrub_stamp":"2024-03-26T03:01:52.734068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.734068+0000","objects_scrubbed":2345,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2186,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398059+0000","last_change":"2024-03-26T03:00:48.212377+0000","last_active":"2024-03-26T03:03:33.398059+0000","last_peered":"2024-03-26T03:03:33.398059+0000","last_clean":"2024-03-26T03:03:33.398059+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:03:33.398059+0000","last_undegraded":"2024-03-26T03:03:33.398059+0000","last_fullsized":"2024-03-26T03:03:33.398059+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:48.212296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:48.212296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397524+0000","last_change":"2024-03-26T03:01:40.975997+0000","last_active":"2024-03-26T03:03:33.397524+0000","last_peered":"2024-03-26T03:03:33.397524+0000","last_clean":"2024-03-26T03:03:33.397524+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:03:33.397524+0000","last_undegraded":"2024-03-26T03:03:33.397524+0000","last_fullsized":"2024-03-26T03:03:33.397524+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:40.975886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.975886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":207,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.393840+0000","last_change":"2024-03-26T03:00:07.927770+0000","last_active":"2024-03-26T03:03:33.393840+0000","last_peered":"2024-03-26T03:03:33.393840+0000","last_clean":"2024-03-26T03:03:33.393840+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:03:33.393840+0000","last_undegraded":"2024-03-26T03:03:33.393840+0000","last_fullsized":"2024-03-26T03:03:33.393840+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:00:07.927648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:07.927648+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"91'9","reported_seq":236,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402781+0000","last_change":"2024-03-26T03:00:37.456923+0000","last_active":"2024-03-26T03:03:33.402781+0000","last_peered":"2024-03-26T03:03:33.402781+0000","last_clean":"2024-03-26T03:03:33.402781+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:03:33.402781+0000","last_undegraded":"2024-03-26T03:03:33.402781+0000","last_fullsized":"2024-03-26T03:03:33.402781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:00:37.456826+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"143'9328","reported_seq":9437,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.904150+0000","last_change":"2024-03-26T03:00:55.434339+0000","last_active":"2024-03-26T03:03:33.904150+0000","last_peered":"2024-03-26T03:03:33.904150+0000","last_clean":"2024-03-26T03:03:33.904150+0000","last_became_active":"2024-03-26T03:00:55.433417+0000","last_became_peered":"2024-03-26T03:00:55.433417+0000","last_unstale":"2024-03-26T03:03:33.904150+0000","last_undegraded":"2024-03-26T03:03:33.904150+0000","last_fullsized":"2024-03-26T03:03:33.904150+0000","mapping_epoch":96,"log_start":"104'3700","ondisk_log_start":"104'3700","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902616,"num_objects":3448,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9328,"num_write_kb":408769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397366+0000","last_change":"2024-03-26T03:00:15.672174+0000","last_active":"2024-03-26T03:03:33.397366+0000","last_peered":"2024-03-26T03:03:33.397366+0000","last_clean":"2024-03-26T03:03:33.397366+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:03:33.397366+0000","last_undegraded":"2024-03-26T03:03:33.397366+0000","last_fullsized":"2024-03-26T03:03:33.397366+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.672090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":169,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408581+0000","last_change":"2024-03-26T03:00:27.990812+0000","last_active":"2024-03-26T03:03:33.408581+0000","last_peered":"2024-03-26T03:03:33.408581+0000","last_clean":"2024-03-26T03:03:33.408581+0000","last_became_active":"2024-03-26T02:59:00.937450+0000","last_became_peered":"2024-03-26T02:59:00.937450+0000","last_unstale":"2024-03-26T03:03:33.408581+0000","last_undegraded":"2024-03-26T03:03:33.408581+0000","last_fullsized":"2024-03-26T03:03:33.408581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.990730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.990730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399435+0000","last_change":"2024-03-26T03:01:47.284940+0000","last_active":"2024-03-26T03:03:33.399435+0000","last_peered":"2024-03-26T03:03:33.399435+0000","last_clean":"2024-03-26T03:03:33.399435+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:03:33.399435+0000","last_undegraded":"2024-03-26T03:03:33.399435+0000","last_fullsized":"2024-03-26T03:03:33.399435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:47.284840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.284840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":205,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391747+0000","last_change":"2024-03-26T03:00:05.879783+0000","last_active":"2024-03-26T03:03:33.391747+0000","last_peered":"2024-03-26T03:03:33.391747+0000","last_clean":"2024-03-26T03:03:33.391747+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:03:33.391747+0000","last_undegraded":"2024-03-26T03:03:33.391747+0000","last_fullsized":"2024-03-26T03:03:33.391747+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:05.879672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:05.879672+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"143'9084","reported_seq":9192,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.870566+0000","last_change":"2024-03-26T03:00:55.436147+0000","last_active":"2024-03-26T03:03:33.870566+0000","last_peered":"2024-03-26T03:03:33.870566+0000","last_clean":"2024-03-26T03:03:33.870566+0000","last_became_active":"2024-03-26T03:00:55.435784+0000","last_became_peered":"2024-03-26T03:00:55.435784+0000","last_unstale":"2024-03-26T03:03:33.870566+0000","last_undegraded":"2024-03-26T03:03:33.870566+0000","last_fullsized":"2024-03-26T03:03:33.870566+0000","mapping_epoch":96,"log_start":"104'3500","ondisk_log_start":"104'3500","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218628096,"num_objects":3336,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":397440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408094+0000","last_change":"2024-03-26T03:00:40.962821+0000","last_active":"2024-03-26T03:03:33.408094+0000","last_peered":"2024-03-26T03:03:33.408094+0000","last_clean":"2024-03-26T03:03:33.408094+0000","last_became_active":"2024-03-26T02:59:02.944279+0000","last_became_peered":"2024-03-26T02:59:02.944279+0000","last_unstale":"2024-03-26T03:03:33.408094+0000","last_undegraded":"2024-03-26T03:03:33.408094+0000","last_fullsized":"2024-03-26T03:03:33.408094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.962737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":234,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391990+0000","last_change":"2024-03-26T03:00:30.182479+0000","last_active":"2024-03-26T03:03:33.391990+0000","last_peered":"2024-03-26T03:03:33.391990+0000","last_clean":"2024-03-26T03:03:33.391990+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:03:33.391990+0000","last_undegraded":"2024-03-26T03:03:33.391990+0000","last_fullsized":"2024-03-26T03:03:33.391990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:30.182384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:30.182384+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402677+0000","last_change":"2024-03-26T03:00:52.439831+0000","last_active":"2024-03-26T03:03:33.402677+0000","last_peered":"2024-03-26T03:03:33.402677+0000","last_clean":"2024-03-26T03:03:33.402677+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:03:33.402677+0000","last_undegraded":"2024-03-26T03:03:33.402677+0000","last_fullsized":"2024-03-26T03:03:33.402677+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.439710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.439710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":200,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402372+0000","last_change":"2024-03-26T03:00:28.371034+0000","last_active":"2024-03-26T03:03:33.402372+0000","last_peered":"2024-03-26T03:03:33.402372+0000","last_clean":"2024-03-26T03:03:33.402372+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:03:33.402372+0000","last_undegraded":"2024-03-26T03:03:33.402372+0000","last_fullsized":"2024-03-26T03:03:33.402372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:28.370925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:28.370925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"143'9138","reported_seq":9245,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.950631+0000","last_change":"2024-03-26T03:00:55.439219+0000","last_active":"2024-03-26T03:03:34.950631+0000","last_peered":"2024-03-26T03:03:34.950631+0000","last_clean":"2024-03-26T03:03:34.950631+0000","last_became_active":"2024-03-26T03:00:55.438925+0000","last_became_peered":"2024-03-26T03:00:55.438925+0000","last_unstale":"2024-03-26T03:03:34.950631+0000","last_undegraded":"2024-03-26T03:03:34.950631+0000","last_fullsized":"2024-03-26T03:03:34.950631+0000","mapping_epoch":96,"log_start":"107'4300","ondisk_log_start":"107'4300","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":165,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392039+0000","last_change":"2024-03-26T03:03:32.396450+0000","last_active":"2024-03-26T03:03:33.392039+0000","last_peered":"2024-03-26T03:03:33.392039+0000","last_clean":"2024-03-26T03:03:33.392039+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:03:33.392039+0000","last_undegraded":"2024-03-26T03:03:33.392039+0000","last_fullsized":"2024-03-26T03:03:33.392039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:32.396317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:32.396317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:57.161788+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":244,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398379+0000","last_change":"2024-03-26T03:01:50.516513+0000","last_active":"2024-03-26T03:03:33.398379+0000","last_peered":"2024-03-26T03:03:33.398379+0000","last_clean":"2024-03-26T03:03:33.398379+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:03:33.398379+0000","last_undegraded":"2024-03-26T03:03:33.398379+0000","last_fullsized":"2024-03-26T03:03:33.398379+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:01:50.516336+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.516336+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402438+0000","last_change":"2024-03-26T03:01:50.576964+0000","last_active":"2024-03-26T03:03:33.402438+0000","last_peered":"2024-03-26T03:03:33.402438+0000","last_clean":"2024-03-26T03:03:33.402438+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:03:33.402438+0000","last_undegraded":"2024-03-26T03:03:33.402438+0000","last_fullsized":"2024-03-26T03:03:33.402438+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:50.576769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.576769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":189,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.392663+0000","last_change":"2024-03-26T03:00:08.975032+0000","last_active":"2024-03-26T03:03:33.392663+0000","last_peered":"2024-03-26T03:03:33.392663+0000","last_clean":"2024-03-26T03:03:33.392663+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:03:33.392663+0000","last_undegraded":"2024-03-26T03:03:33.392663+0000","last_fullsized":"2024-03-26T03:03:33.392663+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:00:08.974874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:08.974874+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"143'9206","reported_seq":9314,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.259442+0000","last_change":"2024-03-26T03:00:55.433696+0000","last_active":"2024-03-26T03:03:34.259442+0000","last_peered":"2024-03-26T03:03:34.259442+0000","last_clean":"2024-03-26T03:03:34.259442+0000","last_became_active":"2024-03-26T03:00:55.433381+0000","last_became_peered":"2024-03-26T03:00:55.433381+0000","last_unstale":"2024-03-26T03:03:34.259442+0000","last_undegraded":"2024-03-26T03:03:34.259442+0000","last_fullsized":"2024-03-26T03:03:34.259442+0000","mapping_epoch":96,"log_start":"104'3500","ondisk_log_start":"104'3500","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227016704,"num_objects":3464,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9206,"num_write_kb":405440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.401900+0000","last_change":"2024-03-26T03:00:37.323420+0000","last_active":"2024-03-26T03:03:33.401900+0000","last_peered":"2024-03-26T03:03:33.401900+0000","last_clean":"2024-03-26T03:03:33.401900+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:03:33.401900+0000","last_undegraded":"2024-03-26T03:03:33.401900+0000","last_fullsized":"2024-03-26T03:03:33.401900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:37.323337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:37.323337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.403437+0000","last_change":"2024-03-26T03:00:45.397106+0000","last_active":"2024-03-26T03:03:33.403437+0000","last_peered":"2024-03-26T03:03:33.403437+0000","last_clean":"2024-03-26T03:03:33.403437+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:03:33.403437+0000","last_undegraded":"2024-03-26T03:03:33.403437+0000","last_fullsized":"2024-03-26T03:03:33.403437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.397023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397752+0000","last_change":"2024-03-26T03:01:33.183481+0000","last_active":"2024-03-26T03:03:33.397752+0000","last_peered":"2024-03-26T03:03:33.397752+0000","last_clean":"2024-03-26T03:03:33.397752+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:03:33.397752+0000","last_undegraded":"2024-03-26T03:03:33.397752+0000","last_fullsized":"2024-03-26T03:03:33.397752+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.183362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.183362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"71'17","reported_seq":202,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.391887+0000","last_change":"2024-03-26T03:00:52.286513+0000","last_active":"2024-03-26T03:03:33.391887+0000","last_peered":"2024-03-26T03:03:33.391887+0000","last_clean":"2024-03-26T03:03:33.391887+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:03:33.391887+0000","last_undegraded":"2024-03-26T03:03:33.391887+0000","last_fullsized":"2024-03-26T03:03:33.391887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:00:52.286392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.286392+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"143'6852","reported_seq":6997,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.856681+0000","last_change":"2024-03-26T03:01:52.545451+0000","last_active":"2024-03-26T03:03:34.856681+0000","last_peered":"2024-03-26T03:03:34.856681+0000","last_clean":"2024-03-26T03:03:34.856681+0000","last_became_active":"2024-03-26T03:01:51.307163+0000","last_became_peered":"2024-03-26T03:01:51.307163+0000","last_unstale":"2024-03-26T03:03:34.856681+0000","last_undegraded":"2024-03-26T03:03:34.856681+0000","last_fullsized":"2024-03-26T03:03:34.856681+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4657","last_scrub_stamp":"2024-03-26T03:01:52.545222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.545222+0000","objects_scrubbed":2326,"log_size":6852,"log_dups_size":0,"ondisk_log_size":6852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":616,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110755840,"num_objects":1690,"num_object_clones":0,"num_object_copies":5070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2198,"num_write_kb":49984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.399254+0000","last_change":"2024-03-26T03:01:37.160971+0000","last_active":"2024-03-26T03:03:33.399254+0000","last_peered":"2024-03-26T03:03:33.399254+0000","last_clean":"2024-03-26T03:03:33.399254+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:03:33.399254+0000","last_undegraded":"2024-03-26T03:03:33.399254+0000","last_fullsized":"2024-03-26T03:03:33.399254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:37.160888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.160888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398087+0000","last_change":"2024-03-26T03:01:43.387197+0000","last_active":"2024-03-26T03:03:33.398087+0000","last_peered":"2024-03-26T03:03:33.398087+0000","last_clean":"2024-03-26T03:03:33.398087+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:03:33.398087+0000","last_undegraded":"2024-03-26T03:03:33.398087+0000","last_fullsized":"2024-03-26T03:03:33.398087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:43.386881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:01:43.386881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.397868+0000","last_change":"2024-03-26T03:01:40.361299+0000","last_active":"2024-03-26T03:03:33.397868+0000","last_peered":"2024-03-26T03:03:33.397868+0000","last_clean":"2024-03-26T03:03:33.397868+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:03:33.397868+0000","last_undegraded":"2024-03-26T03:03:33.397868+0000","last_fullsized":"2024-03-26T03:03:33.397868+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:40.360702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.360702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":254,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.396476+0000","last_change":"2024-03-26T03:01:54.088999+0000","last_active":"2024-03-26T03:03:33.396476+0000","last_peered":"2024-03-26T03:03:33.396476+0000","last_clean":"2024-03-26T03:03:33.396476+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:03:33.396476+0000","last_undegraded":"2024-03-26T03:03:33.396476+0000","last_fullsized":"2024-03-26T03:03:33.396476+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:01:54.088880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.088880+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"143'7057","reported_seq":7204,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.188744+0000","last_change":"2024-03-26T03:01:52.302237+0000","last_active":"2024-03-26T03:03:34.188744+0000","last_peered":"2024-03-26T03:03:34.188744+0000","last_clean":"2024-03-26T03:03:34.188744+0000","last_became_active":"2024-03-26T03:01:51.304269+0000","last_became_peered":"2024-03-26T03:01:51.304269+0000","last_unstale":"2024-03-26T03:03:34.188744+0000","last_undegraded":"2024-03-26T03:03:34.188744+0000","last_fullsized":"2024-03-26T03:03:34.188744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4786","last_scrub_stamp":"2024-03-26T03:01:52.302079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.302079+0000","objects_scrubbed":2466,"log_size":7057,"log_dups_size":0,"ondisk_log_size":7057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2287,"num_write_kb":48768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":167,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.429714+0000","last_change":"2024-03-26T03:00:48.304407+0000","last_active":"2024-03-26T03:03:33.429714+0000","last_peered":"2024-03-26T03:03:33.429714+0000","last_clean":"2024-03-26T03:03:33.429714+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:03:33.429714+0000","last_undegraded":"2024-03-26T03:03:33.429714+0000","last_fullsized":"2024-03-26T03:03:33.429714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:00:48.304319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":171,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.401324+0000","last_change":"2024-03-26T03:00:40.307781+0000","last_active":"2024-03-26T03:03:33.401324+0000","last_peered":"2024-03-26T03:03:33.401324+0000","last_clean":"2024-03-26T03:03:33.401324+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:03:33.401324+0000","last_undegraded":"2024-03-26T03:03:33.401324+0000","last_fullsized":"2024-03-26T03:03:33.401324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:40.307697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.307697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":179,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.402607+0000","last_change":"2024-03-26T02:59:57.244851+0000","last_active":"2024-03-26T03:03:33.402607+0000","last_peered":"2024-03-26T03:03:33.402607+0000","last_clean":"2024-03-26T03:03:33.402607+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:03:33.402607+0000","last_undegraded":"2024-03-26T03:03:33.402607+0000","last_fullsized":"2024-03-26T03:03:33.402607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:57.244756+0000","last_clean_scrub_stamp":"2024-03-26T02:59:57.244756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":208,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.408300+0000","last_change":"2024-03-26T03:00:27.238978+0000","last_active":"2024-03-26T03:03:33.408300+0000","last_peered":"2024-03-26T03:03:33.408300+0000","last_clean":"2024-03-26T03:03:33.408300+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:03:33.408300+0000","last_undegraded":"2024-03-26T03:03:33.408300+0000","last_fullsized":"2024-03-26T03:03:33.408300+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:00:27.238859+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.238859+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"143'8976","reported_seq":9084,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:34.072024+0000","last_change":"2024-03-26T03:00:55.435500+0000","last_active":"2024-03-26T03:03:34.072024+0000","last_peered":"2024-03-26T03:03:34.072024+0000","last_clean":"2024-03-26T03:03:34.072024+0000","last_became_active":"2024-03-26T03:00:55.435115+0000","last_became_peered":"2024-03-26T03:00:55.435115+0000","last_unstale":"2024-03-26T03:03:34.072024+0000","last_undegraded":"2024-03-26T03:03:34.072024+0000","last_fullsized":"2024-03-26T03:03:34.072024+0000","mapping_epoch":96,"log_start":"106'4100","ondisk_log_start":"106'4100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225312768,"num_objects":3438,"num_object_clones":0,"num_object_copies":10314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8976,"num_write_kb":397248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.409211+0000","last_change":"2024-03-26T03:00:39.931986+0000","last_active":"2024-03-26T03:03:33.409211+0000","last_peered":"2024-03-26T03:03:33.409211+0000","last_clean":"2024-03-26T03:03:33.409211+0000","last_became_active":"2024-03-26T02:59:03.206195+0000","last_became_peered":"2024-03-26T02:59:03.206195+0000","last_unstale":"2024-03-26T03:03:33.409211+0000","last_undegraded":"2024-03-26T03:03:33.409211+0000","last_fullsized":"2024-03-26T03:03:33.409211+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:39.931898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:39.931898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.400926+0000","last_change":"2024-03-26T03:01:48.444274+0000","last_active":"2024-03-26T03:03:33.400926+0000","last_peered":"2024-03-26T03:03:33.400926+0000","last_clean":"2024-03-26T03:03:33.400926+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:03:33.400926+0000","last_undegraded":"2024-03-26T03:03:33.400926+0000","last_fullsized":"2024-03-26T03:03:33.400926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:48.444147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:48.444147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":231,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.407769+0000","last_change":"2024-03-26T03:01:34.223673+0000","last_active":"2024-03-26T03:03:33.407769+0000","last_peered":"2024-03-26T03:03:33.407769+0000","last_clean":"2024-03-26T03:03:33.407769+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:03:33.407769+0000","last_undegraded":"2024-03-26T03:03:33.407769+0000","last_fullsized":"2024-03-26T03:03:33.407769+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:01:34.223531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.223531+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":244,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-03-26T03:03:33.398393+0000","last_change":"2024-03-26T03:01:51.306960+0000","last_active":"2024-03-26T03:03:33.398393+0000","last_peered":"2024-03-26T03:03:33.398393+0000","last_clean":"2024-03-26T03:03:33.398393+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:03:33.398393+0000","last_undegraded":"2024-03-26T03:03:33.398393+0000","last_fullsized":"2024-03-26T03:03:33.398393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:01:51.306839+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:01:51.306839+0000","last_clean_scrub_stamp":"2024-03-26T03:01:51.306839+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":3517612056,"num_objects":53677,"num_object_clones":0,"num_object_copies":161031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":332,"num_objects_misplaced":4884,"num_objects_unfound":0,"num_objects_dirty":53677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118308,"num_write_kb":4681761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19315,"num_bytes_recovered":1265827840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11761954816,"data_stored":11761942600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166295,"ondisk_log_size":166295,"up":78,"acting":75,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":268,"num_read_kb":263,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103228,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1542668,"kb_used_data":1407360,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92202484,"statfs":{"total":95995035648,"available":94415343616,"internally_reserved":0,"allocated":1441136640,"data_stored":1440757170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":138541514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430272,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529480,"kb_used_data":1417788,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92215672,"statfs":{"total":95995035648,"available":94428848128,"internally_reserved":0,"allocated":1451814912,"data_stored":1451438944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1359528,"kb_used_data":1247836,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92385624,"statfs":{"total":95995035648,"available":94602878976,"internally_reserved":0,"allocated":1277784064,"data_stored":1277376604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2194712,"kb_used_data":2023308,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91550440,"statfs":{"total":95995035648,"available":93747650560,"internally_reserved":0,"allocated":2071867392,"data_stored":2071476311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761408,"kb_used_data":1603572,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91983744,"statfs":{"total":95995035648,"available":94191353856,"internally_reserved":0,"allocated":1642057728,"data_stored":1641690441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2083932,"kb_used_data":1944464,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91661220,"statfs":{"total":95995035648,"available":93861089280,"internally_reserved":0,"allocated":1991131136,"data_stored":1990734549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920964,"kb_used_data":809276,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92824188,"statfs":{"total":95995035648,"available":95051968512,"internally_reserved":0,"allocated":828698624,"data_stored":828315625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691910,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1196724,"kb_used_data":1089196,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92548428,"statfs":{"total":95995035648,"available":94769590272,"internally_reserved":0,"allocated":1115336704,"data_stored":1114969640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2068385792,"data_stored":2068381720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1265504256,"data_stored":1265500184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1634926592,"data_stored":1634926592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1983774720,"data_stored":1983774720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":822345728,"data_stored":822345728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1103888384,"data_stored":1103888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1449398272,"data_stored":1449394200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1433731072,"data_stored":1433731072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:03:40.330 INFO:tasks.thrashosds.thrasher:Setting 7.15 to [7, 3, 6] 2024-03-26T03:03:40.330 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.15', '7', '3', '6'] 2024-03-26T03:03:40.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 7.15 7 3 6 2024-03-26T03:03:40.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:40.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:40.502 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:40.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:40.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:40.732 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:40.792 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:40.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:40.937 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:40.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:41.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:41.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:41.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:41.220 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:40 smithi192 ceph-mon[33110]: pgmap v187: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 440 op/s; 321/147618 objects degraded (0.217%); 3495/147618 objects misplaced (2.368%) 2024-03-26T03:03:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:40 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:03:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:40 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:03:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:40 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:03:41.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:40 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:03:41.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[29666]: pgmap v187: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 440 op/s; 321/147618 objects degraded (0.217%); 3495/147618 objects misplaced (2.368%) 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[37920]: pgmap v187: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 440 op/s; 321/147618 objects degraded (0.217%); 3495/147618 objects misplaced (2.368%) 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:03:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:40 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:03:41.363 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.379 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:41.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:41.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:41.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:41.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:41.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:41.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:41.804 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:41.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:41.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:41.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:42.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:42.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:42.106 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:41 smithi066 ceph-mon[29666]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T03:03:42.106 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:03:42.107 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:41 smithi066 ceph-mon[37920]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T03:03:42.107 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:03:42.171 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:42.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:42.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:42.407 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:41 smithi192 ceph-mon[33110]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T03:03:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:03:42.509 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.557 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:42.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:42.592 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:154252 7.1 e144/111 7:83faa59b:::benchmark_data_smithi066_76917_object54069:head v 144'7173, mlcod=144'7171)", 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.316583+0000", 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.014653940000000001, 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014686232, 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.593 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316583+0000", 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316583+0000", 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316585+0000", 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3400000000000001e-06 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316593+0000", 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6000000000000007e-06 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316608+0000", 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4516e-05 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316618+0000", 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0702e-05 2024-03-26T03:03:42.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316740+0000", 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012199 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.316765+0000", 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4927999999999999e-05 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.596 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:154254 7.3 7:c42187de:::benchmark_data_smithi066_76917_object54071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.317227+0000", 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.014009959000000001, 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014128781, 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 154254 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.317227+0000", 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.317227+0000", 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.317229+0000", 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5009999999999999e-06 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.317234+0000", 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.775e-06 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.317244+0000", 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0328e-05 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.317250+0000", 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5570000000000001e-06 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.323266+0000", 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0060151830000000003 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.323377+0000", 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000111864 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.323738+0000", 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000360942 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.330853+0000", 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0071141479999999998 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:154256 7.c e144/96 7:326d5275:::benchmark_data_smithi066_76917_object54073:head v 144'9586, mlcod=144'9584)", 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.318161+0000", 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.013075730000000001, 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013307658999999999, 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:03:42.601 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318161+0000", 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318161+0000", 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318163+0000", 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.178e-06 2024-03-26T03:03:42.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318170+0000", 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1289999999999998e-06 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318178+0000", 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9330000000000001e-06 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318185+0000", 2024-03-26T03:03:42.603 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7769999999999997e-06 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.320868+0000", 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002683768 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.320896+0000", 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7546999999999999e-05 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.331433+0000", 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010537542 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.331472+0000", 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8415999999999999e-05 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.605 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:154257 7.11 e144/121 7:8f23f419:::benchmark_data_smithi066_76917_object54074:head v 144'7138, mlcod=144'7136)", 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.318215+0000", 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.013022076, 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013348864, 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318215+0000", 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318215+0000", 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318216+0000", 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6300000000000004e-07 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318221+0000", 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5310000000000001e-06 2024-03-26T03:03:42.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318229+0000", 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3579999999999997e-06 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.318235+0000", 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9120000000000003e-06 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.319231+0000", 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00099593399999999997 2024-03-26T03:03:42.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.319248+0000", 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7079e-05 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:154260 7.1 e144/111 7:83c237be:::benchmark_data_smithi066_76917_object54077:head v 144'7174, mlcod=144'7171)", 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.322807+0000", 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0084305930000000001, 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.008825711, 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.609 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.322807+0000", 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.322807+0000", 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.610 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.322807+0000", 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1900000000000002e-07 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.322813+0000", 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7030000000000003e-06 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.322819+0000", 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3289999999999999e-06 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.611 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.322824+0000", 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1070000000000004e-06 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327442+0000", 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046171500000000004 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327464+0000", 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2629999999999998e-05 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:154262 7.7 e144/111 7:e058cda8:::benchmark_data_smithi066_76917_object54079:head v 144'7285, mlcod=144'7283)", 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.327038+0000", 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0041993040000000001, 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046578360000000003, 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327038+0000", 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327038+0000", 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327039+0000", 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6199999999999996e-07 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327045+0000", 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6279999999999999e-06 2024-03-26T03:03:42.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327053+0000", 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2740000000000002e-06 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.327059+0000", 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2450000000000003e-06 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.330436+0000", 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0033771510000000001 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.330457+0000", 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.052e-05 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:154263 7.1 e144/111 7:83070d68:::benchmark_data_smithi066_76917_object54080:head v 144'7175, mlcod=144'7171)", 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:42.328134+0000", 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.003103251, 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003654736, 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:03:42.617 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.328134+0000", 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.328134+0000", 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.328135+0000", 2024-03-26T03:03:42.618 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2869999999999999e-06 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.328143+0000", 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.746e-06 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.328152+0000", 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3149999999999998e-06 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.328159+0000", 2024-03-26T03:03:42.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8800000000000002e-06 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.329825+0000", 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0016656749999999999 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:42.329844+0000", 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9513000000000001e-05 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.620 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:03:42.621 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:03:42.621 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 7 2024-03-26T03:03:42.621 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:03:42.657 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.703 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:42.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:42.804 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:42.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:42.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:42.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:42.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:43.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:43.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:43.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:43.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:42 smithi192 ceph-mon[33110]: pgmap v189: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 493 op/s; 321/147618 objects degraded (0.217%); 3495/147618 objects misplaced (2.368%) 2024-03-26T03:03:43.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:43.333 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:42 smithi066 ceph-mon[29666]: pgmap v189: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 493 op/s; 321/147618 objects degraded (0.217%); 3495/147618 objects misplaced (2.368%) 2024-03-26T03:03:43.333 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:43 smithi066 ceph-mon[37920]: pgmap v189: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 493 op/s; 321/147618 objects degraded (0.217%); 3495/147618 objects misplaced (2.368%) 2024-03-26T03:03:43.368 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:43.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:03:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.411 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:43.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:43.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:43.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:43.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:43.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:43.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:43.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:43.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:43.991 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:43.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:44.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:44.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:43 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 321/147618 objects degraded (0.217%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:44.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:43 smithi066 ceph-mon[29666]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T03:03:44.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:03:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:03:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:03:44.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 321/147618 objects degraded (0.217%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:44.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:44 smithi066 ceph-mon[37920]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T03:03:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.148 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:44.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:44.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:43 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 321/147618 objects degraded (0.217%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:44.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:43 smithi192 ceph-mon[33110]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T03:03:44.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:44.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:44.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:44.411 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:44.469 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:44.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:44.521 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:44.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:44.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:44.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:44.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:44.795 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:44.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:44.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:44.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:45.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:45.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:45.059 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[29666]: pgmap v191: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 321/142182 objects degraded (0.226%); 3495/142182 objects misplaced (2.458%); 5.1 MiB/s, 83 objects/s recovering 2024-03-26T03:03:45.059 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[29666]: 5.13 scrub starts 2024-03-26T03:03:45.059 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[29666]: 5.13 scrub ok 2024-03-26T03:03:45.059 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[29666]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T03:03:45.059 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[37920]: pgmap v191: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 321/142182 objects degraded (0.226%); 3495/142182 objects misplaced (2.458%); 5.1 MiB/s, 83 objects/s recovering 2024-03-26T03:03:45.059 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[37920]: 5.13 scrub starts 2024-03-26T03:03:45.060 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[37920]: 5.13 scrub ok 2024-03-26T03:03:45.060 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:45 smithi066 ceph-mon[37920]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T03:03:45.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:45.190 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:45 smithi192 ceph-mon[33110]: pgmap v191: 158 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 321/142182 objects degraded (0.226%); 3495/142182 objects misplaced (2.458%); 5.1 MiB/s, 83 objects/s recovering 2024-03-26T03:03:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:45 smithi192 ceph-mon[33110]: 5.13 scrub starts 2024-03-26T03:03:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:45 smithi192 ceph-mon[33110]: 5.13 scrub ok 2024-03-26T03:03:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:45 smithi192 ceph-mon[33110]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T03:03:45.292 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:45.347 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:45.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:45.504 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:45.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:45.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:45.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:45.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:45.891 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:45.936 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:45.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:46.015 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:46 smithi066 ceph-mon[37920]: 7.10 scrub starts 2024-03-26T03:03:46.016 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:46 smithi066 ceph-mon[37920]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T03:03:46.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:46.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:46.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:46.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:46.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:46 smithi066 ceph-mon[29666]: 7.10 scrub starts 2024-03-26T03:03:46.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:46 smithi066 ceph-mon[29666]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T03:03:46.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:46.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:46 smithi192 ceph-mon[33110]: 7.10 scrub starts 2024-03-26T03:03:46.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:46 smithi192 ceph-mon[33110]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T03:03:46.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:46.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:46.475 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:46.577 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:46.612 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:46.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:46.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:46.844 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:46.874 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:46.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:46.976 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:47.012 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:47.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:47.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:47.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:47.253 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: 5.12 scrub starts 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: 5.12 scrub ok 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: 7.10 scrub ok 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: pgmap v194: 158 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovering+undersized+remapped, 155 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1017 B/s rd, 0 B/s wr, 914 op/s; 1434/133827 objects misplaced (1.072%); 8.5 MiB/s, 200 objects/s recovering 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: 7.14 scrub starts 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:47.280 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[29666]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: 5.12 scrub starts 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: 5.12 scrub ok 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: 7.10 scrub ok 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: pgmap v194: 158 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovering+undersized+remapped, 155 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1017 B/s rd, 0 B/s wr, 914 op/s; 1434/133827 objects misplaced (1.072%); 8.5 MiB/s, 200 objects/s recovering 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: 7.14 scrub starts 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:47.281 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:47 smithi066 ceph-mon[37920]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T03:03:47.307 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:47.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:47.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: 5.12 scrub starts 2024-03-26T03:03:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: 5.12 scrub ok 2024-03-26T03:03:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: 7.10 scrub ok 2024-03-26T03:03:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: pgmap v194: 158 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovering+undersized+remapped, 155 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1017 B/s rd, 0 B/s wr, 914 op/s; 1434/133827 objects misplaced (1.072%); 8.5 MiB/s, 200 objects/s recovering 2024-03-26T03:03:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: 7.14 scrub starts 2024-03-26T03:03:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:47 smithi192 ceph-mon[33110]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T03:03:47.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:47.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:47.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:47.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:47.813 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:47.875 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:47.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:47.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:48.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:48.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[37920]: 7.14 scrub ok 2024-03-26T03:03:48.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[37920]: 4.1d scrub starts 2024-03-26T03:03:48.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[37920]: 4.1d scrub ok 2024-03-26T03:03:48.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:03:48.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:03:48.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[29666]: 7.14 scrub ok 2024-03-26T03:03:48.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[29666]: 4.1d scrub starts 2024-03-26T03:03:48.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[29666]: 4.1d scrub ok 2024-03-26T03:03:48.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:03:48.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:48 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:03:48.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:48.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:48.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:48.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:48.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:48.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:48 smithi192 ceph-mon[33110]: 7.14 scrub ok 2024-03-26T03:03:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:48 smithi192 ceph-mon[33110]: 4.1d scrub starts 2024-03-26T03:03:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:48 smithi192 ceph-mon[33110]: 4.1d scrub ok 2024-03-26T03:03:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:48 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:03:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:48 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:03:48.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:48.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:48.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:48.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:48.803 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:48.829 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:03:48.829 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:03:48.863 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:03:48.881 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:48.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:48.913 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:03:48.962 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:03:48.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:49.008 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:03:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:49.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:49.062 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:03:49.067 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:03:49.105 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:03:49.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:49.127 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:03:49.128 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:03:49.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:49.166 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:03:49.167 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:03:49.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:49.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:49.313 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: 7.19 scrub starts 2024-03-26T03:03:49.313 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: pgmap v196: 158 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovering+undersized+remapped, 155 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 1434/133827 objects misplaced (1.072%); 11 MiB/s, 246 objects/s recovering 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: 7.19 scrub ok 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: 4.c scrub starts 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: 4.c scrub ok 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/152674486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/413372289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: 7.19 scrub starts 2024-03-26T03:03:49.314 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: pgmap v196: 158 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovering+undersized+remapped, 155 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 1434/133827 objects misplaced (1.072%); 11 MiB/s, 246 objects/s recovering 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: 7.19 scrub ok 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: 4.c scrub starts 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: 4.c scrub ok 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/152674486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/413372289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:49.315 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:49.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:49.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:49.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:03:49.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: 7.19 scrub starts 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: pgmap v196: 158 pgs: 1 active+undersized+remapped, 1 peering, 1 active+recovering+undersized+remapped, 155 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 1434/133827 objects misplaced (1.072%); 11 MiB/s, 246 objects/s recovering 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: 7.19 scrub ok 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: 4.c scrub starts 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: 4.c scrub ok 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/152674486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/413372289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:03:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:03:49.512 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.580 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:49.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:49.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:49.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:49.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:49.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:49.944 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:49.988 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:49.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:50.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: 7.12 scrub starts 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/413372289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[29666]: 7.12 scrub ok 2024-03-26T03:03:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: 7.12 scrub starts 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/413372289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:50.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:50 smithi066 ceph-mon[37920]: 7.12 scrub ok 2024-03-26T03:03:50.112 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:03:50.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:50.219 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:50.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.279 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:50.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:50.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.427 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:50.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:03:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:03:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: 7.12 scrub starts 2024-03-26T03:03:50.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:03:50.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/413372289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:03:50.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T03:03:50.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:03:50.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:50 smithi192 ceph-mon[33110]: 7.12 scrub ok 2024-03-26T03:03:50.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:50.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:50.657 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.719 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:50.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:50.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:50.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:50.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:50.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:50.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:51.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:51.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: pgmap v199: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 491/120993 objects misplaced (0.406%) 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: 4.11 deep-scrub starts 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: 4.11 deep-scrub ok 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: 4.9 scrub starts 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[29666]: 4.9 scrub ok 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:03:51.133 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: pgmap v199: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 491/120993 objects misplaced (0.406%) 2024-03-26T03:03:51.134 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: 4.11 deep-scrub starts 2024-03-26T03:03:51.134 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: 4.11 deep-scrub ok 2024-03-26T03:03:51.134 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: 4.9 scrub starts 2024-03-26T03:03:51.134 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:51 smithi066 ceph-mon[37920]: 4.9 scrub ok 2024-03-26T03:03:51.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:51.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:51.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:51.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:51.474 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: pgmap v199: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 491/120993 objects misplaced (0.406%) 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: 4.11 deep-scrub starts 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: 4.11 deep-scrub ok 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: 4.9 scrub starts 2024-03-26T03:03:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:51 smithi192 ceph-mon[33110]: 4.9 scrub ok 2024-03-26T03:03:51.518 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:51.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:51.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:51.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:51.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:51.884 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:51.931 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:51.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:51.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:52.033 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:52.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:52.077 INFO:teuthology.orchestra.run.smithi066.stderr:set 7.15 pg_upmap mapping to [7,3,6] 2024-03-26T03:03:52.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:52.220 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:52.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:52.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:52.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:52 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3565629433' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.15", "id": [7, 3, 6]}]: dispatch 2024-03-26T03:03:52.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:52 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3565629433' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.15", "id": [7, 3, 6]}]: dispatch 2024-03-26T03:03:52.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:52.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:52 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3565629433' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.15", "id": [7, 3, 6]}]: dispatch 2024-03-26T03:03:52.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:52.603 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:52.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:52.734 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:52.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:52.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:52.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:52.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:52.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:53.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:53.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:53.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:53.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:53.337 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:03:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: pgmap v200: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 491/120993 objects misplaced (0.406%) 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3565629433' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.15", "id": [7, 3, 6]}]': finished 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: 4.14 deep-scrub starts 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[29666]: 4.14 deep-scrub ok 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:03:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: pgmap v200: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 491/120993 objects misplaced (0.406%) 2024-03-26T03:03:53.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3565629433' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.15", "id": [7, 3, 6]}]': finished 2024-03-26T03:03:53.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T03:03:53.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: 4.14 deep-scrub starts 2024-03-26T03:03:53.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-mon[37920]: 4.14 deep-scrub ok 2024-03-26T03:03:53.368 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:53.397 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:53.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: pgmap v200: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 156 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 491/120993 objects misplaced (0.406%) 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3565629433' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.15", "id": [7, 3, 6]}]': finished 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T03:03:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: 4.14 deep-scrub starts 2024-03-26T03:03:53.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:53 smithi192 ceph-mon[33110]: 4.14 deep-scrub ok 2024-03-26T03:03:53.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:53.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:53.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:53.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:03:53.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:53.780 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:53.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:53.913 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:53.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:53.989 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:03:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:03:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:03:54.013 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:54.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:54.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:54.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:54.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.298 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:54.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: 5.1c scrub starts 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: 5.1c scrub ok 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-26T03:03:54.304 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: 5.1c scrub starts 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: 5.1c scrub ok 2024-03-26T03:03:54.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-26T03:03:54.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:54.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:54.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: 5.1c scrub starts 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: 5.1c scrub ok 2024-03-26T03:03:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-26T03:03:54.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:54.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:54.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.731 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:54.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:54.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:54.880 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:54.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:54.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:55.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:55.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:55.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:55.228 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:03:55.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.275 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:55.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:55.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:03:55.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:03:55.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 6.0 deep-scrub starts 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 6.0 deep-scrub ok 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: pgmap v203: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 7.15 starting backfill to osd.3 from (0'0,0'0] MAX to 150'7446 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 7.15 starting backfill to osd.6 from (0'0,0'0] MAX to 150'7446 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 7.17 scrub starts 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 6.11 scrub starts 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 6.11 scrub ok 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[29666]: 7.17 scrub ok 2024-03-26T03:03:55.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 6.0 deep-scrub starts 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 6.0 deep-scrub ok 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: pgmap v203: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 7.15 starting backfill to osd.3 from (0'0,0'0] MAX to 150'7446 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 7.15 starting backfill to osd.6 from (0'0,0'0] MAX to 150'7446 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 7.17 scrub starts 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 6.11 scrub starts 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 6.11 scrub ok 2024-03-26T03:03:55.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:55 smithi066 ceph-mon[37920]: 7.17 scrub ok 2024-03-26T03:03:55.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:03:55.437 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:03:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:03:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:03:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 6.0 deep-scrub starts 2024-03-26T03:03:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 6.0 deep-scrub ok 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: pgmap v203: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 7.15 starting backfill to osd.3 from (0'0,0'0] MAX to 150'7446 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 7.15 starting backfill to osd.6 from (0'0,0'0] MAX to 150'7446 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 7.17 scrub starts 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 6.11 scrub starts 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 6.11 scrub ok 2024-03-26T03:03:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:55 smithi192 ceph-mon[33110]: 7.17 scrub ok 2024-03-26T03:03:55.539 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:55.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:55.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:55.746 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:55.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:55.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:55.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:55.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:56.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:56.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:56.148 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:56.250 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:56.296 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:56.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:03:56.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:03:56.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[29666]: 6.f scrub starts 2024-03-26T03:03:56.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[29666]: 6.f scrub ok 2024-03-26T03:03:56.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:03:56.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:03:56.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[37920]: 6.f scrub starts 2024-03-26T03:03:56.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:56 smithi066 ceph-mon[37920]: 6.f scrub ok 2024-03-26T03:03:56.399 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:56.442 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:56.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:56 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:03:56.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:56 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:03:56.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:56 smithi192 ceph-mon[33110]: 6.f scrub starts 2024-03-26T03:03:56.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:56 smithi192 ceph-mon[33110]: 6.f scrub ok 2024-03-26T03:03:56.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:56.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:56.681 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:56.721 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:56.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:56.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:56.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:56.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:56.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:57.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:57.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:57.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:57.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[29666]: pgmap v205: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[37920]: pgmap v205: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:03:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:57 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:03:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:57 smithi192 ceph-mon[33110]: pgmap v205: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:03:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:57 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:03:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:57 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:03:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:57 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:03:57.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:57 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:03:57.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:03:57.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:57.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:57.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:57.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:57.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:57.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:57.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:58.035 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:58.080 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:58.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:58.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:58.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:58.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:58.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[29666]: 7.13 scrub starts 2024-03-26T03:03:58.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[29666]: 4.d deep-scrub starts 2024-03-26T03:03:58.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[29666]: 4.d deep-scrub ok 2024-03-26T03:03:58.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[29666]: 7.13 scrub ok 2024-03-26T03:03:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[37920]: 7.13 scrub starts 2024-03-26T03:03:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[37920]: 4.d deep-scrub starts 2024-03-26T03:03:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[37920]: 4.d deep-scrub ok 2024-03-26T03:03:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:58 smithi066 ceph-mon[37920]: 7.13 scrub ok 2024-03-26T03:03:58.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:03:58.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:03:58.401 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T03:03:58.401 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.596620 2024-03-26T03:03:58.401 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 5 0.5966197683897516 2024-03-26T03:03:58.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:58.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:58 smithi192 ceph-mon[33110]: 7.13 scrub starts 2024-03-26T03:03:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:58 smithi192 ceph-mon[33110]: 4.d deep-scrub starts 2024-03-26T03:03:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:58 smithi192 ceph-mon[33110]: 4.d deep-scrub ok 2024-03-26T03:03:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:58 smithi192 ceph-mon[33110]: 7.13 scrub ok 2024-03-26T03:03:58.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:58.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:58.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:58.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:58.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:03:58.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:58.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:03:58.974 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:59.025 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:59.127 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:59.172 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:59.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:03:59.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:03:59.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:03:59.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:03:59.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:03:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:03:59.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:59 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:03:59.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:59 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:03:59.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:59 smithi192 ceph-mon[33110]: 7.18 scrub starts 2024-03-26T03:03:59.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:59 smithi192 ceph-mon[33110]: pgmap v206: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:03:59.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:03:59 smithi192 ceph-mon[33110]: 7.18 scrub ok 2024-03-26T03:03:59.536 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:03:59.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.564 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:59.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:03:59.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[37920]: 7.18 scrub starts 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[37920]: pgmap v206: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[37920]: 7.18 scrub ok 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[29666]: 7.18 scrub starts 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[29666]: pgmap v206: 158 pgs: 1 creating+peering, 1 remapped+peering, 156 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:03:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:03:59 smithi066 ceph-mon[29666]: 7.18 scrub ok 2024-03-26T03:03:59.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:03:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.712 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:59.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:03:59.812 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:03:59.855 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:03:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:03:59.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:03:59.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:03:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:03:59.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:00.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:00.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:00.217 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:00.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:00.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:00.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:00.474 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:00.504 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:00.607 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:00.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:00.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:00.658 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:00.760 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.800 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:00.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:00.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:00.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:00.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:01.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:01.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:01.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:01.214 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:01.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:01.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:01.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:01.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:01.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: 2.d deep-scrub starts 2024-03-26T03:04:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: 2.d deep-scrub ok 2024-03-26T03:04:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: 4.16 scrub starts 2024-03-26T03:04:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: 4.16 scrub ok 2024-03-26T03:04:01.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: pgmap v207: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 840/99606 objects misplaced (0.843%) 2024-03-26T03:04:01.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:04:01.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:01 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:04:01.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.603 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:01.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: 2.d deep-scrub starts 2024-03-26T03:04:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: 2.d deep-scrub ok 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: 4.16 scrub starts 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: 4.16 scrub ok 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: pgmap v207: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 840/99606 objects misplaced (0.843%) 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: 2.d deep-scrub starts 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: 2.d deep-scrub ok 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: 4.16 scrub starts 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: 4.16 scrub ok 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: pgmap v207: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 840/99606 objects misplaced (0.843%) 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:04:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:01 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:04:01.704 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:01.749 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:01.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:01.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:01.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:01.976 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:02.026 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:02.127 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:02.159 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:02.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:02.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:02.395 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.429 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:02.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:02 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:04:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:02 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:04:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:02 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:04:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:02 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:04:02.531 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:02.570 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:04:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:02 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:04:02.672 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.702 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:02.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:02.794 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:02.795 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:02.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:02.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:02.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:02.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:02.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:03.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.085 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:03.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:03.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:03.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:03.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:03.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:03.453 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: 4.4 scrub starts 2024-03-26T03:04:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: 4.4 scrub ok 2024-03-26T03:04:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:04:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:04:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: pgmap v208: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 840/95820 objects misplaced (0.877%) 2024-03-26T03:04:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T03:04:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: 4.15 scrub starts 2024-03-26T03:04:03.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:03 smithi192 ceph-mon[33110]: 4.15 scrub ok 2024-03-26T03:04:03.494 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:03.596 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: 4.4 scrub starts 2024-03-26T03:04:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: 4.4 scrub ok 2024-03-26T03:04:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: pgmap v208: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 840/95820 objects misplaced (0.877%) 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: 4.15 scrub starts 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[29666]: 4.15 scrub ok 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: 4.4 scrub starts 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: 4.4 scrub ok 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: pgmap v208: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.9 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 840/95820 objects misplaced (0.877%) 2024-03-26T03:04:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T03:04:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: 4.15 scrub starts 2024-03-26T03:04:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-mon[37920]: 4.15 scrub ok 2024-03-26T03:04:03.640 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:03.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:03.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:03.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:03.945 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:03.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:04.046 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:04.082 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:04:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:04:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:04:04.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:04.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:04.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:04.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:04.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: 2.15 deep-scrub starts 2024-03-26T03:04:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: 2.15 deep-scrub ok 2024-03-26T03:04:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:04:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:04:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T03:04:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: 7.15 scrub starts 2024-03-26T03:04:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:04 smithi192 ceph-mon[33110]: 7.15 scrub ok 2024-03-26T03:04:04.495 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: 2.15 deep-scrub starts 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: 2.15 deep-scrub ok 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: 7.15 scrub starts 2024-03-26T03:04:04.502 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[29666]: 7.15 scrub ok 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: 2.15 deep-scrub starts 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: 2.15 deep-scrub ok 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: 7.15 scrub starts 2024-03-26T03:04:04.503 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:04 smithi066 ceph-mon[37920]: 7.15 scrub ok 2024-03-26T03:04:04.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:04.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:04.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:04.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:04.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:04.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:04.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:04.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:05.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:05.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:05.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:05.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:05.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:05.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 4.1e scrub starts 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: pgmap v211: 158 pgs: 158 active+clean; 1.8 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 4.1e scrub ok 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:05.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 4.1e scrub starts 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: pgmap v211: 158 pgs: 158 active+clean; 1.8 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 4.1e scrub ok 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:05.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:05 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:05.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:04:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 4.1e scrub starts 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: pgmap v211: 158 pgs: 158 active+clean; 1.8 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 4.1e scrub ok 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:05 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:05.517 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:05.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:05.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:05.754 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:05.807 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:05.840 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:05.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:04:05.910 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:05.988 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:05.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:06.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:06.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:06.148 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:06.211 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:06.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 7.11 scrub starts 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: 7.11 scrub ok 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:04:06.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:06 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:06.282 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:06.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:06.416 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:06.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:04:06.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 7.11 scrub starts 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: 7.11 scrub ok 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:06.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 7.11 scrub starts 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: 7.11 scrub ok 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:04:06.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:06 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:06.518 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:06.568 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:06.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:06.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:06.807 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:06.854 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:06.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:06.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:06.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:07.092 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:07.143 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:07.186 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:04:07.245 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:07.290 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:07.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:07.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:04:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:04:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: pgmap v212: 158 pgs: 158 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 6.5 scrub starts 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: 6.5 scrub ok 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2448478737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:07 smithi192 ceph-mon[33110]: from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:07.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:07.573 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:07.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: pgmap v212: 158 pgs: 158 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 6.5 scrub starts 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: 6.5 scrub ok 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2448478737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[29666]: from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: pgmap v212: 158 pgs: 158 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 6.5 scrub starts 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: 6.5 scrub ok 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2448478737' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:07.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:07 smithi066 ceph-mon[37920]: from='client.24844 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:07.676 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:07.724 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:07.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:07.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:07.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:07.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:07.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:08.100 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:08.145 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:08.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:08.205 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:08.205 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:08.247 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:08.289 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[29666]: 4.b deep-scrub starts 2024-03-26T03:04:08.289 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[29666]: 4.b deep-scrub ok 2024-03-26T03:04:08.289 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:08.289 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[29666]: from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:04:08.289 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[29666]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T03:04:08.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[37920]: 4.b deep-scrub starts 2024-03-26T03:04:08.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[37920]: 4.b deep-scrub ok 2024-03-26T03:04:08.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:08.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[37920]: from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:04:08.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:08 smithi066 ceph-mon[37920]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T03:04:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:08.291 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:08.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:08.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:08.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:08.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:08.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:08 smithi192 ceph-mon[33110]: 4.b deep-scrub starts 2024-03-26T03:04:08.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:08 smithi192 ceph-mon[33110]: 4.b deep-scrub ok 2024-03-26T03:04:08.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:08 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:08.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:08 smithi192 ceph-mon[33110]: from='client.24844 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:04:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:08 smithi192 ceph-mon[33110]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T03:04:08.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:08.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:08.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:08.672 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:08.731 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:08.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:08.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:08.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:08.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:08.968 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:09.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:04:09.011 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:09.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:09.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:09.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:09.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:09.409 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:09.425 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.462 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:09.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:09.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:04:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:04:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:09 smithi192 ceph-mon[33110]: pgmap v214: 158 pgs: 158 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:04:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:04:09.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:09.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[29666]: pgmap v214: 158 pgs: 158 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[37920]: pgmap v214: 158 pgs: 158 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:04:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:04:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:09.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:09.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:09.836 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:09.875 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:09.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:09.979 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:10.026 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:10.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:10.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:10.169 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:10.181 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:04:10.272 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:10.390 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:10.402 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:10.403 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:10 smithi066 ceph-mon[29666]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T03:04:10.403 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:10.403 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:10 smithi066 ceph-mon[37920]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T03:04:10.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:10.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:10.602 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:10.602 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:10 smithi192 ceph-mon[33110]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T03:04:10.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:10.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:10.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:10.766 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:04:10.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:10.813 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:04:10.854 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:04:10.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:10.898 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:04:10.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:10.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:10.939 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:04:10.980 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:04:11.022 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:04:11.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:11.068 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:11.070 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:04:11.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:04:11.171 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:11.215 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:11.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:11.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:11.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:11.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:11.510 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:11.511 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:04:11.511 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:04:11.511 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:04:11.511 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:11.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.633 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:11.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:11.734 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:11 smithi192 ceph-mon[33110]: pgmap v216: 158 pgs: 158 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:04:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2823983669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:11.760 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:11 smithi066 ceph-mon[29666]: pgmap v216: 158 pgs: 158 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:04:11.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2823983669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:11.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:11 smithi066 ceph-mon[37920]: pgmap v216: 158 pgs: 158 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:04:11.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2823983669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:11.776 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:11.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:11.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:11.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:12.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:12.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:12.149 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:12.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:12.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:04:12.296 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:12.344 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:12.357 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:12.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.501 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:12.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:12.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:12.627 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:12.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:12.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:12.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:12.889 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:12.934 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:12.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:13.036 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:13.082 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:13.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:13.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:13.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:13.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:13.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:13.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:13.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:13.406 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.5 primary-affinity to 0.59662 (8391002) 2024-03-26T03:04:13.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:13.604 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:13.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:13.648 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[29666]: pgmap v217: 158 pgs: 158 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 387 op/s 2024-03-26T03:04:13.648 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:04:13.648 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:04:13.648 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3823696910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]: dispatch 2024-03-26T03:04:13.649 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[29666]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]: dispatch 2024-03-26T03:04:13.649 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[37920]: pgmap v217: 158 pgs: 158 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 387 op/s 2024-03-26T03:04:13.649 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:04:13.649 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:04:13.649 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3823696910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]: dispatch 2024-03-26T03:04:13.649 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-mon[37920]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]: dispatch 2024-03-26T03:04:13.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:13 smithi192 ceph-mon[33110]: pgmap v217: 158 pgs: 158 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 387 op/s 2024-03-26T03:04:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:13 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:04:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:13 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:04:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:13 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3823696910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]: dispatch 2024-03-26T03:04:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:13 smithi192 ceph-mon[33110]: from='client.24853 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]: dispatch 2024-03-26T03:04:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:13.764 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:13.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:13.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:13.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:13.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:13.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:14.010 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:04:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:04:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:04:14.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:14.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:14.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:14.132 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:04:14.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:14.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:14.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:14.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:14.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:14.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:14.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:14.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:14.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:14 smithi192 ceph-mon[33110]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]': finished 2024-03-26T03:04:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:14 smithi192 ceph-mon[33110]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T03:04:14.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:14.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:14.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:14.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:14 smithi066 ceph-mon[29666]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]': finished 2024-03-26T03:04:14.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:14 smithi066 ceph-mon[29666]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T03:04:14.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:14 smithi066 ceph-mon[37920]: from='client.24853 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5966197683897516}]': finished 2024-03-26T03:04:14.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:14 smithi066 ceph-mon[37920]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T03:04:14.709 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:14.772 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:14.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28229 7.1 7:81134404:::benchmark_data_smithi066_76917_object28228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.292280+0000", 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 179.55809755999999, 2024-03-26T03:04:14.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.110167232, 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28229 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292280+0000", 2024-03-26T03:04:14.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292280+0000", 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292282+0000", 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292321+0000", 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8554999999999999e-05 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292344+0000", 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2906999999999999e-05 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292350+0000", 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1510000000000001e-06 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292408+0000", 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8455999999999997e-05 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292540+0000", 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131292 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292689+0000", 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014918900000000001 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295453+0000", 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027638609999999998 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295465+0000", 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2139e-05 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402380+0000", 2024-03-26T03:04:14.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.106915063 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402418+0000", 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7787999999999998e-05 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402447+0000", 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9796999999999999e-05 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28233 7.1 7:88f223fb:::benchmark_data_smithi066_76917_object28232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 179.55800624299999, 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11016767, 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28233 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.860 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6199999999999996e-07 2024-03-26T03:04:14.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292414+0000", 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-05 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292422+0000", 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8599999999999993e-06 2024-03-26T03:04:14.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292428+0000", 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2960000000000004e-06 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.293764+0000", 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001336215 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.293895+0000", 2024-03-26T03:04:14.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013044700000000001 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.294096+0000", 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020120000000000001 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295610+0000", 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015145429999999999 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.296215+0000", 2024-03-26T03:04:14.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060502799999999997 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402479+0000", 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10626340099999999 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402514+0000", 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4782e-05 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402539+0000", 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5636e-05 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28781 7.1 7:87d55f3c:::benchmark_data_smithi066_76917_object28780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.644242+0000", 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 179.20613574000001, 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.142996018, 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.866 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28781 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644242+0000", 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644242+0000", 2024-03-26T03:04:14.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644244+0000", 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.232e-06 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644294+0000", 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9721e-05 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644314+0000", 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0225999999999999e-05 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644324+0000", 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4550000000000001e-06 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644401+0000", 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7171999999999999e-05 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644519+0000", 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118093 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644644+0000", 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012525399999999999 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645900+0000", 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012556819999999999 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.646099+0000", 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019884899999999999 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787137+0000", 2024-03-26T03:04:14.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14103827899999999 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787191+0000", 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4150999999999997e-05 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787238+0000", 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6903999999999997e-05 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:29443 7.1 7:8992cdbb:::benchmark_data_smithi066_76917_object29442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.091655+0000", 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 178.75872311000001, 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17979831600000001, 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 29443 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091655+0000", 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091655+0000", 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091657+0000", 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9319999999999999e-06 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091723+0000", 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6415999999999996e-05 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091736+0000", 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2965e-05 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091744+0000", 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8890000000000002e-06 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091809+0000", 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5334999999999997e-05 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091930+0000", 2024-03-26T03:04:14.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120327 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.092053+0000", 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012322900000000001 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.093050+0000", 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00099677399999999997 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.093374+0000", 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032402400000000001 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.271342+0000", 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17796842700000001 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.271409+0000", 2024-03-26T03:04:14.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7268999999999996e-05 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.271453+0000", 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3729000000000001e-05 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:29453 7.1 7:8ca24ebb:::benchmark_data_smithi066_76917_object29452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.101594+0000", 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 178.74878424900001, 2024-03-26T03:04:14.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17664884, 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 29453 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101594+0000", 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101594+0000", 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101595+0000", 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.796e-06 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101653+0000", 2024-03-26T03:04:14.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7182999999999998e-05 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101663+0000", 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.062e-05 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101669+0000", 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6049999999999997e-06 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101722+0000", 2024-03-26T03:04:14.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2827000000000001e-05 2024-03-26T03:04:14.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101823+0000", 2024-03-26T03:04:14.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010179 2024-03-26T03:04:14.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101919+0000", 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5328000000000006e-05 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.102836+0000", 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00091741399999999997 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.102981+0000", 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014494200000000001 2024-03-26T03:04:14.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.278161+0000", 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.175180164 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.278212+0000", 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0963e-05 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.278242+0000", 2024-03-26T03:04:14.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0207999999999999e-05 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30122 7.1 7:8b71f513:::benchmark_data_smithi066_76917_object30121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.589752+0000", 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 178.260625796, 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.163766524, 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30122 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589752+0000", 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589752+0000", 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589754+0000", 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9290000000000001e-06 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589808+0000", 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.359e-05 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589823+0000", 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5143000000000001e-05 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589831+0000", 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4370000000000007e-06 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589891+0000", 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9944000000000001e-05 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.590029+0000", 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013794100000000001 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.590172+0000", 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014341199999999999 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591392+0000", 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001219176 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591642+0000", 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025028600000000001 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.753429+0000", 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16178671 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.753472+0000", 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3648e-05 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.753519+0000", 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6307999999999998e-05 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30127 7.1 7:808933d7:::benchmark_data_smithi066_76917_object30126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.592242+0000", 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 178.25813538599999, 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16545906399999999, 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.894 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30127 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592242+0000", 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592242+0000", 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592244+0000", 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7880000000000001e-06 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592286+0000", 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1542999999999998e-05 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592299+0000", 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2835e-05 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592307+0000", 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6899999999999998e-06 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592365+0000", 2024-03-26T03:04:14.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7985000000000001e-05 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592486+0000", 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120586 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592668+0000", 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018237200000000001 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.595129+0000", 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024603810000000002 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.595628+0000", 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049977500000000005 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757605+0000", 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.161976328 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757662+0000", 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7012999999999998e-05 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757701+0000", 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9768000000000003e-05 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31273 7.1 7:8ded8593:::benchmark_data_smithi066_76917_object31272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 177.11679728600001, 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14420369, 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31273 2024-03-26T03:04:14.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9999999999999996e-07 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733615+0000", 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3147e-05 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733623+0000", 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7309999999999999e-06 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733631+0000", 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9680000000000006e-06 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.734918+0000", 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012864090000000001 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.735011+0000", 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3194000000000003e-05 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.735122+0000", 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000110952 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.736425+0000", 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001302896 2024-03-26T03:04:14.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.736477+0000", 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2564e-05 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.877677+0000", 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14120012800000001 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.877738+0000", 2024-03-26T03:04:14.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0130000000000002e-05 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.877784+0000", 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6671000000000001e-05 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31280 7.1 7:8c9719fd:::benchmark_data_smithi066_76917_object31279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.738079+0000", 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 177.11229919799999, 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14374624499999999, 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31280 2024-03-26T03:04:14.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738079+0000", 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738079+0000", 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738080+0000", 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3820000000000001e-06 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738152+0000", 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1687999999999995e-05 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738165+0000", 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3115000000000001e-05 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738173+0000", 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5790000000000004e-06 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738216+0000", 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2929000000000002e-05 2024-03-26T03:04:14.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738340+0000", 2024-03-26T03:04:14.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:14.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012344499999999999 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738519+0000", 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017883499999999999 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.739797+0000", 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012786220000000001 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.740145+0000", 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034736400000000002 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881757+0000", 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14161243200000001 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881798+0000", 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0618999999999999e-05 2024-03-26T03:04:14.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881825+0000", 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7234999999999999e-05 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31288 7.1 7:899370f3:::benchmark_data_smithi066_76917_object31287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.741047+0000", 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 177.10933103299999, 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.140931367, 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31288 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741047+0000", 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741047+0000", 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741049+0000", 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9080000000000002e-06 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741107+0000", 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8177999999999998e-05 2024-03-26T03:04:14.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741123+0000", 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6472000000000001e-05 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741132+0000", 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4079999999999998e-06 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741193+0000", 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1387e-05 2024-03-26T03:04:14.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741314+0000", 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120697 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741445+0000", 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131128 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.742377+0000", 2024-03-26T03:04:14.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00093240100000000002 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.742544+0000", 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016628199999999999 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881873+0000", 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.139329279 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881928+0000", 2024-03-26T03:04:14.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5538000000000003e-05 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881978+0000", 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9688999999999998e-05 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:70253 7.1 7:879a87ed:::benchmark_data_smithi066_76917_object70252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.078364+0000", 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 149.77201406200001, 2024-03-26T03:04:14.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.127659565, 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 70253 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078364+0000", 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078364+0000", 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078366+0000", 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3089999999999998e-06 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078416+0000", 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0214999999999999e-05 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078427+0000", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0276e-05 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078432+0000", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7209999999999999e-06 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078500+0000", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7341999999999993e-05 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078629+0000", 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012928200000000001 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078766+0000", 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137179 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.082939+0000", 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041725970000000001 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.085752+0000", 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028134219999999999 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205898+0000", 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.120146137 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205958+0000", 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9562999999999999e-05 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206023+0000", 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5522e-05 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:70256 7.1 7:80dfd34f:::benchmark_data_smithi066_76917_object70255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.079991+0000", 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 149.77038664700001, 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.129241039, 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 70256 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.079991+0000", 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.079991+0000", 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.079993+0000", 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.604e-06 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.080022+0000", 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9312e-05 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.080034+0000", 2024-03-26T03:04:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1834000000000001e-05 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.080042+0000", 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9130000000000003e-06 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.081063+0000", 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010207129999999999 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.081186+0000", 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000123743 2024-03-26T03:04:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.081305+0000", 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011904999999999999 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.083049+0000", 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001744128 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205854+0000", 2024-03-26T03:04:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.122804116 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.209083+0000", 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003229043 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.209186+0000", 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103151 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.209232+0000", 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6431999999999999e-05 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:70259 7.9 e107/96 7:9c43e8b1:::benchmark_data_smithi066_76917_object70258:head v 107'4278, mlcod=107'4276)", 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.088001+0000", 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 149.76237644299999, 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13058771799999999, 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088001+0000", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088001+0000", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088003+0000", 2024-03-26T03:04:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.556e-06 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088044+0000", 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1539999999999999e-05 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088054+0000", 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0030999999999999e-05 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088062+0000", 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3019999999999997e-06 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206537+0000", 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.118474827 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206559+0000", 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2141e-05 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.218546+0000", 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011987506 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.218589+0000", 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2815000000000003e-05 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:70266 7.3 e107/96 7:cc21e419:::benchmark_data_smithi066_76917_object70265:head v 107'4406, mlcod=107'4404)", 2024-03-26T03:04:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.088266+0000", 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 149.76211165500001, 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.122352424, 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088266+0000", 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088266+0000", 2024-03-26T03:04:14.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088267+0000", 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.048e-06 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088328+0000", 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0368000000000002e-05 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205571+0000", 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11724303799999999 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205587+0000", 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5962999999999999e-05 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206072+0000", 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00048576000000000001 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206093+0000", 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0253000000000001e-05 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.210542+0000", 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044494640000000002 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.210619+0000", 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6530000000000001e-05 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:75520 7.8 e110/96)", 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.269554+0000", 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 144.58082376499999, 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.120205865, 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269554+0000", 2024-03-26T03:04:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269554+0000", 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269564+0000", 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0151e-05 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269623+0000", 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8856e-05 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269637+0000", 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3495e-05 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269647+0000", 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0515e-05 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.389740+0000", 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12009255100000001 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.389760+0000", 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0296999999999999e-05 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:75526 7.3 e110/96)", 2024-03-26T03:04:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.275946+0000", 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 144.57443168399999, 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.110026236, 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.275946+0000", 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.275946+0000", 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.275949+0000", 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6199999999999999e-06 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276029+0000", 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0006999999999995e-05 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276051+0000", 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2232999999999999e-05 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276061+0000", 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9000000000000001e-06 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.385943+0000", 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.109882037 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.385972+0000", 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9439000000000001e-05 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:75525 7.4 e110/96)", 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.276113+0000", 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 144.57426497500001, 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11648288499999999, 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276113+0000", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276113+0000", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276114+0000", 2024-03-26T03:04:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6599999999999994e-07 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276151+0000", 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7635000000000001e-05 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276159+0000", 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3300000000000001e-06 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276164+0000", 2024-03-26T03:04:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0459999999999996e-06 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.392574+0000", 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.116410069 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.392596+0000", 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1838999999999999e-05 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75533 7.11 7:884ac876:::benchmark_data_smithi066_76917_object75532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.320059+0000", 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 144.53031856600001, 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.028185074, 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75533 2024-03-26T03:04:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320059+0000", 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320059+0000", 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320063+0000", 2024-03-26T03:04:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8959999999999996e-06 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320130+0000", 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6713999999999999e-05 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320161+0000", 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0799000000000001e-05 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320176+0000", 2024-03-26T03:04:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5477000000000001e-05 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.393808+0000", 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.073632049000000005 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.393820+0000", 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2065999999999999e-05 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.340179+0000", 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94635899000000001 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.340309+0000", 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013020100000000001 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.340475+0000", 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016509200000000001 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.341934+0000", 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014593360000000001 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.342178+0000", 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024444700000000001 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.348143+0000", 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059647160000000001 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.348201+0000", 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7819000000000003e-05 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.348244+0000", 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3472e-05 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:111993 7.1 7:83b0452a:::benchmark_data_smithi066_76917_object11810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:04:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:50.996705+0000", 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 83.853672739000004, 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13555013299999999, 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111993 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996705+0000", 2024-03-26T03:04:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996705+0000", 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996708+0000", 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4880000000000001e-06 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996714+0000", 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7240000000000003e-06 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996730+0000", 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5498e-05 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996739+0000", 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4779999999999997e-06 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996813+0000", 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3743000000000002e-05 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996948+0000", 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000135227 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.997298+0000", 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034946199999999998 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.999128+0000", 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001829883 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.999771+0000", 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064344100000000002 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.132141+0000", 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13236953600000001 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.132203+0000", 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2395000000000001e-05 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.132255+0000", 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2258e-05 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112710 7.16 7:6af1cf5d:::benchmark_data_smithi066_76917_object12527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:52.511950+0000", 2024-03-26T03:04:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 82.338427805999999, 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.161410046, 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112710 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511950+0000", 2024-03-26T03:04:14.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511950+0000", 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511952+0000", 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7659999999999999e-06 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511957+0000", 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3689999999999998e-06 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511980+0000", 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3217000000000001e-05 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511989+0000", 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7260000000000008e-06 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.512069+0000", 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9432999999999997e-05 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.512188+0000", 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119064 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.512531+0000", 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034361400000000001 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.514375+0000", 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001843337 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.514542+0000", 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016794499999999999 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.673196+0000", 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.15865315499999999 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.673288+0000", 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2441000000000005e-05 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.673360+0000", 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1978999999999996e-05 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:14.972 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:14.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:15.039 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:15.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:15.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:15.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:15.274 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:15.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:15.423 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:15.476 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:15 smithi066 ceph-mon[29666]: pgmap v219: 158 pgs: 3 peering, 155 active+clean; 1.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 598 B/s rd, 0 B/s wr, 441 op/s 2024-03-26T03:04:15.476 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:15 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T03:04:15.476 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:15 smithi066 ceph-mon[29666]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T03:04:15.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:15 smithi066 ceph-mon[37920]: pgmap v219: 158 pgs: 3 peering, 155 active+clean; 1.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 598 B/s rd, 0 B/s wr, 441 op/s 2024-03-26T03:04:15.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:15 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T03:04:15.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:15 smithi066 ceph-mon[37920]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T03:04:15.525 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:15.576 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:15.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:15.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:15 smithi192 ceph-mon[33110]: pgmap v219: 158 pgs: 3 peering, 155 active+clean; 1.1 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 598 B/s rd, 0 B/s wr, 441 op/s 2024-03-26T03:04:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:15 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T03:04:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:15 smithi192 ceph-mon[33110]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T03:04:15.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:15.850 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:15.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:15.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:15.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:16.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:16.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:16.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:16.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:04:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.271 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:16.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:16.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:16.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:16.436 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:04:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4211096701' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:16 smithi192 ceph-mon[33110]: from='client.24859 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:16 smithi192 ceph-mon[33110]: pgmap v221: 158 pgs: 5 peering, 153 active+clean; 962 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:04:16.510 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:16.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4211096701' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:16 smithi066 ceph-mon[29666]: from='client.24859 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:16 smithi066 ceph-mon[29666]: pgmap v221: 158 pgs: 5 peering, 153 active+clean; 962 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:04:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4211096701' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:16 smithi066 ceph-mon[37920]: from='client.24859 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:16 smithi066 ceph-mon[37920]: pgmap v221: 158 pgs: 5 peering, 153 active+clean; 962 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 614 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:04:16.617 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:16.668 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.719 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:16.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:16.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:16.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:16.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:16.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:16.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:17.092 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:04:17.092 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:17.135 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:04:17.142 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:17.181 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:04:17.231 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:04:17.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:17.268 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:04:17.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:17.314 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:04:17.358 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:04:17.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:17.396 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:04:17.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:04:17.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:17.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:17.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:17.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[29666]: from='client.24859 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[29666]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3012575411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[37920]: from='client.24859 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[37920]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T03:04:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3012575411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:17.642 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.700 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:17.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:17 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:17 smithi192 ceph-mon[33110]: from='client.24859 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:04:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:17 smithi192 ceph-mon[33110]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T03:04:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3012575411' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:17.801 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:17.847 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:17.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:17.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:17.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:18.077 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:18.122 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:18.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:18.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:18.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:18.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:18.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:18.558 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:18.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:18.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:18 smithi192 ceph-mon[33110]: pgmap v223: 158 pgs: 5 peering, 153 active+clean; 962 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 978 op/s 2024-03-26T03:04:18.802 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:18.842 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:18.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:18 smithi066 ceph-mon[29666]: pgmap v223: 158 pgs: 5 peering, 153 active+clean; 962 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 978 op/s 2024-03-26T03:04:18.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:18 smithi066 ceph-mon[37920]: pgmap v223: 158 pgs: 5 peering, 153 active+clean; 962 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 978 op/s 2024-03-26T03:04:18.944 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:18.989 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:18.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:19.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:19.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:19.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:19.136 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:19.136 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:19.236 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:19.282 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:19.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:19.361 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:04:19.361 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:04:19.362 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-26T03:04:19.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:04:19.384 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:19.462 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:19.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:19.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:04:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:04:19.565 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:19.633 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:19.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:19.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:04:19.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:19.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:19.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:19.820 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:19.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:19.938 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:19.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:20.039 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:20.101 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:20.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:19 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:20.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:19 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:20.131 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:19 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:20.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:04:20.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:20.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:20.347 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.394 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:20.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:20.495 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:20.496 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:20.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:04:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.545 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:20.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:20.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:20.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:20.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:20.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:20.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:20.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:20.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:21.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:21.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:21.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:21.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:21.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:20 smithi192 ceph-mon[33110]: pgmap v224: 158 pgs: 158 active+clean; 667 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-26T03:04:21.302 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:21.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:21.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:21.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:20 smithi066 ceph-mon[29666]: pgmap v224: 158 pgs: 158 active+clean; 667 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-26T03:04:21.362 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:20 smithi066 ceph-mon[37920]: pgmap v224: 158 pgs: 158 active+clean; 667 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-03-26T03:04:21.382 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:21.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:21.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:21.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:21.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:21.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:21.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:21.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:21.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:21.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:21.970 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:21 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-26T03:04:21.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:22.038 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:22.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:22.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:22.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:21 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-26T03:04:22.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:22.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:21 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-26T03:04:22.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:22.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:22.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:22.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:22.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:22.602 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:04:22.609 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:04:22.668 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:22.708 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:22.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:22.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:22.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:22.996 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:22.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:23.089 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:23.089 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:23.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:23.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:22 smithi066 ceph-mon[29666]: pgmap v225: 158 pgs: 158 active+clean; 584 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T03:04:23.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:22 smithi066 ceph-mon[37920]: pgmap v225: 158 pgs: 158 active+clean; 584 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T03:04:23.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:23.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:23.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:22 smithi192 ceph-mon[33110]: pgmap v225: 158 pgs: 158 active+clean; 584 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T03:04:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:23.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:23.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:23.490 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:23.533 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:23.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:23.681 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:23.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:23.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:23.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:23.970 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:23.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:24.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:24.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:23 smithi066 ceph-mon[37920]: from='client.24871 v1:172.21.15.66:0/86765829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:04:24.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:24.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:04:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:04:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:04:24.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:23 smithi066 ceph-mon[29666]: from='client.24871 v1:172.21.15.66:0/86765829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:04:24.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:24.117 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:24.218 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:24.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:23 smithi192 ceph-mon[33110]: from='client.24871 v1:172.21.15.66:0/86765829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:04:24.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:24.261 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:24.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:24.405 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:24.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:24.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:24.641 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:24.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:24.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:24.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:24.927 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:24.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:24.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:25.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:25.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:25.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:25.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:25.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:25.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:25.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:24 smithi066 ceph-mon[37920]: pgmap v226: 158 pgs: 158 active+clean; 436 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 532 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T03:04:25.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:24 smithi066 ceph-mon[29666]: pgmap v226: 158 pgs: 158 active+clean; 436 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 532 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T03:04:25.366 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:25.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:24 smithi192 ceph-mon[33110]: pgmap v226: 158 pgs: 158 active+clean; 436 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 532 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T03:04:25.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:25.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:25.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:25.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:25.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:25.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:25.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:25.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:25.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:26.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:26.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.161 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:26.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:26.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:26.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:26.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:04:26.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.456 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:26.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:26.521 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-03-26T03:04:22.025715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":611916224,"num_objects":9559,"num_object_clones":0,"num_object_copies":28677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9559,"num_whiteouts":0,"num_read":1535,"num_read_kb":1349,"num_write":117173,"num_write_kb":2877086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152475,"ondisk_log_size":152475,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":483,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3485296,"kb_used_data":2258472,"kb_used_omap":12,"kb_used_meta":1226675,"kb_avail":746475920,"statfs":{"total":767960285184,"available":764391342080,"internally_reserved":0,"allocated":2312675328,"data_stored":2309454480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1256115760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,7,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-615841792,"num_objects":-9397,"num_object_clones":0,"num_object_copies":-28191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9397,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006965"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422540+0000","last_change":"2024-03-26T03:00:51.383778+0000","last_active":"2024-03-26T03:04:16.422540+0000","last_peered":"2024-03-26T03:04:16.422540+0000","last_clean":"2024-03-26T03:04:16.422540+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:04:16.422540+0000","last_undegraded":"2024-03-26T03:04:16.422540+0000","last_fullsized":"2024-03-26T03:04:16.422540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.383695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.383695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.421991+0000","last_change":"2024-03-26T03:03:27.420047+0000","last_active":"2024-03-26T03:04:16.421991+0000","last_peered":"2024-03-26T03:04:16.421991+0000","last_clean":"2024-03-26T03:04:16.421991+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:04:16.421991+0000","last_undegraded":"2024-03-26T03:04:16.421991+0000","last_fullsized":"2024-03-26T03:04:16.421991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:27.419925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:27.419925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:56.331025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428740+0000","last_change":"2024-03-26T03:01:49.986072+0000","last_active":"2024-03-26T03:04:16.428740+0000","last_peered":"2024-03-26T03:04:16.428740+0000","last_clean":"2024-03-26T03:04:16.428740+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:04:16.428740+0000","last_undegraded":"2024-03-26T03:04:16.428740+0000","last_fullsized":"2024-03-26T03:04:16.428740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.985144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.985144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":283,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425214+0000","last_change":"2024-03-26T03:00:50.380208+0000","last_active":"2024-03-26T03:04:16.425214+0000","last_peered":"2024-03-26T03:04:16.425214+0000","last_clean":"2024-03-26T03:04:16.425214+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:04:16.425214+0000","last_undegraded":"2024-03-26T03:04:16.425214+0000","last_fullsized":"2024-03-26T03:04:16.425214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:00:50.380106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:50.380106+0000","objects_scrubbed":11,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422056+0000","last_change":"2024-03-26T03:04:01.340828+0000","last_active":"2024-03-26T03:04:16.422056+0000","last_peered":"2024-03-26T03:04:16.422056+0000","last_clean":"2024-03-26T03:04:16.422056+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:04:16.422056+0000","last_undegraded":"2024-03-26T03:04:16.422056+0000","last_fullsized":"2024-03-26T03:04:16.422056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.340706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.340706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:08.035568+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423557+0000","last_change":"2024-03-26T03:03:51.706901+0000","last_active":"2024-03-26T03:04:16.423557+0000","last_peered":"2024-03-26T03:04:16.423557+0000","last_clean":"2024-03-26T03:04:16.423557+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:04:16.423557+0000","last_undegraded":"2024-03-26T03:04:16.423557+0000","last_fullsized":"2024-03-26T03:04:16.423557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:51.706771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:51.706771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:21.243226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423216+0000","last_change":"2024-03-26T03:00:41.464704+0000","last_active":"2024-03-26T03:04:16.423216+0000","last_peered":"2024-03-26T03:04:16.423216+0000","last_clean":"2024-03-26T03:04:16.423216+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:04:16.423216+0000","last_undegraded":"2024-03-26T03:04:16.423216+0000","last_fullsized":"2024-03-26T03:04:16.423216+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:41.464590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.464590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":236,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.421683+0000","last_change":"2024-03-26T03:00:41.276506+0000","last_active":"2024-03-26T03:04:16.421683+0000","last_peered":"2024-03-26T03:04:16.421683+0000","last_clean":"2024-03-26T03:04:16.421683+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:04:16.421683+0000","last_undegraded":"2024-03-26T03:04:16.421683+0000","last_fullsized":"2024-03-26T03:04:16.421683+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:00:41.276382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.276382+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:18.154766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"160'8330","reported_seq":15510,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:20.024985+0000","last_change":"2024-03-26T03:03:58.513439+0000","last_active":"2024-03-26T03:04:20.024985+0000","last_peered":"2024-03-26T03:04:20.024985+0000","last_clean":"2024-03-26T03:04:20.024985+0000","last_became_active":"2024-03-26T03:03:30.396029+0000","last_became_peered":"2024-03-26T03:03:30.396029+0000","last_unstale":"2024-03-26T03:04:20.024985+0000","last_undegraded":"2024-03-26T03:04:20.024985+0000","last_fullsized":"2024-03-26T03:04:20.024985+0000","mapping_epoch":141,"log_start":"100'2290","ondisk_log_start":"100'2290","created":111,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'7589","last_scrub_stamp":"2024-03-26T03:03:58.513121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:58.513121+0000","objects_scrubbed":1015,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:00.434899+0000","scrub_duration":549,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":752,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432424+0000","last_change":"2024-03-26T03:04:01.275984+0000","last_active":"2024-03-26T03:04:16.432424+0000","last_peered":"2024-03-26T03:04:16.432424+0000","last_clean":"2024-03-26T03:04:16.432424+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:04:16.432424+0000","last_undegraded":"2024-03-26T03:04:16.432424+0000","last_fullsized":"2024-03-26T03:04:16.432424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.275846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.275846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:19.631705+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430137+0000","last_change":"2024-03-26T03:01:39.191861+0000","last_active":"2024-03-26T03:04:16.430137+0000","last_peered":"2024-03-26T03:04:16.430137+0000","last_clean":"2024-03-26T03:04:16.430137+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:04:16.430137+0000","last_undegraded":"2024-03-26T03:04:16.430137+0000","last_fullsized":"2024-03-26T03:04:16.430137+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:39.191709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:39.191709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431354+0000","last_change":"2024-03-26T03:04:04.517568+0000","last_active":"2024-03-26T03:04:16.431354+0000","last_peered":"2024-03-26T03:04:16.431354+0000","last_clean":"2024-03-26T03:04:16.431354+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:04:16.431354+0000","last_undegraded":"2024-03-26T03:04:16.431354+0000","last_fullsized":"2024-03-26T03:04:16.431354+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.517463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.517463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:26.130310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":230,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423474+0000","last_change":"2024-03-26T03:01:45.305022+0000","last_active":"2024-03-26T03:04:16.423474+0000","last_peered":"2024-03-26T03:04:16.423474+0000","last_clean":"2024-03-26T03:04:16.423474+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:04:16.423474+0000","last_undegraded":"2024-03-26T03:04:16.423474+0000","last_fullsized":"2024-03-26T03:04:16.423474+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:01:45.304877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:45.304877+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:18.154800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"160'8107","reported_seq":17727,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.930681+0000","last_change":"2024-03-26T03:03:48.153764+0000","last_active":"2024-03-26T03:04:18.930681+0000","last_peered":"2024-03-26T03:04:18.930681+0000","last_clean":"2024-03-26T03:04:18.930681+0000","last_became_active":"2024-03-26T03:03:46.971553+0000","last_became_peered":"2024-03-26T03:03:46.971553+0000","last_unstale":"2024-03-26T03:04:18.930681+0000","last_undegraded":"2024-03-26T03:04:18.930681+0000","last_fullsized":"2024-03-26T03:04:18.930681+0000","mapping_epoch":147,"log_start":"99'701","ondisk_log_start":"99'701","created":111,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"149'7192","last_scrub_stamp":"2024-03-26T03:03:48.153465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:48.153465+0000","objects_scrubbed":1231,"log_size":7406,"log_dups_size":0,"ondisk_log_size":7406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:54.256390+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":921,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432652+0000","last_change":"2024-03-26T03:04:05.990912+0000","last_active":"2024-03-26T03:04:16.432652+0000","last_peered":"2024-03-26T03:04:16.432652+0000","last_clean":"2024-03-26T03:04:16.432652+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:04:16.432652+0000","last_undegraded":"2024-03-26T03:04:16.432652+0000","last_fullsized":"2024-03-26T03:04:16.432652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:05.990787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.990787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:08.333153+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432868+0000","last_change":"2024-03-26T03:01:42.231002+0000","last_active":"2024-03-26T03:04:16.432868+0000","last_peered":"2024-03-26T03:04:16.432868+0000","last_clean":"2024-03-26T03:04:16.432868+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:04:16.432868+0000","last_undegraded":"2024-03-26T03:04:16.432868+0000","last_fullsized":"2024-03-26T03:04:16.432868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:42.230855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.230855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429453+0000","last_change":"2024-03-26T03:01:50.265230+0000","last_active":"2024-03-26T03:04:16.429453+0000","last_peered":"2024-03-26T03:04:16.429453+0000","last_clean":"2024-03-26T03:04:16.429453+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:04:16.429453+0000","last_undegraded":"2024-03-26T03:04:16.429453+0000","last_fullsized":"2024-03-26T03:04:16.429453+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:50.265114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.265114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":256,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422156+0000","last_change":"2024-03-26T03:03:34.444543+0000","last_active":"2024-03-26T03:04:16.422156+0000","last_peered":"2024-03-26T03:04:16.422156+0000","last_clean":"2024-03-26T03:04:16.422156+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:04:16.422156+0000","last_undegraded":"2024-03-26T03:04:16.422156+0000","last_fullsized":"2024-03-26T03:04:16.422156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:03:34.444409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:34.444409+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:56.613497+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423346+0000","last_change":"2024-03-26T03:00:52.383684+0000","last_active":"2024-03-26T03:04:16.423346+0000","last_peered":"2024-03-26T03:04:16.423346+0000","last_clean":"2024-03-26T03:04:16.423346+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:04:16.423346+0000","last_undegraded":"2024-03-26T03:04:16.423346+0000","last_fullsized":"2024-03-26T03:04:16.423346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.383607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.383607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423255+0000","last_change":"2024-03-26T03:03:53.357848+0000","last_active":"2024-03-26T03:04:16.423255+0000","last_peered":"2024-03-26T03:04:16.423255+0000","last_clean":"2024-03-26T03:04:16.423255+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:04:16.423255+0000","last_undegraded":"2024-03-26T03:04:16.423255+0000","last_fullsized":"2024-03-26T03:04:16.423255+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:07.568573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422662+0000","last_change":"2024-03-26T03:01:49.896560+0000","last_active":"2024-03-26T03:04:16.422662+0000","last_peered":"2024-03-26T03:04:16.422662+0000","last_clean":"2024-03-26T03:04:16.422662+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:04:16.422662+0000","last_undegraded":"2024-03-26T03:04:16.422662+0000","last_fullsized":"2024-03-26T03:04:16.422662+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":252,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424780+0000","last_change":"2024-03-26T03:03:47.301632+0000","last_active":"2024-03-26T03:04:16.424780+0000","last_peered":"2024-03-26T03:04:16.424780+0000","last_clean":"2024-03-26T03:04:16.424780+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:04:16.424780+0000","last_undegraded":"2024-03-26T03:04:16.424780+0000","last_fullsized":"2024-03-26T03:04:16.424780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:56.599247+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423067+0000","last_change":"2024-03-26T03:03:40.345342+0000","last_active":"2024-03-26T03:04:16.423067+0000","last_peered":"2024-03-26T03:04:16.423067+0000","last_clean":"2024-03-26T03:04:16.423067+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:04:16.423067+0000","last_undegraded":"2024-03-26T03:04:16.423067+0000","last_fullsized":"2024-03-26T03:04:16.423067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:40.345078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:40.345078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:50.713195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424139+0000","last_change":"2024-03-26T03:04:03.635491+0000","last_active":"2024-03-26T03:04:16.424139+0000","last_peered":"2024-03-26T03:04:16.424139+0000","last_clean":"2024-03-26T03:04:16.424139+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:04:16.424139+0000","last_undegraded":"2024-03-26T03:04:16.424139+0000","last_fullsized":"2024-03-26T03:04:16.424139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:03.634773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:03.634773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:09.151370+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429424+0000","last_change":"2024-03-26T03:01:34.889068+0000","last_active":"2024-03-26T03:04:16.429424+0000","last_peered":"2024-03-26T03:04:16.429424+0000","last_clean":"2024-03-26T03:04:16.429424+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:04:16.429424+0000","last_undegraded":"2024-03-26T03:04:16.429424+0000","last_fullsized":"2024-03-26T03:04:16.429424+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.888991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.888991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:16.792078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":242,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431384+0000","last_change":"2024-03-26T03:01:34.386450+0000","last_active":"2024-03-26T03:04:16.431384+0000","last_peered":"2024-03-26T03:04:16.431384+0000","last_clean":"2024-03-26T03:04:16.431384+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:04:16.431384+0000","last_undegraded":"2024-03-26T03:04:16.431384+0000","last_fullsized":"2024-03-26T03:04:16.431384+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:01:34.386302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.386302+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423460+0000","last_change":"2024-03-26T03:03:53.661347+0000","last_active":"2024-03-26T03:04:16.423460+0000","last_peered":"2024-03-26T03:04:16.423460+0000","last_clean":"2024-03-26T03:04:16.423460+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:04:16.423460+0000","last_undegraded":"2024-03-26T03:04:16.423460+0000","last_fullsized":"2024-03-26T03:04:16.423460+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.661258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.661258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:57.457915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428702+0000","last_change":"2024-03-26T03:01:41.862376+0000","last_active":"2024-03-26T03:04:16.428702+0000","last_peered":"2024-03-26T03:04:16.428702+0000","last_clean":"2024-03-26T03:04:16.428702+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:04:16.428702+0000","last_undegraded":"2024-03-26T03:04:16.428702+0000","last_fullsized":"2024-03-26T03:04:16.428702+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:41.862250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:01:41.862250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425366+0000","last_change":"2024-03-26T03:00:49.420677+0000","last_active":"2024-03-26T03:04:16.425366+0000","last_peered":"2024-03-26T03:04:16.425366+0000","last_clean":"2024-03-26T03:04:16.425366+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:04:16.425366+0000","last_undegraded":"2024-03-26T03:04:16.425366+0000","last_fullsized":"2024-03-26T03:04:16.425366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:00:49.420583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:49.420583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":248,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430329+0000","last_change":"2024-03-26T03:03:51.933569+0000","last_active":"2024-03-26T03:04:16.430329+0000","last_peered":"2024-03-26T03:04:16.430329+0000","last_clean":"2024-03-26T03:04:16.430329+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:04:16.430329+0000","last_undegraded":"2024-03-26T03:04:16.430329+0000","last_fullsized":"2024-03-26T03:04:16.430329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:03:51.933431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:51.933431+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:07.287564+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429922+0000","last_change":"2024-03-26T03:01:46.312995+0000","last_active":"2024-03-26T03:04:16.429922+0000","last_peered":"2024-03-26T03:04:16.429922+0000","last_clean":"2024-03-26T03:04:16.429922+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:04:16.429922+0000","last_undegraded":"2024-03-26T03:04:16.429922+0000","last_fullsized":"2024-03-26T03:04:16.429922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:01:46.312845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:46.312845+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":289,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430280+0000","last_change":"2024-03-26T03:04:02.892852+0000","last_active":"2024-03-26T03:04:16.430280+0000","last_peered":"2024-03-26T03:04:16.430280+0000","last_clean":"2024-03-26T03:04:16.430280+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:04:16.430280+0000","last_undegraded":"2024-03-26T03:04:16.430280+0000","last_fullsized":"2024-03-26T03:04:16.430280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:02.892717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.892717+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:09.499962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.433164+0000","last_change":"2024-03-26T03:01:54.322573+0000","last_active":"2024-03-26T03:04:16.433164+0000","last_peered":"2024-03-26T03:04:16.433164+0000","last_clean":"2024-03-26T03:04:16.433164+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:04:16.433164+0000","last_undegraded":"2024-03-26T03:04:16.433164+0000","last_fullsized":"2024-03-26T03:04:16.433164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:54.322450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.322450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":244,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428587+0000","last_change":"2024-03-26T03:04:04.095435+0000","last_active":"2024-03-26T03:04:16.428587+0000","last_peered":"2024-03-26T03:04:16.428587+0000","last_clean":"2024-03-26T03:04:16.428587+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:04:16.428587+0000","last_undegraded":"2024-03-26T03:04:16.428587+0000","last_fullsized":"2024-03-26T03:04:16.428587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:04:04.095182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.095182+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:33.443816+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"160'8440","reported_seq":16084,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.986211+0000","last_change":"2024-03-26T03:03:49.285019+0000","last_active":"2024-03-26T03:04:18.986211+0000","last_peered":"2024-03-26T03:04:18.986211+0000","last_clean":"2024-03-26T03:04:18.986211+0000","last_became_active":"2024-03-26T03:02:52.287145+0000","last_became_peered":"2024-03-26T03:02:52.287145+0000","last_unstale":"2024-03-26T03:04:18.986211+0000","last_undegraded":"2024-03-26T03:04:18.986211+0000","last_fullsized":"2024-03-26T03:04:18.986211+0000","mapping_epoch":131,"log_start":"99'1363","ondisk_log_start":"99'1363","created":111,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":5,"last_scrub":"150'7426","last_scrub_stamp":"2024-03-26T03:03:49.284870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:49.284870+0000","objects_scrubbed":1306,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:18.995908+0000","scrub_duration":399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18677760,"num_objects":285,"num_object_clones":0,"num_object_copies":855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425265+0000","last_change":"2024-03-26T03:01:34.770522+0000","last_active":"2024-03-26T03:04:16.425265+0000","last_peered":"2024-03-26T03:04:16.425265+0000","last_clean":"2024-03-26T03:04:16.425265+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:04:16.425265+0000","last_undegraded":"2024-03-26T03:04:16.425265+0000","last_fullsized":"2024-03-26T03:04:16.425265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.770057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.770057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429424+0000","last_change":"2024-03-26T03:01:32.215005+0000","last_active":"2024-03-26T03:04:16.429424+0000","last_peered":"2024-03-26T03:04:16.429424+0000","last_clean":"2024-03-26T03:04:16.429424+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:04:16.429424+0000","last_undegraded":"2024-03-26T03:04:16.429424+0000","last_fullsized":"2024-03-26T03:04:16.429424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.214884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.214884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430059+0000","last_change":"2024-03-26T03:00:29.597498+0000","last_active":"2024-03-26T03:04:16.430059+0000","last_peered":"2024-03-26T03:04:16.430059+0000","last_clean":"2024-03-26T03:04:16.430059+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:04:16.430059+0000","last_undegraded":"2024-03-26T03:04:16.430059+0000","last_fullsized":"2024-03-26T03:04:16.430059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:29.597404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:29.597404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:16.791987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"150'23","reported_seq":254,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425678+0000","last_change":"2024-03-26T03:03:50.334581+0000","last_active":"2024-03-26T03:04:16.425678+0000","last_peered":"2024-03-26T03:04:16.425678+0000","last_clean":"2024-03-26T03:04:16.425678+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:04:16.425678+0000","last_undegraded":"2024-03-26T03:04:16.425678+0000","last_fullsized":"2024-03-26T03:04:16.425678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:15.704178+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"160'8322","reported_seq":15541,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.877994+0000","last_change":"2024-03-26T03:03:57.444452+0000","last_active":"2024-03-26T03:04:19.877994+0000","last_peered":"2024-03-26T03:04:19.877994+0000","last_clean":"2024-03-26T03:04:19.877994+0000","last_became_active":"2024-03-26T03:02:53.163455+0000","last_became_peered":"2024-03-26T03:02:53.163455+0000","last_unstale":"2024-03-26T03:04:19.877994+0000","last_undegraded":"2024-03-26T03:04:19.877994+0000","last_fullsized":"2024-03-26T03:04:19.877994+0000","mapping_epoch":132,"log_start":"100'2326","ondisk_log_start":"100'2326","created":111,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'7586","last_scrub_stamp":"2024-03-26T03:03:57.444059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:57.444059+0000","objects_scrubbed":1036,"log_size":5996,"log_dups_size":0,"ondisk_log_size":5996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:04.458720+0000","scrub_duration":476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428850+0000","last_change":"2024-03-26T03:01:44.279749+0000","last_active":"2024-03-26T03:04:16.428850+0000","last_peered":"2024-03-26T03:04:16.428850+0000","last_clean":"2024-03-26T03:04:16.428850+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:04:16.428850+0000","last_undegraded":"2024-03-26T03:04:16.428850+0000","last_fullsized":"2024-03-26T03:04:16.428850+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:44.279630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:44.279630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429074+0000","last_change":"2024-03-26T03:04:04.867638+0000","last_active":"2024-03-26T03:04:16.429074+0000","last_peered":"2024-03-26T03:04:16.429074+0000","last_clean":"2024-03-26T03:04:16.429074+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:04:16.429074+0000","last_undegraded":"2024-03-26T03:04:16.429074+0000","last_fullsized":"2024-03-26T03:04:16.429074+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.867528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.867528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:08.297967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.433229+0000","last_change":"2024-03-26T03:01:33.180277+0000","last_active":"2024-03-26T03:04:16.433229+0000","last_peered":"2024-03-26T03:04:16.433229+0000","last_clean":"2024-03-26T03:04:16.433229+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:04:16.433229+0000","last_undegraded":"2024-03-26T03:04:16.433229+0000","last_fullsized":"2024-03-26T03:04:16.433229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.180027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.180027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406394+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422554+0000","last_change":"2024-03-26T03:01:37.776671+0000","last_active":"2024-03-26T03:04:16.422554+0000","last_peered":"2024-03-26T03:04:16.422554+0000","last_clean":"2024-03-26T03:04:16.422554+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:04:16.422554+0000","last_undegraded":"2024-03-26T03:04:16.422554+0000","last_fullsized":"2024-03-26T03:04:16.422554+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"160'8378","reported_seq":13395,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.744943+0000","last_change":"2024-03-26T03:03:45.916231+0000","last_active":"2024-03-26T03:04:19.744943+0000","last_peered":"2024-03-26T03:04:19.744943+0000","last_clean":"2024-03-26T03:04:19.744943+0000","last_became_active":"2024-03-26T03:03:44.962959+0000","last_became_peered":"2024-03-26T03:03:44.962959+0000","last_unstale":"2024-03-26T03:04:19.744943+0000","last_undegraded":"2024-03-26T03:04:19.744943+0000","last_fullsized":"2024-03-26T03:04:19.744943+0000","mapping_epoch":145,"log_start":"103'3348","ondisk_log_start":"103'3348","created":111,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'7317","last_scrub_stamp":"2024-03-26T03:03:45.915842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:45.915842+0000","objects_scrubbed":1341,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.723936+0000","scrub_duration":591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1072,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425122+0000","last_change":"2024-03-26T03:03:54.412437+0000","last_active":"2024-03-26T03:04:16.425122+0000","last_peered":"2024-03-26T03:04:16.425122+0000","last_clean":"2024-03-26T03:04:16.425122+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:04:16.425122+0000","last_undegraded":"2024-03-26T03:04:16.425122+0000","last_fullsized":"2024-03-26T03:04:16.425122+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:23.413024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424559+0000","last_change":"2024-03-26T03:03:45.582599+0000","last_active":"2024-03-26T03:04:16.424559+0000","last_peered":"2024-03-26T03:04:16.424559+0000","last_clean":"2024-03-26T03:04:16.424559+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:04:16.424559+0000","last_undegraded":"2024-03-26T03:04:16.424559+0000","last_fullsized":"2024-03-26T03:04:16.424559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:45.582463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:45.582463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:10.559400+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423868+0000","last_change":"2024-03-26T03:04:02.578194+0000","last_active":"2024-03-26T03:04:16.423868+0000","last_peered":"2024-03-26T03:04:16.423868+0000","last_clean":"2024-03-26T03:04:16.423868+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:04:16.423868+0000","last_undegraded":"2024-03-26T03:04:16.423868+0000","last_fullsized":"2024-03-26T03:04:16.423868+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.578096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.831727+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423390+0000","last_change":"2024-03-26T03:01:42.243460+0000","last_active":"2024-03-26T03:04:16.423390+0000","last_peered":"2024-03-26T03:04:16.423390+0000","last_clean":"2024-03-26T03:04:16.423390+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:04:16.423390+0000","last_undegraded":"2024-03-26T03:04:16.423390+0000","last_fullsized":"2024-03-26T03:04:16.423390+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:01:42.243318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.243318+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:18.154832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.11","version":"160'8305","reported_seq":13178,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.913870+0000","last_change":"2024-03-26T03:04:05.674928+0000","last_active":"2024-03-26T03:04:18.913870+0000","last_peered":"2024-03-26T03:04:18.913870+0000","last_clean":"2024-03-26T03:04:18.913870+0000","last_became_active":"2024-03-26T03:02:27.284219+0000","last_became_peered":"2024-03-26T03:02:27.284219+0000","last_unstale":"2024-03-26T03:04:18.913870+0000","last_undegraded":"2024-03-26T03:04:18.913870+0000","last_fullsized":"2024-03-26T03:04:18.913870+0000","mapping_epoch":121,"log_start":"102'3067","ondisk_log_start":"102'3067","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'7788","last_scrub_stamp":"2024-03-26T03:04:05.674754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.674754+0000","objects_scrubbed":833,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:06.131179+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":314,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":519,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.426702+0000","last_change":"2024-03-26T03:00:44.275620+0000","last_active":"2024-03-26T03:04:16.426702+0000","last_peered":"2024-03-26T03:04:16.426702+0000","last_clean":"2024-03-26T03:04:16.426702+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:04:16.426702+0000","last_undegraded":"2024-03-26T03:04:16.426702+0000","last_fullsized":"2024-03-26T03:04:16.426702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:44.275533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:44.275533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422875+0000","last_change":"2024-03-26T03:03:44.286188+0000","last_active":"2024-03-26T03:04:16.422875+0000","last_peered":"2024-03-26T03:04:16.422875+0000","last_clean":"2024-03-26T03:04:16.422875+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:04:16.422875+0000","last_undegraded":"2024-03-26T03:04:16.422875+0000","last_fullsized":"2024-03-26T03:04:16.422875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:03.152337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429815+0000","last_change":"2024-03-26T03:01:33.891331+0000","last_active":"2024-03-26T03:04:16.429815+0000","last_peered":"2024-03-26T03:04:16.429815+0000","last_clean":"2024-03-26T03:04:16.429815+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:04:16.429815+0000","last_undegraded":"2024-03-26T03:04:16.429815+0000","last_fullsized":"2024-03-26T03:04:16.429815+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.891247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.891247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:16.792103+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424445+0000","last_change":"2024-03-26T03:01:40.604091+0000","last_active":"2024-03-26T03:04:16.424445+0000","last_peered":"2024-03-26T03:04:16.424445+0000","last_clean":"2024-03-26T03:04:16.424445+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:04:16.424445+0000","last_undegraded":"2024-03-26T03:04:16.424445+0000","last_fullsized":"2024-03-26T03:04:16.424445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-26T03:01:40.603834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.603834+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"160'8408","reported_seq":13470,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.870197+0000","last_change":"2024-03-26T03:03:31.833200+0000","last_active":"2024-03-26T03:04:18.870197+0000","last_peered":"2024-03-26T03:04:18.870197+0000","last_clean":"2024-03-26T03:04:18.870197+0000","last_became_active":"2024-03-26T03:02:38.934091+0000","last_became_peered":"2024-03-26T03:02:38.934091+0000","last_unstale":"2024-03-26T03:04:18.870197+0000","last_undegraded":"2024-03-26T03:04:18.870197+0000","last_fullsized":"2024-03-26T03:04:18.870197+0000","mapping_epoch":125,"log_start":"100'1887","ondisk_log_start":"100'1887","created":111,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":5,"last_scrub":"142'6947","last_scrub_stamp":"2024-03-26T03:03:31.832793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:31.832793+0000","objects_scrubbed":1782,"log_size":6521,"log_dups_size":0,"ondisk_log_size":6521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:59.087361+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424071+0000","last_change":"2024-03-26T03:01:53.315527+0000","last_active":"2024-03-26T03:04:16.424071+0000","last_peered":"2024-03-26T03:04:16.424071+0000","last_clean":"2024-03-26T03:04:16.424071+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:04:16.424071+0000","last_undegraded":"2024-03-26T03:04:16.424071+0000","last_fullsized":"2024-03-26T03:04:16.424071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:01:53.315442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.315442+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:18.154920+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":278,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.427223+0000","last_change":"2024-03-26T03:00:32.418258+0000","last_active":"2024-03-26T03:04:16.427223+0000","last_peered":"2024-03-26T03:04:16.427223+0000","last_clean":"2024-03-26T03:04:16.427223+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:04:16.427223+0000","last_undegraded":"2024-03-26T03:04:16.427223+0000","last_fullsized":"2024-03-26T03:04:16.427223+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:00:32.418162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.418162+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428270+0000","last_change":"2024-03-26T03:00:45.281332+0000","last_active":"2024-03-26T03:04:16.428270+0000","last_peered":"2024-03-26T03:04:16.428270+0000","last_clean":"2024-03-26T03:04:16.428270+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:04:16.428270+0000","last_undegraded":"2024-03-26T03:04:16.428270+0000","last_fullsized":"2024-03-26T03:04:16.428270+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.281248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.281248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":61,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431440+0000","last_change":"2024-03-26T03:04:14.416461+0000","last_active":"2024-03-26T03:04:16.431440+0000","last_peered":"2024-03-26T03:04:16.431440+0000","last_clean":"2024-03-26T03:04:16.431440+0000","last_became_active":"2024-03-26T03:04:14.415940+0000","last_became_peered":"2024-03-26T03:04:14.415940+0000","last_unstale":"2024-03-26T03:04:16.431440+0000","last_undegraded":"2024-03-26T03:04:16.431440+0000","last_fullsized":"2024-03-26T03:04:16.431440+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:24.578052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.17","version":"160'8479","reported_seq":18363,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.987884+0000","last_change":"2024-03-26T03:03:54.528437+0000","last_active":"2024-03-26T03:04:18.987884+0000","last_peered":"2024-03-26T03:04:18.987884+0000","last_clean":"2024-03-26T03:04:18.987884+0000","last_became_active":"2024-03-26T03:03:54.092641+0000","last_became_peered":"2024-03-26T03:03:54.092641+0000","last_unstale":"2024-03-26T03:04:18.987884+0000","last_undegraded":"2024-03-26T03:04:18.987884+0000","last_fullsized":"2024-03-26T03:04:18.987884+0000","mapping_epoch":152,"log_start":"103'3383","ondisk_log_start":"103'3383","created":111,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"153'7624","last_scrub_stamp":"2024-03-26T03:03:54.528093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.528093+0000","objects_scrubbed":1193,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:08.649810+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.427974+0000","last_change":"2024-03-26T03:01:39.883186+0000","last_active":"2024-03-26T03:04:16.427974+0000","last_peered":"2024-03-26T03:04:16.427974+0000","last_clean":"2024-03-26T03:04:16.427974+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:04:16.427974+0000","last_undegraded":"2024-03-26T03:04:16.427974+0000","last_fullsized":"2024-03-26T03:04:16.427974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:39.883059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:39.883059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":285,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432619+0000","last_change":"2024-03-26T03:03:55.245804+0000","last_active":"2024-03-26T03:04:16.432619+0000","last_peered":"2024-03-26T03:04:16.432619+0000","last_clean":"2024-03-26T03:04:16.432619+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:04:16.432619+0000","last_undegraded":"2024-03-26T03:04:16.432619+0000","last_fullsized":"2024-03-26T03:04:16.432619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:03:55.245713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:55.245713+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:24.948940+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432512+0000","last_change":"2024-03-26T03:01:49.448360+0000","last_active":"2024-03-26T03:04:16.432512+0000","last_peered":"2024-03-26T03:04:16.432512+0000","last_clean":"2024-03-26T03:04:16.432512+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:04:16.432512+0000","last_undegraded":"2024-03-26T03:04:16.432512+0000","last_fullsized":"2024-03-26T03:04:16.432512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.448224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.448224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406549+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"151'29","reported_seq":263,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428829+0000","last_change":"2024-03-26T03:03:52.392314+0000","last_active":"2024-03-26T03:04:16.428829+0000","last_peered":"2024-03-26T03:04:16.428829+0000","last_clean":"2024-03-26T03:04:16.428829+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:04:16.428829+0000","last_undegraded":"2024-03-26T03:04:16.428829+0000","last_fullsized":"2024-03-26T03:04:16.428829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:15.632896+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"160'8259","reported_seq":17862,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.619121+0000","last_change":"2024-03-26T03:03:46.596643+0000","last_active":"2024-03-26T03:04:19.619121+0000","last_peered":"2024-03-26T03:04:19.619121+0000","last_clean":"2024-03-26T03:04:19.619121+0000","last_became_active":"2024-03-26T03:03:17.295355+0000","last_became_peered":"2024-03-26T03:03:17.295355+0000","last_unstale":"2024-03-26T03:04:19.619121+0000","last_undegraded":"2024-03-26T03:04:19.619121+0000","last_fullsized":"2024-03-26T03:04:19.619121+0000","mapping_epoch":138,"log_start":"103'3168","ondisk_log_start":"103'3168","created":111,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'7227","last_scrub_stamp":"2024-03-26T03:03:46.596395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:03:46.596395+0000","objects_scrubbed":1335,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:04.645129+0000","scrub_duration":297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726336,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429239+0000","last_change":"2024-03-26T03:01:34.228140+0000","last_active":"2024-03-26T03:04:16.429239+0000","last_peered":"2024-03-26T03:04:16.429239+0000","last_clean":"2024-03-26T03:04:16.429239+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:04:16.429239+0000","last_undegraded":"2024-03-26T03:04:16.429239+0000","last_fullsized":"2024-03-26T03:04:16.429239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.227979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.227979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.433047+0000","last_change":"2024-03-26T03:03:56.206476+0000","last_active":"2024-03-26T03:04:16.433047+0000","last_peered":"2024-03-26T03:04:16.433047+0000","last_clean":"2024-03-26T03:04:16.433047+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:04:16.433047+0000","last_undegraded":"2024-03-26T03:04:16.433047+0000","last_fullsized":"2024-03-26T03:04:16.433047+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:56.206361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:56.206361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:15.870388+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429166+0000","last_change":"2024-03-26T03:01:32.933502+0000","last_active":"2024-03-26T03:04:16.429166+0000","last_peered":"2024-03-26T03:04:16.429166+0000","last_clean":"2024-03-26T03:04:16.429166+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:04:16.429166+0000","last_undegraded":"2024-03-26T03:04:16.429166+0000","last_fullsized":"2024-03-26T03:04:16.429166+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.933389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.933389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:16.792228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":254,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425657+0000","last_change":"2024-03-26T03:01:50.876168+0000","last_active":"2024-03-26T03:04:16.425657+0000","last_peered":"2024-03-26T03:04:16.425657+0000","last_clean":"2024-03-26T03:04:16.425657+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:04:16.425657+0000","last_undegraded":"2024-03-26T03:04:16.425657+0000","last_fullsized":"2024-03-26T03:04:16.425657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.15","version":"160'8292","reported_seq":16330,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.004619+0000","last_change":"2024-03-26T03:04:03.643062+0000","last_active":"2024-03-26T03:04:19.004619+0000","last_peered":"2024-03-26T03:04:19.004619+0000","last_clean":"2024-03-26T03:04:19.004619+0000","last_became_active":"2024-03-26T03:04:03.139505+0000","last_became_peered":"2024-03-26T03:04:03.139505+0000","last_unstale":"2024-03-26T03:04:19.004619+0000","last_undegraded":"2024-03-26T03:04:19.004619+0000","last_fullsized":"2024-03-26T03:04:19.004619+0000","mapping_epoch":154,"log_start":"102'3113","ondisk_log_start":"102'3113","created":111,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'7741","last_scrub_stamp":"2024-03-26T03:04:03.642715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:04:03.642715+0000","objects_scrubbed":896,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:27.174393+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":343,"num_object_clones":0,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430970+0000","last_change":"2024-03-26T03:01:52.467228+0000","last_active":"2024-03-26T03:04:16.430970+0000","last_peered":"2024-03-26T03:04:16.430970+0000","last_clean":"2024-03-26T03:04:16.430970+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:04:16.430970+0000","last_undegraded":"2024-03-26T03:04:16.430970+0000","last_fullsized":"2024-03-26T03:04:16.430970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:52.467134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.467134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423489+0000","last_change":"2024-03-26T03:01:38.802880+0000","last_active":"2024-03-26T03:04:16.423489+0000","last_peered":"2024-03-26T03:04:16.423489+0000","last_clean":"2024-03-26T03:04:16.423489+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:04:16.423489+0000","last_undegraded":"2024-03-26T03:04:16.423489+0000","last_fullsized":"2024-03-26T03:04:16.423489+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:38.802745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:38.802745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432467+0000","last_change":"2024-03-26T03:01:46.408351+0000","last_active":"2024-03-26T03:04:16.432467+0000","last_peered":"2024-03-26T03:04:16.432467+0000","last_clean":"2024-03-26T03:04:16.432467+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:04:16.432467+0000","last_undegraded":"2024-03-26T03:04:16.432467+0000","last_fullsized":"2024-03-26T03:04:16.432467+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:01:46.408202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:46.408202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":257,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428289+0000","last_change":"2024-03-26T03:03:59.969483+0000","last_active":"2024-03-26T03:04:16.428289+0000","last_peered":"2024-03-26T03:04:16.428289+0000","last_clean":"2024-03-26T03:04:16.428289+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:04:16.428289+0000","last_undegraded":"2024-03-26T03:04:16.428289+0000","last_fullsized":"2024-03-26T03:04:16.428289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:03:59.969202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:59.969202+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:24.335149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"160'11964","reported_seq":12106,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.696956+0000","last_change":"2024-03-26T03:00:55.439899+0000","last_active":"2024-03-26T03:04:19.696956+0000","last_peered":"2024-03-26T03:04:19.696956+0000","last_clean":"2024-03-26T03:04:19.696956+0000","last_became_active":"2024-03-26T03:00:55.439557+0000","last_became_peered":"2024-03-26T03:00:55.439557+0000","last_unstale":"2024-03-26T03:04:19.696956+0000","last_undegraded":"2024-03-26T03:04:19.696956+0000","last_fullsized":"2024-03-26T03:04:19.696956+0000","mapping_epoch":96,"log_start":"135'7523","ondisk_log_start":"135'7523","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:22.671364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":564,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":400896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425071+0000","last_change":"2024-03-26T03:00:35.424045+0000","last_active":"2024-03-26T03:04:16.425071+0000","last_peered":"2024-03-26T03:04:16.425071+0000","last_clean":"2024-03-26T03:04:16.425071+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:04:16.425071+0000","last_undegraded":"2024-03-26T03:04:16.425071+0000","last_fullsized":"2024-03-26T03:04:16.425071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.423904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.423904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":230,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423900+0000","last_change":"2024-03-26T03:01:43.256768+0000","last_active":"2024-03-26T03:04:16.423900+0000","last_peered":"2024-03-26T03:04:16.423900+0000","last_clean":"2024-03-26T03:04:16.423900+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:04:16.423900+0000","last_undegraded":"2024-03-26T03:04:16.423900+0000","last_fullsized":"2024-03-26T03:04:16.423900+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:01:43.256605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:43.256605+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:18.154864+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":285,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422285+0000","last_change":"2024-03-26T03:03:50.380060+0000","last_active":"2024-03-26T03:04:16.422285+0000","last_peered":"2024-03-26T03:04:16.422285+0000","last_clean":"2024-03-26T03:04:16.422285+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:04:16.422285+0000","last_undegraded":"2024-03-26T03:04:16.422285+0000","last_fullsized":"2024-03-26T03:04:16.422285+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:04.773506+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432557+0000","last_change":"2024-03-26T03:01:35.410181+0000","last_active":"2024-03-26T03:04:16.432557+0000","last_peered":"2024-03-26T03:04:16.432557+0000","last_clean":"2024-03-26T03:04:16.432557+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:04:16.432557+0000","last_undegraded":"2024-03-26T03:04:16.432557+0000","last_fullsized":"2024-03-26T03:04:16.432557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:35.410060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:35.410060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"160'8310","reported_seq":8490,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.899427+0000","last_change":"2024-03-26T03:01:51.817423+0000","last_active":"2024-03-26T03:04:18.899427+0000","last_peered":"2024-03-26T03:04:18.899427+0000","last_clean":"2024-03-26T03:04:18.899427+0000","last_became_active":"2024-03-26T03:01:51.288990+0000","last_became_peered":"2024-03-26T03:01:51.288990+0000","last_unstale":"2024-03-26T03:04:18.899427+0000","last_undegraded":"2024-03-26T03:04:18.899427+0000","last_fullsized":"2024-03-26T03:04:18.899427+0000","mapping_epoch":96,"log_start":"100'1810","ondisk_log_start":"100'1810","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4725","last_scrub_stamp":"2024-03-26T03:01:51.817210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:51.817210+0000","objects_scrubbed":2411,"log_size":6500,"log_dups_size":0,"ondisk_log_size":6500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":47616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.426797+0000","last_change":"2024-03-26T03:03:53.959327+0000","last_active":"2024-03-26T03:04:16.426797+0000","last_peered":"2024-03-26T03:04:16.426797+0000","last_clean":"2024-03-26T03:04:16.426797+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:04:16.426797+0000","last_undegraded":"2024-03-26T03:04:16.426797+0000","last_fullsized":"2024-03-26T03:04:16.426797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.959237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:03.079778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423817+0000","last_change":"2024-03-26T03:00:27.327214+0000","last_active":"2024-03-26T03:04:16.423817+0000","last_peered":"2024-03-26T03:04:16.423817+0000","last_clean":"2024-03-26T03:04:16.423817+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:04:16.423817+0000","last_undegraded":"2024-03-26T03:04:16.423817+0000","last_fullsized":"2024-03-26T03:04:16.423817+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:27.327116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:27.327116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"92'22","reported_seq":242,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424513+0000","last_change":"2024-03-26T03:00:42.399658+0000","last_active":"2024-03-26T03:04:16.424513+0000","last_peered":"2024-03-26T03:04:16.424513+0000","last_clean":"2024-03-26T03:04:16.424513+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:04:16.424513+0000","last_undegraded":"2024-03-26T03:04:16.424513+0000","last_fullsized":"2024-03-26T03:04:16.424513+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:00:42.399566+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":289,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428547+0000","last_change":"2024-03-26T03:01:40.859597+0000","last_active":"2024-03-26T03:04:16.428547+0000","last_peered":"2024-03-26T03:04:16.428547+0000","last_clean":"2024-03-26T03:04:16.428547+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:04:16.428547+0000","last_undegraded":"2024-03-26T03:04:16.428547+0000","last_fullsized":"2024-03-26T03:04:16.428547+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:01:40.859466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.859466+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.877265+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"160'8362","reported_seq":8546,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.956584+0000","last_change":"2024-03-26T03:01:52.740255+0000","last_active":"2024-03-26T03:04:18.956584+0000","last_peered":"2024-03-26T03:04:18.956584+0000","last_clean":"2024-03-26T03:04:18.956584+0000","last_became_active":"2024-03-26T03:01:51.303857+0000","last_became_peered":"2024-03-26T03:01:51.303857+0000","last_unstale":"2024-03-26T03:04:18.956584+0000","last_undegraded":"2024-03-26T03:04:18.956584+0000","last_fullsized":"2024-03-26T03:04:18.956584+0000","mapping_epoch":96,"log_start":"99'987","ondisk_log_start":"99'987","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4820","last_scrub_stamp":"2024-03-26T03:01:52.740121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.740121+0000","objects_scrubbed":2415,"log_size":7375,"log_dups_size":0,"ondisk_log_size":7375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3543,"num_write_kb":46656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422970+0000","last_change":"2024-03-26T03:01:33.720224+0000","last_active":"2024-03-26T03:04:16.422970+0000","last_peered":"2024-03-26T03:04:16.422970+0000","last_clean":"2024-03-26T03:04:16.422970+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:04:16.422970+0000","last_undegraded":"2024-03-26T03:04:16.422970+0000","last_fullsized":"2024-03-26T03:04:16.422970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.719795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.719795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424218+0000","last_change":"2024-03-26T03:04:14.420602+0000","last_active":"2024-03-26T03:04:16.424218+0000","last_peered":"2024-03-26T03:04:16.424218+0000","last_clean":"2024-03-26T03:04:16.424218+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:04:16.424218+0000","last_undegraded":"2024-03-26T03:04:16.424218+0000","last_fullsized":"2024-03-26T03:04:16.424218+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:06.319990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:06.319990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:25.788201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":353,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423754+0000","last_change":"2024-03-26T03:04:01.624276+0000","last_active":"2024-03-26T03:04:16.423754+0000","last_peered":"2024-03-26T03:04:16.423754+0000","last_clean":"2024-03-26T03:04:16.423754+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:04:16.423754+0000","last_undegraded":"2024-03-26T03:04:16.423754+0000","last_fullsized":"2024-03-26T03:04:16.423754+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:04:01.623984+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.623984+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:05.016333+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428235+0000","last_change":"2024-03-26T03:04:01.972787+0000","last_active":"2024-03-26T03:04:16.428235+0000","last_peered":"2024-03-26T03:04:16.428235+0000","last_clean":"2024-03-26T03:04:16.428235+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:04:16.428235+0000","last_undegraded":"2024-03-26T03:04:16.428235+0000","last_fullsized":"2024-03-26T03:04:16.428235+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.972671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.972671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:29.228532+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"160'8184","reported_seq":8366,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:20.018474+0000","last_change":"2024-03-26T03:01:53.378883+0000","last_active":"2024-03-26T03:04:20.018474+0000","last_peered":"2024-03-26T03:04:20.018474+0000","last_clean":"2024-03-26T03:04:20.018474+0000","last_became_active":"2024-03-26T03:01:51.301399+0000","last_became_peered":"2024-03-26T03:01:51.301399+0000","last_unstale":"2024-03-26T03:04:20.018474+0000","last_undegraded":"2024-03-26T03:04:20.018474+0000","last_fullsized":"2024-03-26T03:04:20.018474+0000","mapping_epoch":96,"log_start":"100'2256","ondisk_log_start":"100'2256","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4665","last_scrub_stamp":"2024-03-26T03:01:53.378673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.378673+0000","objects_scrubbed":2342,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.877435+0000","scrub_duration":480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":247,"num_object_clones":0,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":45568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431525+0000","last_change":"2024-03-26T03:04:06.566392+0000","last_active":"2024-03-26T03:04:16.431525+0000","last_peered":"2024-03-26T03:04:16.431525+0000","last_clean":"2024-03-26T03:04:16.431525+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:04:16.431525+0000","last_undegraded":"2024-03-26T03:04:16.431525+0000","last_fullsized":"2024-03-26T03:04:16.431525+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:06.566290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:06.566290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:10.335523+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":544,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:17.678378+0000","last_change":"2024-03-26T03:03:57.681467+0000","last_active":"2024-03-26T03:04:17.678378+0000","last_peered":"2024-03-26T03:04:17.678378+0000","last_clean":"2024-03-26T03:04:17.678378+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:04:17.678378+0000","last_undegraded":"2024-03-26T03:04:17.678378+0000","last_fullsized":"2024-03-26T03:04:17.678378+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:03:57.679470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:03:57.679470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:10.270071+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430747+0000","last_change":"2024-03-26T03:01:45.357919+0000","last_active":"2024-03-26T03:04:16.430747+0000","last_peered":"2024-03-26T03:04:16.430747+0000","last_clean":"2024-03-26T03:04:16.430747+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:04:16.430747+0000","last_undegraded":"2024-03-26T03:04:16.430747+0000","last_fullsized":"2024-03-26T03:04:16.430747+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:45.357787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:45.357787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":248,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422808+0000","last_change":"2024-03-26T03:00:23.305418+0000","last_active":"2024-03-26T03:04:16.422808+0000","last_peered":"2024-03-26T03:04:16.422808+0000","last_clean":"2024-03-26T03:04:16.422808+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:04:16.422808+0000","last_undegraded":"2024-03-26T03:04:16.422808+0000","last_fullsized":"2024-03-26T03:04:16.422808+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:00:23.305307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:23.305307+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431785+0000","last_change":"2024-03-26T03:01:44.370403+0000","last_active":"2024-03-26T03:04:16.431785+0000","last_peered":"2024-03-26T03:04:16.431785+0000","last_clean":"2024-03-26T03:04:16.431785+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:04:16.431785+0000","last_undegraded":"2024-03-26T03:04:16.431785+0000","last_fullsized":"2024-03-26T03:04:16.431785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:44.370278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:44.370278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"160'8293","reported_seq":8472,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.962316+0000","last_change":"2024-03-26T03:01:54.158951+0000","last_active":"2024-03-26T03:04:19.962316+0000","last_peered":"2024-03-26T03:04:19.962316+0000","last_clean":"2024-03-26T03:04:19.962316+0000","last_became_active":"2024-03-26T03:01:51.328380+0000","last_became_peered":"2024-03-26T03:01:51.328380+0000","last_unstale":"2024-03-26T03:04:19.962316+0000","last_undegraded":"2024-03-26T03:04:19.962316+0000","last_fullsized":"2024-03-26T03:04:19.962316+0000","mapping_epoch":96,"log_start":"100'2293","ondisk_log_start":"100'2293","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4780","last_scrub_stamp":"2024-03-26T03:01:54.158811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.158811+0000","objects_scrubbed":2374,"log_size":6000,"log_dups_size":0,"ondisk_log_size":6000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.877323+0000","scrub_duration":297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18415616,"num_objects":281,"num_object_clones":0,"num_object_copies":843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":45440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.433481+0000","last_change":"2024-03-26T03:01:47.363671+0000","last_active":"2024-03-26T03:04:16.433481+0000","last_peered":"2024-03-26T03:04:16.433481+0000","last_clean":"2024-03-26T03:04:16.433481+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:04:16.433481+0000","last_undegraded":"2024-03-26T03:04:16.433481+0000","last_fullsized":"2024-03-26T03:04:16.433481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:47.363550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.363550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":222,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425967+0000","last_change":"2024-03-26T03:00:45.454030+0000","last_active":"2024-03-26T03:04:16.425967+0000","last_peered":"2024-03-26T03:04:16.425967+0000","last_clean":"2024-03-26T03:04:16.425967+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:04:16.425967+0000","last_undegraded":"2024-03-26T03:04:16.425967+0000","last_fullsized":"2024-03-26T03:04:16.425967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:00:45.453925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.453925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":226,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432959+0000","last_change":"2024-03-26T03:01:32.148305+0000","last_active":"2024-03-26T03:04:16.432959+0000","last_peered":"2024-03-26T03:04:16.432959+0000","last_clean":"2024-03-26T03:04:16.432959+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:04:16.432959+0000","last_undegraded":"2024-03-26T03:04:16.432959+0000","last_fullsized":"2024-03-26T03:04:16.432959+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.148136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.148136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406357+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":460,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428975+0000","last_change":"2024-03-26T03:01:38.142321+0000","last_active":"2024-03-26T03:04:16.428975+0000","last_peered":"2024-03-26T03:04:16.428975+0000","last_clean":"2024-03-26T03:04:16.428975+0000","last_became_active":"2024-03-26T02:58:47.885504+0000","last_became_peered":"2024-03-26T02:58:47.885504+0000","last_unstale":"2024-03-26T03:04:16.428975+0000","last_undegraded":"2024-03-26T03:04:16.428975+0000","last_fullsized":"2024-03-26T03:04:16.428975+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:01:38.142194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:01:38.142194+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":304,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430467+0000","last_change":"2024-03-26T03:01:42.020043+0000","last_active":"2024-03-26T03:04:16.430467+0000","last_peered":"2024-03-26T03:04:16.430467+0000","last_clean":"2024-03-26T03:04:16.430467+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:04:16.430467+0000","last_undegraded":"2024-03-26T03:04:16.430467+0000","last_fullsized":"2024-03-26T03:04:16.430467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:01:42.019820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.019820+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429701+0000","last_change":"2024-03-26T03:01:48.249675+0000","last_active":"2024-03-26T03:04:16.429701+0000","last_peered":"2024-03-26T03:04:16.429701+0000","last_clean":"2024-03-26T03:04:16.429701+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:04:16.429701+0000","last_undegraded":"2024-03-26T03:04:16.429701+0000","last_fullsized":"2024-03-26T03:04:16.429701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:48.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:48.249567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"160'8218","reported_seq":8396,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.976162+0000","last_change":"2024-03-26T03:01:52.635320+0000","last_active":"2024-03-26T03:04:18.976162+0000","last_peered":"2024-03-26T03:04:18.976162+0000","last_clean":"2024-03-26T03:04:18.976162+0000","last_became_active":"2024-03-26T03:01:51.316580+0000","last_became_peered":"2024-03-26T03:01:51.316580+0000","last_unstale":"2024-03-26T03:04:18.976162+0000","last_undegraded":"2024-03-26T03:04:18.976162+0000","last_fullsized":"2024-03-26T03:04:18.976162+0000","mapping_epoch":96,"log_start":"102'2863","ondisk_log_start":"102'2863","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4738","last_scrub_stamp":"2024-03-26T03:01:52.635130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.635130+0000","objects_scrubbed":2393,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.688527+0000","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":283,"num_object_clones":0,"num_object_copies":849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3480,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424162+0000","last_change":"2024-03-26T03:00:35.391826+0000","last_active":"2024-03-26T03:04:16.424162+0000","last_peered":"2024-03-26T03:04:16.424162+0000","last_clean":"2024-03-26T03:04:16.424162+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:04:16.424162+0000","last_undegraded":"2024-03-26T03:04:16.424162+0000","last_fullsized":"2024-03-26T03:04:16.424162+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.391746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.391746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":220,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.426949+0000","last_change":"2024-03-26T03:00:32.315656+0000","last_active":"2024-03-26T03:04:16.426949+0000","last_peered":"2024-03-26T03:04:16.426949+0000","last_clean":"2024-03-26T03:04:16.426949+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:04:16.426949+0000","last_undegraded":"2024-03-26T03:04:16.426949+0000","last_fullsized":"2024-03-26T03:04:16.426949+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:00:32.315559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428925+0000","last_change":"2024-03-26T03:03:53.328993+0000","last_active":"2024-03-26T03:04:16.428925+0000","last_peered":"2024-03-26T03:04:16.428925+0000","last_clean":"2024-03-26T03:04:16.428925+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:04:16.428925+0000","last_undegraded":"2024-03-26T03:04:16.428925+0000","last_fullsized":"2024-03-26T03:04:16.428925+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.328899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.328899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:09.422895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":235,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428071+0000","last_change":"2024-03-26T03:01:33.900763+0000","last_active":"2024-03-26T03:04:16.428071+0000","last_peered":"2024-03-26T03:04:16.428071+0000","last_clean":"2024-03-26T03:04:16.428071+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:04:16.428071+0000","last_undegraded":"2024-03-26T03:04:16.428071+0000","last_fullsized":"2024-03-26T03:04:16.428071+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:33.900630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.900630+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.433418+0000","last_change":"2024-03-26T03:04:05.281972+0000","last_active":"2024-03-26T03:04:16.433418+0000","last_peered":"2024-03-26T03:04:16.433418+0000","last_clean":"2024-03-26T03:04:16.433418+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:04:16.433418+0000","last_undegraded":"2024-03-26T03:04:16.433418+0000","last_fullsized":"2024-03-26T03:04:16.433418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:05.281823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.281823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:34.500521+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"160'8360","reported_seq":8541,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.941189+0000","last_change":"2024-03-26T03:01:51.998675+0000","last_active":"2024-03-26T03:04:18.941189+0000","last_peered":"2024-03-26T03:04:18.941189+0000","last_clean":"2024-03-26T03:04:18.941189+0000","last_became_active":"2024-03-26T03:01:51.311443+0000","last_became_peered":"2024-03-26T03:01:51.311443+0000","last_unstale":"2024-03-26T03:04:18.941189+0000","last_undegraded":"2024-03-26T03:04:18.941189+0000","last_fullsized":"2024-03-26T03:04:18.941189+0000","mapping_epoch":96,"log_start":"99'1041","ondisk_log_start":"99'1041","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4793","last_scrub_stamp":"2024-03-26T03:01:51.998539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:51.998539+0000","objects_scrubbed":2401,"log_size":7319,"log_dups_size":0,"ondisk_log_size":7319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19988480,"num_objects":305,"num_object_clones":0,"num_object_copies":915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3580,"num_write_kb":47488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423652+0000","last_change":"2024-03-26T03:04:04.660434+0000","last_active":"2024-03-26T03:04:16.423652+0000","last_peered":"2024-03-26T03:04:16.423652+0000","last_clean":"2024-03-26T03:04:16.423652+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:04:16.423652+0000","last_undegraded":"2024-03-26T03:04:16.423652+0000","last_fullsized":"2024-03-26T03:04:16.423652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.660320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.660320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:20.401750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431605+0000","last_change":"2024-03-26T03:00:49.223170+0000","last_active":"2024-03-26T03:04:16.431605+0000","last_peered":"2024-03-26T03:04:16.431605+0000","last_clean":"2024-03-26T03:04:16.431605+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:04:16.431605+0000","last_undegraded":"2024-03-26T03:04:16.431605+0000","last_fullsized":"2024-03-26T03:04:16.431605+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:49.223092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:49.223092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":265,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428155+0000","last_change":"2024-03-26T03:01:47.898281+0000","last_active":"2024-03-26T03:04:16.428155+0000","last_peered":"2024-03-26T03:04:16.428155+0000","last_clean":"2024-03-26T03:04:16.428155+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:04:16.428155+0000","last_undegraded":"2024-03-26T03:04:16.428155+0000","last_fullsized":"2024-03-26T03:04:16.428155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:01:47.898135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.898135+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429511+0000","last_change":"2024-03-26T03:03:49.866282+0000","last_active":"2024-03-26T03:04:16.429511+0000","last_peered":"2024-03-26T03:04:16.429511+0000","last_clean":"2024-03-26T03:04:16.429511+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:04:16.429511+0000","last_undegraded":"2024-03-26T03:04:16.429511+0000","last_fullsized":"2024-03-26T03:04:16.429511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:49.866190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:49.866190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:54.723169+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"160'8319","reported_seq":8503,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.715064+0000","last_change":"2024-03-26T03:01:53.411487+0000","last_active":"2024-03-26T03:04:19.715064+0000","last_peered":"2024-03-26T03:04:19.715064+0000","last_clean":"2024-03-26T03:04:19.715064+0000","last_became_active":"2024-03-26T03:01:51.345923+0000","last_became_peered":"2024-03-26T03:01:51.345923+0000","last_unstale":"2024-03-26T03:04:19.715064+0000","last_undegraded":"2024-03-26T03:04:19.715064+0000","last_fullsized":"2024-03-26T03:04:19.715064+0000","mapping_epoch":96,"log_start":"103'3295","ondisk_log_start":"103'3295","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4741","last_scrub_stamp":"2024-03-26T03:01:53.411373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.411373+0000","objects_scrubbed":2420,"log_size":5024,"log_dups_size":0,"ondisk_log_size":5024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:22.671560+0000","scrub_duration":509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":45120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":200,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422443+0000","last_change":"2024-03-26T03:03:29.432831+0000","last_active":"2024-03-26T03:04:16.422443+0000","last_peered":"2024-03-26T03:04:16.422443+0000","last_clean":"2024-03-26T03:04:16.422443+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:04:16.422443+0000","last_undegraded":"2024-03-26T03:04:16.422443+0000","last_fullsized":"2024-03-26T03:04:16.422443+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:29.432698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:29.432698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:38.308985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423293+0000","last_change":"2024-03-26T03:00:17.286899+0000","last_active":"2024-03-26T03:04:16.423293+0000","last_peered":"2024-03-26T03:04:16.423293+0000","last_clean":"2024-03-26T03:04:16.423293+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:04:16.423293+0000","last_undegraded":"2024-03-26T03:04:16.423293+0000","last_fullsized":"2024-03-26T03:04:16.423293+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:00:17.286796+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":285,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423467+0000","last_change":"2024-03-26T03:01:36.769019+0000","last_active":"2024-03-26T03:04:16.423467+0000","last_peered":"2024-03-26T03:04:16.423467+0000","last_clean":"2024-03-26T03:04:16.423467+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:04:16.423467+0000","last_undegraded":"2024-03-26T03:04:16.423467+0000","last_fullsized":"2024-03-26T03:04:16.423467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":204,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422383+0000","last_change":"2024-03-26T03:01:41.237613+0000","last_active":"2024-03-26T03:04:16.422383+0000","last_peered":"2024-03-26T03:04:16.422383+0000","last_clean":"2024-03-26T03:04:16.422383+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:04:16.422383+0000","last_undegraded":"2024-03-26T03:04:16.422383+0000","last_fullsized":"2024-03-26T03:04:16.422383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:41.236598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:41.236598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:18.154892+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"160'8383","reported_seq":8562,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.697251+0000","last_change":"2024-03-26T03:01:52.734222+0000","last_active":"2024-03-26T03:04:19.697251+0000","last_peered":"2024-03-26T03:04:19.697251+0000","last_clean":"2024-03-26T03:04:19.697251+0000","last_became_active":"2024-03-26T03:01:51.309762+0000","last_became_peered":"2024-03-26T03:01:51.309762+0000","last_unstale":"2024-03-26T03:04:19.697251+0000","last_undegraded":"2024-03-26T03:04:19.697251+0000","last_fullsized":"2024-03-26T03:04:19.697251+0000","mapping_epoch":96,"log_start":"103'3239","ondisk_log_start":"103'3239","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'4791","last_scrub_stamp":"2024-03-26T03:01:52.734068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.734068+0000","objects_scrubbed":2345,"log_size":5144,"log_dups_size":0,"ondisk_log_size":5144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:22.671501+0000","scrub_duration":801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16646144,"num_objects":254,"num_object_clones":0,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3595,"num_write_kb":48128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430467+0000","last_change":"2024-03-26T03:00:48.212377+0000","last_active":"2024-03-26T03:04:16.430467+0000","last_peered":"2024-03-26T03:04:16.430467+0000","last_clean":"2024-03-26T03:04:16.430467+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:04:16.430467+0000","last_undegraded":"2024-03-26T03:04:16.430467+0000","last_fullsized":"2024-03-26T03:04:16.430467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:48.212296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:48.212296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430152+0000","last_change":"2024-03-26T03:01:40.975997+0000","last_active":"2024-03-26T03:04:16.430152+0000","last_peered":"2024-03-26T03:04:16.430152+0000","last_clean":"2024-03-26T03:04:16.430152+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:04:16.430152+0000","last_undegraded":"2024-03-26T03:04:16.430152+0000","last_fullsized":"2024-03-26T03:04:16.430152+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:40.975886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.975886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:16.792172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":249,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422049+0000","last_change":"2024-03-26T03:03:35.345801+0000","last_active":"2024-03-26T03:04:16.422049+0000","last_peered":"2024-03-26T03:04:16.422049+0000","last_clean":"2024-03-26T03:04:16.422049+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:04:16.422049+0000","last_undegraded":"2024-03-26T03:04:16.422049+0000","last_fullsized":"2024-03-26T03:04:16.422049+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:03:35.345668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:35.345668+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:41.962347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"91'9","reported_seq":279,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.425447+0000","last_change":"2024-03-26T03:00:37.456923+0000","last_active":"2024-03-26T03:04:16.425447+0000","last_peered":"2024-03-26T03:04:16.425447+0000","last_clean":"2024-03-26T03:04:16.425447+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:04:16.425447+0000","last_undegraded":"2024-03-26T03:04:16.425447+0000","last_fullsized":"2024-03-26T03:04:16.425447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:00:37.456826+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"160'12198","reported_seq":12341,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.985233+0000","last_change":"2024-03-26T03:00:55.434339+0000","last_active":"2024-03-26T03:04:18.985233+0000","last_peered":"2024-03-26T03:04:18.985233+0000","last_clean":"2024-03-26T03:04:18.985233+0000","last_became_active":"2024-03-26T03:00:55.433417+0000","last_became_peered":"2024-03-26T03:00:55.433417+0000","last_unstale":"2024-03-26T03:04:18.985233+0000","last_undegraded":"2024-03-26T03:04:18.985233+0000","last_fullsized":"2024-03-26T03:04:18.985233+0000","mapping_epoch":96,"log_start":"127'6684","ondisk_log_start":"127'6684","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814296,"num_objects":578,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12198,"num_write_kb":408769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.426920+0000","last_change":"2024-03-26T03:00:15.672174+0000","last_active":"2024-03-26T03:04:16.426920+0000","last_peered":"2024-03-26T03:04:16.426920+0000","last_clean":"2024-03-26T03:04:16.426920+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:04:16.426920+0000","last_undegraded":"2024-03-26T03:04:16.426920+0000","last_fullsized":"2024-03-26T03:04:16.426920+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:00:15.672090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":14,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424067+0000","last_change":"2024-03-26T03:04:14.416201+0000","last_active":"2024-03-26T03:04:16.424067+0000","last_peered":"2024-03-26T03:04:16.424067+0000","last_clean":"2024-03-26T03:04:16.424067+0000","last_became_active":"2024-03-26T03:04:14.415770+0000","last_became_peered":"2024-03-26T03:04:14.415770+0000","last_unstale":"2024-03-26T03:04:16.424067+0000","last_undegraded":"2024-03-26T03:04:16.424067+0000","last_fullsized":"2024-03-26T03:04:16.424067+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:00.224975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:00.224975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:20.942429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429023+0000","last_change":"2024-03-26T03:01:47.284940+0000","last_active":"2024-03-26T03:04:16.429023+0000","last_peered":"2024-03-26T03:04:16.429023+0000","last_clean":"2024-03-26T03:04:16.429023+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:04:16.429023+0000","last_undegraded":"2024-03-26T03:04:16.429023+0000","last_fullsized":"2024-03-26T03:04:16.429023+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:47.284840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.284840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":247,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.421972+0000","last_change":"2024-03-26T03:03:37.340501+0000","last_active":"2024-03-26T03:04:16.421972+0000","last_peered":"2024-03-26T03:04:16.421972+0000","last_clean":"2024-03-26T03:04:16.421972+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:04:16.421972+0000","last_undegraded":"2024-03-26T03:04:16.421972+0000","last_fullsized":"2024-03-26T03:04:16.421972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:03:37.340409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:37.340409+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:49.178750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"160'11792","reported_seq":11934,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:18.965528+0000","last_change":"2024-03-26T03:00:55.436147+0000","last_active":"2024-03-26T03:04:18.965528+0000","last_peered":"2024-03-26T03:04:18.965528+0000","last_clean":"2024-03-26T03:04:18.965528+0000","last_became_active":"2024-03-26T03:00:55.435784+0000","last_became_peered":"2024-03-26T03:00:55.435784+0000","last_unstale":"2024-03-26T03:04:18.965528+0000","last_undegraded":"2024-03-26T03:04:18.965528+0000","last_fullsized":"2024-03-26T03:04:18.965528+0000","mapping_epoch":96,"log_start":"123'6305","ondisk_log_start":"123'6305","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11792,"num_write_kb":397440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":14,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423288+0000","last_change":"2024-03-26T03:04:14.431790+0000","last_active":"2024-03-26T03:04:16.423288+0000","last_peered":"2024-03-26T03:04:16.423288+0000","last_clean":"2024-03-26T03:04:16.423288+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:04:16.423288+0000","last_undegraded":"2024-03-26T03:04:16.423288+0000","last_fullsized":"2024-03-26T03:04:16.423288+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.305791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.305791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:29.825899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":285,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423747+0000","last_change":"2024-03-26T03:03:56.360899+0000","last_active":"2024-03-26T03:04:16.423747+0000","last_peered":"2024-03-26T03:04:16.423747+0000","last_clean":"2024-03-26T03:04:16.423747+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:04:16.423747+0000","last_undegraded":"2024-03-26T03:04:16.423747+0000","last_fullsized":"2024-03-26T03:04:16.423747+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:03:56.360747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:56.360747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:23.660024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423463+0000","last_change":"2024-03-26T03:00:52.439831+0000","last_active":"2024-03-26T03:04:16.423463+0000","last_peered":"2024-03-26T03:04:16.423463+0000","last_clean":"2024-03-26T03:04:16.423463+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:04:16.423463+0000","last_undegraded":"2024-03-26T03:04:16.423463+0000","last_fullsized":"2024-03-26T03:04:16.423463+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.439710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.439710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":242,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422983+0000","last_change":"2024-03-26T03:04:05.712151+0000","last_active":"2024-03-26T03:04:16.422983+0000","last_peered":"2024-03-26T03:04:16.422983+0000","last_clean":"2024-03-26T03:04:16.422983+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:04:16.422983+0000","last_undegraded":"2024-03-26T03:04:16.422983+0000","last_fullsized":"2024-03-26T03:04:16.422983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:05.711979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.711979+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:33.910298+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"160'11870","reported_seq":12012,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:20.055426+0000","last_change":"2024-03-26T03:00:55.439219+0000","last_active":"2024-03-26T03:04:20.055426+0000","last_peered":"2024-03-26T03:04:20.055426+0000","last_clean":"2024-03-26T03:04:20.055426+0000","last_became_active":"2024-03-26T03:00:55.438925+0000","last_became_peered":"2024-03-26T03:00:55.438925+0000","last_unstale":"2024-03-26T03:04:20.055426+0000","last_undegraded":"2024-03-26T03:04:20.055426+0000","last_fullsized":"2024-03-26T03:04:20.055426+0000","mapping_epoch":96,"log_start":"134'7100","ondisk_log_start":"134'7100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.877123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11870,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":200,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423805+0000","last_change":"2024-03-26T03:03:32.396450+0000","last_active":"2024-03-26T03:04:16.423805+0000","last_peered":"2024-03-26T03:04:16.423805+0000","last_clean":"2024-03-26T03:04:16.423805+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:04:16.423805+0000","last_undegraded":"2024-03-26T03:04:16.423805+0000","last_fullsized":"2024-03-26T03:04:16.423805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:32.396317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:32.396317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:57.161788+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":287,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432207+0000","last_change":"2024-03-26T03:01:50.516513+0000","last_active":"2024-03-26T03:04:16.432207+0000","last_peered":"2024-03-26T03:04:16.432207+0000","last_clean":"2024-03-26T03:04:16.432207+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:04:16.432207+0000","last_undegraded":"2024-03-26T03:04:16.432207+0000","last_fullsized":"2024-03-26T03:04:16.432207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:01:50.516336+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.516336+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424335+0000","last_change":"2024-03-26T03:01:50.576964+0000","last_active":"2024-03-26T03:04:16.424335+0000","last_peered":"2024-03-26T03:04:16.424335+0000","last_clean":"2024-03-26T03:04:16.424335+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:04:16.424335+0000","last_undegraded":"2024-03-26T03:04:16.424335+0000","last_fullsized":"2024-03-26T03:04:16.424335+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:50.576769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.576769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.423148+0000","last_change":"2024-03-26T03:03:48.428215+0000","last_active":"2024-03-26T03:04:16.423148+0000","last_peered":"2024-03-26T03:04:16.423148+0000","last_clean":"2024-03-26T03:04:16.423148+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:04:16.423148+0000","last_undegraded":"2024-03-26T03:04:16.423148+0000","last_fullsized":"2024-03-26T03:04:16.423148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:03:48.427958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:48.427958+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:03.321011+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"160'12068","reported_seq":12208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.301973+0000","last_change":"2024-03-26T03:00:55.433696+0000","last_active":"2024-03-26T03:04:19.301973+0000","last_peered":"2024-03-26T03:04:19.301973+0000","last_clean":"2024-03-26T03:04:19.301973+0000","last_became_active":"2024-03-26T03:00:55.433381+0000","last_became_peered":"2024-03-26T03:00:55.433381+0000","last_unstale":"2024-03-26T03:04:19.301973+0000","last_undegraded":"2024-03-26T03:04:19.301973+0000","last_fullsized":"2024-03-26T03:04:19.301973+0000","mapping_epoch":96,"log_start":"122'6400","ondisk_log_start":"122'6400","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12068,"num_write_kb":405440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431619+0000","last_change":"2024-03-26T03:03:55.839464+0000","last_active":"2024-03-26T03:04:16.431619+0000","last_peered":"2024-03-26T03:04:16.431619+0000","last_clean":"2024-03-26T03:04:16.431619+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:04:16.431619+0000","last_undegraded":"2024-03-26T03:04:16.431619+0000","last_fullsized":"2024-03-26T03:04:16.431619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:55.839275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:55.839275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:25.215925+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424061+0000","last_change":"2024-03-26T03:00:45.397106+0000","last_active":"2024-03-26T03:04:16.424061+0000","last_peered":"2024-03-26T03:04:16.424061+0000","last_clean":"2024-03-26T03:04:16.424061+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:04:16.424061+0000","last_undegraded":"2024-03-26T03:04:16.424061+0000","last_fullsized":"2024-03-26T03:04:16.424061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.397023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429265+0000","last_change":"2024-03-26T03:01:33.183481+0000","last_active":"2024-03-26T03:04:16.429265+0000","last_peered":"2024-03-26T03:04:16.429265+0000","last_clean":"2024-03-26T03:04:16.429265+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:04:16.429265+0000","last_undegraded":"2024-03-26T03:04:16.429265+0000","last_fullsized":"2024-03-26T03:04:16.429265+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.183362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.183362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"153'25","reported_seq":252,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.422873+0000","last_change":"2024-03-26T03:03:57.378897+0000","last_active":"2024-03-26T03:04:16.422873+0000","last_peered":"2024-03-26T03:04:16.422873+0000","last_clean":"2024-03-26T03:04:16.422873+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:04:16.422873+0000","last_undegraded":"2024-03-26T03:04:16.422873+0000","last_fullsized":"2024-03-26T03:04:16.422873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'25","last_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:03:57.378721+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:00.647021+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"160'8297","reported_seq":8476,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:20.007719+0000","last_change":"2024-03-26T03:01:52.545451+0000","last_active":"2024-03-26T03:04:20.007719+0000","last_peered":"2024-03-26T03:04:20.007719+0000","last_clean":"2024-03-26T03:04:20.007719+0000","last_became_active":"2024-03-26T03:01:51.307163+0000","last_became_peered":"2024-03-26T03:01:51.307163+0000","last_unstale":"2024-03-26T03:04:20.007719+0000","last_undegraded":"2024-03-26T03:04:20.007719+0000","last_fullsized":"2024-03-26T03:04:20.007719+0000","mapping_epoch":96,"log_start":"100'2418","ondisk_log_start":"100'2418","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4657","last_scrub_stamp":"2024-03-26T03:01:52.545222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.545222+0000","objects_scrubbed":2326,"log_size":5879,"log_dups_size":0,"ondisk_log_size":5879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.877379+0000","scrub_duration":616,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":49984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428872+0000","last_change":"2024-03-26T03:01:37.160971+0000","last_active":"2024-03-26T03:04:16.428872+0000","last_peered":"2024-03-26T03:04:16.428872+0000","last_clean":"2024-03-26T03:04:16.428872+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:04:16.428872+0000","last_undegraded":"2024-03-26T03:04:16.428872+0000","last_fullsized":"2024-03-26T03:04:16.428872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:37.160888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.160888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.430790+0000","last_change":"2024-03-26T03:01:43.387197+0000","last_active":"2024-03-26T03:04:16.430790+0000","last_peered":"2024-03-26T03:04:16.430790+0000","last_clean":"2024-03-26T03:04:16.430790+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:04:16.430790+0000","last_undegraded":"2024-03-26T03:04:16.430790+0000","last_fullsized":"2024-03-26T03:04:16.430790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:43.386881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:01:43.386881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431195+0000","last_change":"2024-03-26T03:01:40.361299+0000","last_active":"2024-03-26T03:04:16.431195+0000","last_peered":"2024-03-26T03:04:16.431195+0000","last_clean":"2024-03-26T03:04:16.431195+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:04:16.431195+0000","last_undegraded":"2024-03-26T03:04:16.431195+0000","last_fullsized":"2024-03-26T03:04:16.431195+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:40.360702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.360702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":288,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.429318+0000","last_change":"2024-03-26T03:01:54.088999+0000","last_active":"2024-03-26T03:04:16.429318+0000","last_peered":"2024-03-26T03:04:16.429318+0000","last_clean":"2024-03-26T03:04:16.429318+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:04:16.429318+0000","last_undegraded":"2024-03-26T03:04:16.429318+0000","last_fullsized":"2024-03-26T03:04:16.429318+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:01:54.088880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.088880+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:16.792258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"160'8451","reported_seq":8632,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.238459+0000","last_change":"2024-03-26T03:01:52.302237+0000","last_active":"2024-03-26T03:04:19.238459+0000","last_peered":"2024-03-26T03:04:19.238459+0000","last_clean":"2024-03-26T03:04:19.238459+0000","last_became_active":"2024-03-26T03:01:51.304269+0000","last_became_peered":"2024-03-26T03:01:51.304269+0000","last_unstale":"2024-03-26T03:04:19.238459+0000","last_undegraded":"2024-03-26T03:04:19.238459+0000","last_fullsized":"2024-03-26T03:04:19.238459+0000","mapping_epoch":96,"log_start":"99'922","ondisk_log_start":"99'922","created":96,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4786","last_scrub_stamp":"2024-03-26T03:01:52.302079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.302079+0000","objects_scrubbed":2466,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3681,"num_write_kb":48768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.427236+0000","last_change":"2024-03-26T03:00:48.304407+0000","last_active":"2024-03-26T03:04:16.427236+0000","last_peered":"2024-03-26T03:04:16.427236+0000","last_clean":"2024-03-26T03:04:16.427236+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:04:16.427236+0000","last_undegraded":"2024-03-26T03:04:16.427236+0000","last_fullsized":"2024-03-26T03:04:16.427236+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:00:48.304319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.432862+0000","last_change":"2024-03-26T03:00:40.307781+0000","last_active":"2024-03-26T03:04:16.432862+0000","last_peered":"2024-03-26T03:04:16.432862+0000","last_clean":"2024-03-26T03:04:16.432862+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:04:16.432862+0000","last_undegraded":"2024-03-26T03:04:16.432862+0000","last_fullsized":"2024-03-26T03:04:16.432862+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:40.307697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:00:40.307697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.424008+0000","last_change":"2024-03-26T03:03:59.711554+0000","last_active":"2024-03-26T03:04:16.424008+0000","last_peered":"2024-03-26T03:04:16.424008+0000","last_clean":"2024-03-26T03:04:16.424008+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:04:16.424008+0000","last_undegraded":"2024-03-26T03:04:16.424008+0000","last_fullsized":"2024-03-26T03:04:16.424008+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:03:59.710985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:25.977819+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":250,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428357+0000","last_change":"2024-03-26T03:04:06.968953+0000","last_active":"2024-03-26T03:04:16.428357+0000","last_peered":"2024-03-26T03:04:16.428357+0000","last_clean":"2024-03-26T03:04:16.428357+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:04:16.428357+0000","last_undegraded":"2024-03-26T03:04:16.428357+0000","last_fullsized":"2024-03-26T03:04:16.428357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_clean_scrub_stamp":"2024-03-26T03:04:06.968735+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:17.994202+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"160'11816","reported_seq":11958,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:19.133856+0000","last_change":"2024-03-26T03:00:55.435500+0000","last_active":"2024-03-26T03:04:19.133856+0000","last_peered":"2024-03-26T03:04:19.133856+0000","last_clean":"2024-03-26T03:04:19.133856+0000","last_became_active":"2024-03-26T03:00:55.435115+0000","last_became_peered":"2024-03-26T03:00:55.435115+0000","last_unstale":"2024-03-26T03:04:19.133856+0000","last_undegraded":"2024-03-26T03:04:19.133856+0000","last_fullsized":"2024-03-26T03:04:19.133856+0000","mapping_epoch":96,"log_start":"130'6800","ondisk_log_start":"130'6800","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:54.410365+0000","last_clean_scrub_stamp":"2024-03-26T03:00:54.410365+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.534390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":397248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.428920+0000","last_change":"2024-03-26T03:04:14.423661+0000","last_active":"2024-03-26T03:04:16.428920+0000","last_peered":"2024-03-26T03:04:16.428920+0000","last_clean":"2024-03-26T03:04:16.428920+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:04:16.428920+0000","last_undegraded":"2024-03-26T03:04:16.428920+0000","last_fullsized":"2024-03-26T03:04:16.428920+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:39.931898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:39.931898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431563+0000","last_change":"2024-03-26T03:01:48.444274+0000","last_active":"2024-03-26T03:04:16.431563+0000","last_peered":"2024-03-26T03:04:16.431563+0000","last_clean":"2024-03-26T03:04:16.431563+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:04:16.431563+0000","last_undegraded":"2024-03-26T03:04:16.431563+0000","last_fullsized":"2024-03-26T03:04:16.431563+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:48.444147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:48.444147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":265,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.433073+0000","last_change":"2024-03-26T03:01:34.223673+0000","last_active":"2024-03-26T03:04:16.433073+0000","last_peered":"2024-03-26T03:04:16.433073+0000","last_clean":"2024-03-26T03:04:16.433073+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:04:16.433073+0000","last_undegraded":"2024-03-26T03:04:16.433073+0000","last_fullsized":"2024-03-26T03:04:16.433073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:01:34.223531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.223531+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:17.406515+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":287,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:04:16.431331+0000","last_change":"2024-03-26T03:01:51.306960+0000","last_active":"2024-03-26T03:04:16.431331+0000","last_peered":"2024-03-26T03:04:16.431331+0000","last_clean":"2024-03-26T03:04:16.431331+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:04:16.431331+0000","last_undegraded":"2024-03-26T03:04:16.431331+0000","last_fullsized":"2024-03-26T03:04:16.431331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:01:51.306839+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:01:51.306839+0000","last_clean_scrub_stamp":"2024-03-26T03:01:51.306839+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:04:21.688312+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":611319832,"num_objects":9329,"num_object_clones":0,"num_object_copies":27987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116366,"num_write_kb":2874625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2215391232,"data_stored":2215379016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151623,"ondisk_log_size":151623,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":313,"num_read_kb":308,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":641,"ondisk_log_size":641,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103237,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464760,"kb_used_data":293360,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93280392,"statfs":{"total":95995035648,"available":95519121408,"internally_reserved":0,"allocated":300400640,"data_stored":300003378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430281,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":536692,"kb_used_data":375020,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93208460,"statfs":{"total":95995035648,"available":95445463040,"internally_reserved":0,"allocated":384020480,"data_stored":383626720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297368,"kb_used_data":180108,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93447784,"statfs":{"total":95995035648,"available":95690530816,"internally_reserved":0,"allocated":184430592,"data_stored":184005340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":559748,"kb_used_data":366140,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93185404,"statfs":{"total":95995035648,"available":95421853696,"internally_reserved":0,"allocated":374927360,"data_stored":374518487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":493228,"kb_used_data":321828,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93251924,"statfs":{"total":95995035648,"available":95489970176,"internally_reserved":0,"allocated":329551872,"data_stored":329166793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583112,"kb_used_data":395008,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":93162040,"statfs":{"total":95995035648,"available":95397928960,"internally_reserved":0,"allocated":404488192,"data_stored":404073813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":192543175},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986259,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":241284,"kb_used_data":129592,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93503868,"statfs":{"total":95995035648,"available":95747960832,"internally_reserved":0,"allocated":132702208,"data_stored":132296119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691919,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309104,"kb_used_data":197416,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93436048,"statfs":{"total":95995035648,"available":95678513152,"internally_reserved":0,"allocated":202153984,"data_stored":201763830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367398912,"data_stored":367394840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168628224,"data_stored":168624152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":314245120,"data_stored":314245120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401801216,"data_stored":401801216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117506048,"data_stored":117506048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185466880,"data_stored":185466880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":373231616,"data_stored":373227544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":287113216,"data_stored":287113216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:04:26.527 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T03:04:26.527 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-26T03:04:26.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:26.621 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:26.706 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:26.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:26.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:26.860 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:26.915 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:26.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:27.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:27.069 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:27.124 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:27 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:04:27.124 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:27 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:04:27.124 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:27 smithi192 ceph-mon[33110]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T03:04:27.124 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:27 smithi192 ceph-mon[33110]: pgmap v228: 158 pgs: 158 active+clean; 244 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T03:04:27.175 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:27.234 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:27.285 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:04:27.285 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:04:27.285 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[29666]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T03:04:27.285 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[29666]: pgmap v228: 158 pgs: 158 active+clean; 244 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T03:04:27.286 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:04:27.286 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:04:27.286 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[37920]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T03:04:27.286 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:27 smithi066 ceph-mon[37920]: pgmap v228: 158 pgs: 158 active+clean; 244 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T03:04:27.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:27.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:27.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:27.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:27.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:27.640 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:04:27.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:27.692 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:04:27.730 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:04:27.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:27.771 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:04:27.794 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:27.816 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:04:27.862 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:04:27.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:27.908 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:04:27.919 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:27.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:27.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:27.970 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:04:27.977 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:04:27.977 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:04:27.977 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:04:27.978 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:04:27.978 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:04:27.978 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:04:27.978 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:04:27.978 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:04:28.056 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Removed 100180 objects 2024-03-26T03:04:28.056 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Clean up completed and total clean up time :113.777 2024-03-26T03:04:28.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:28.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:28.185 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:28.211 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3531103258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:28.211 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3531103258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:28.214 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:28.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3531103258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:28.316 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.344 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:28.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:28.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:28.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:28.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:28.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:28.636 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:21265 7.0 7:0d96ba56:::benchmark_data_smithi066_76917_object21264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:11.471853+0000", 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 197.16241717899999, 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.099009812000000003, 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 21265 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471853+0000", 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471853+0000", 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471855+0000", 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.81e-06 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471918+0000", 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3675000000000003e-05 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471929+0000", 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0781000000000001e-05 2024-03-26T03:04:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471937+0000", 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2230000000000007e-06 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.471996+0000", 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8457e-05 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472127+0000", 2024-03-26T03:04:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013131699999999999 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472233+0000", 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010618799999999999 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.569586+0000", 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.097352290999999994 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.569627+0000", 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1118999999999997e-05 2024-03-26T03:04:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.570787+0000", 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011606990000000001 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.570822+0000", 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4084000000000001e-05 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.570863+0000", 2024-03-26T03:04:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1168000000000003e-05 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:21267 7.0 7:0ffa4a88:::benchmark_data_smithi066_76917_object21266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:11.472377+0000", 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 197.16189262, 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.098786088999999994, 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 21267 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472377+0000", 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472377+0000", 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472379+0000", 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1000000000000001e-06 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472457+0000", 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8145000000000005e-05 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472467+0000", 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.026e-05 2024-03-26T03:04:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.472474+0000", 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4320000000000002e-06 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.473172+0000", 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00069803699999999998 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.473262+0000", 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9320999999999994e-05 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.568035+0000", 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.094773202000000001 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.570928+0000", 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028935570000000002 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.570996+0000", 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7181999999999995e-05 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.571014+0000", 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8190999999999999e-05 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.571093+0000", 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9246999999999995e-05 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:11.571164+0000", 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0414999999999996e-05 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:27534 7.0 7:0c401e1e:::benchmark_data_smithi066_76917_object27533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:14.797704+0000", 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 193.836565759, 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.172682524, 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 27534 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797704+0000", 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797704+0000", 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797706+0000", 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797760+0000", 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3449000000000003e-05 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797769+0000", 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.149e-06 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797776+0000", 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.379e-06 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797885+0000", 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010837500000000001 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797960+0000", 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4871999999999997e-05 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.798055+0000", 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5304999999999996e-05 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799033+0000", 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00097775100000000001 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799202+0000", 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000169798 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.970321+0000", 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.171118767 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.970357+0000", 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5686999999999999e-05 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.970387+0000", 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9873e-05 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:27540 7.0 7:0b0e0d9b:::benchmark_data_smithi066_76917_object27539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:14.799714+0000", 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 193.834555615, 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17334993200000001, 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 27540 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799714+0000", 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799714+0000", 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799716+0000", 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4020000000000001e-06 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799778+0000", 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2278000000000002e-05 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799788+0000", 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0219e-05 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799795+0000", 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4749999999999998e-06 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799847+0000", 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1823000000000002e-05 2024-03-26T03:04:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799974+0000", 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127353 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.800120+0000", 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000145852 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.800972+0000", 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085239699999999996 2024-03-26T03:04:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.801046+0000", 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3663000000000003e-05 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.973003+0000", 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17195730000000001 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.973039+0000", 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5882e-05 2024-03-26T03:04:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.973064+0000", 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5287999999999999e-05 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28224 7.0 7:0aa9b815:::benchmark_data_smithi066_76917_object28223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.289009+0000", 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 193.34526118100001, 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.119753289, 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28224 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289009+0000", 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289009+0000", 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289010+0000", 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.474e-06 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289095+0000", 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4634000000000003e-05 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289110+0000", 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4963999999999999e-05 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289118+0000", 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5850000000000002e-06 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289233+0000", 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115214 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289350+0000", 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011717700000000001 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289479+0000", 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012877200000000001 2024-03-26T03:04:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.290665+0000", 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011862450000000001 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291515+0000", 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085024500000000002 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408656+0000", 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.117140335 2024-03-26T03:04:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408715+0000", 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9061999999999998e-05 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408762+0000", 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7581999999999999e-05 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28227 7.0 7:0aa0c260:::benchmark_data_smithi066_76917_object28226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.291818+0000", 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 193.342452154, 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.117099724, 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28227 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291818+0000", 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291818+0000", 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291819+0000", 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.457e-06 2024-03-26T03:04:28.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291883+0000", 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3862000000000006e-05 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291892+0000", 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1179999999999998e-06 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291900+0000", 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6960000000000005e-06 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291976+0000", 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5934999999999997e-05 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292055+0000", 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9031999999999998e-05 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292144+0000", 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8855000000000002e-05 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.294043+0000", 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001898765 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295519+0000", 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014764419999999999 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408823+0000", 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.113304047 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408877+0000", 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3791e-05 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408918+0000", 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0723999999999998e-05 2024-03-26T03:04:28.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28236 7.0 7:0e3637c8:::benchmark_data_smithi066_76917_object28235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.292650+0000", 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 193.34162045799999, 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.116428117, 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28236 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292650+0000", 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292650+0000", 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292651+0000", 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2839999999999999e-06 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292743+0000", 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2077999999999995e-05 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292758+0000", 2024-03-26T03:04:28.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5021000000000001e-05 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292766+0000", 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.405e-06 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292872+0000", 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105282 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292988+0000", 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116025 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.293099+0000", 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011086600000000001 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.294187+0000", 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00108884 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295696+0000", 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015085540000000001 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408973+0000", 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11327743699999999 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.409042+0000", 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8937999999999996e-05 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.409078+0000", 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5386999999999998e-05 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28771 7.0 7:019680ad:::benchmark_data_smithi066_76917_object28770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.642646+0000", 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.991623745, 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.143119834, 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.681 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28771 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642646+0000", 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642646+0000", 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642649+0000", 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2120000000000002e-06 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642706+0000", 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7716000000000001e-05 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642722+0000", 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5274999999999999e-05 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642729+0000", 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3069999999999997e-06 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642798+0000", 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9510000000000004e-05 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642935+0000", 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013685600000000001 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643041+0000", 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010560000000000001 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645176+0000", 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021349870000000001 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645798+0000", 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00062242299999999999 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.785647+0000", 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13984848999999999 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.785710+0000", 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3287000000000006e-05 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.785766+0000", 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6171000000000001e-05 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28776 7.0 7:06c4cb01:::benchmark_data_smithi066_76917_object28775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.643359+0000", 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.99091123400001, 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.144621905, 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.687 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28776 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643359+0000", 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643359+0000", 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643360+0000", 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4869999999999999e-06 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643400+0000", 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9524000000000003e-05 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643415+0000", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4773999999999999e-05 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643422+0000", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7409999999999992e-06 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643921+0000", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000498925 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644054+0000", 2024-03-26T03:04:28.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013238799999999999 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644202+0000", 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000148825 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645323+0000", 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001120081 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645911+0000", 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00058858000000000001 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787861+0000", 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.141949363 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787923+0000", 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2137000000000005e-05 2024-03-26T03:04:28.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787981+0000", 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.808e-05 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:29450 7.0 7:0cfae8a6:::benchmark_data_smithi066_76917_object29449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.097295+0000", 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.53697513500001, 2024-03-26T03:04:28.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17839643499999999, 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 29450 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097295+0000", 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097295+0000", 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097297+0000", 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6700000000000001e-06 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097352+0000", 2024-03-26T03:04:28.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4947000000000001e-05 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097362+0000", 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0404000000000001e-05 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097368+0000", 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5890000000000002e-06 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097412+0000", 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4308999999999999e-05 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097514+0000", 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102359 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097613+0000", 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8985000000000002e-05 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.098601+0000", 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00098787399999999991 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.098978+0000", 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037711100000000002 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.275600+0000", 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17662185899999999 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.275639+0000", 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9023000000000002e-05 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.275691+0000", 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2305000000000002e-05 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30628 7.0 7:0c9ca97b:::benchmark_data_smithi066_76917_object30627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.174269+0000", 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 191.46000074599999, 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.098556974000000006, 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.700 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30628 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174269+0000", 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174269+0000", 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174271+0000", 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.753e-06 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174327+0000", 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6332000000000002e-05 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174340+0000", 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.259e-05 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174348+0000", 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1110000000000008e-06 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174408+0000", 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9562000000000003e-05 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174527+0000", 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119318 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.174669+0000", 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000141779 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.175831+0000", 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001162646 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176354+0000", 2024-03-26T03:04:28.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052286300000000004 2024-03-26T03:04:28.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.272710+0000", 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.096355576999999998 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.272774+0000", 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4406999999999995e-05 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.272826+0000", 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2036000000000001e-05 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30630 7.0 7:0f4d6a3d:::benchmark_data_smithi066_76917_object30629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.176419+0000", 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 191.45785130600001, 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.097018954000000004, 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30630 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176419+0000", 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176419+0000", 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176420+0000", 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2999999999999999e-07 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176478+0000", 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8470000000000001e-05 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176487+0000", 2024-03-26T03:04:28.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0999999999999993e-06 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176495+0000", 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6420000000000007e-06 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176531+0000", 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5726000000000003e-05 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176611+0000", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0103000000000002e-05 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.176701+0000", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0655000000000005e-05 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.177558+0000", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085616000000000002 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.178509+0000", 2024-03-26T03:04:28.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00095111600000000001 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.273347+0000", 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.094838063 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.273409+0000", 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1934999999999996e-05 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.273438+0000", 2024-03-26T03:04:28.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9054e-05 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30716 7.0 7:0202be7f:::benchmark_data_smithi066_76917_object30715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.300707+0000", 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 191.33356306499999, 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11267152599999999, 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30716 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300707+0000", 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300707+0000", 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300709+0000", 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0499999999999999e-06 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300761+0000", 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2040000000000003e-05 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300776+0000", 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4598e-05 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300783+0000", 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6009999999999997e-06 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300850+0000", 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7045000000000006e-05 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300993+0000", 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014279000000000001 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.301135+0000", 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014195700000000001 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302196+0000", 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010605759999999999 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.303765+0000", 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001568975 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413269+0000", 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.109504431 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413327+0000", 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8439000000000001e-05 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413379+0000", 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1023999999999998e-05 2024-03-26T03:04:28.718 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30717 7.0 7:0d0913d8:::benchmark_data_smithi066_76917_object30716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.302714+0000", 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 191.33155634900001, 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.110809716, 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.719 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30717 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302714+0000", 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302714+0000", 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302718+0000", 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4900000000000002e-06 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302775+0000", 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6983e-05 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302785+0000", 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0084999999999999e-05 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302793+0000", 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5299999999999999e-06 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302840+0000", 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7524000000000001e-05 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302960+0000", 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120081 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.303069+0000", 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108811 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.303877+0000", 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080767800000000004 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.304274+0000", 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000396769 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413423+0000", 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.109148934 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413480+0000", 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7497000000000002e-05 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413523+0000", 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3334000000000002e-05 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.724 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31281 7.0 7:06258b42:::benchmark_data_smithi066_76917_object31280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.738133+0000", 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 190.89613715999999, 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14132778900000001, 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31281 2024-03-26T03:04:28.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738133+0000", 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738133+0000", 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738135+0000", 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.091e-06 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738200+0000", 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4888000000000006e-05 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738212+0000", 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1803e-05 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738218+0000", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2659999999999998e-06 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738275+0000", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7308000000000001e-05 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738414+0000", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000138329 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738551+0000", 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013751200000000001 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.739875+0000", 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001323942 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.740111+0000", 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000235658 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.879381+0000", 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13927038999999999 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.879419+0000", 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7885000000000001e-05 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.879461+0000", 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1717000000000001e-05 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.730 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75536 7.0 7:000359a7:::benchmark_data_smithi066_76917_object75535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 158.310864573, 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.004732272, 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75536 2024-03-26T03:04:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323409+0000", 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3119999999999998e-06 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323453+0000", 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4338999999999997e-05 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323474+0000", 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1109000000000001e-05 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323488+0000", 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3397000000000001e-05 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.363800+0000", 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040311903000000003 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.363816+0000", 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6526000000000001e-05 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313106+0000", 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94928997599999998 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313268+0000", 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016208999999999999 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313443+0000", 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174558 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.316322+0000", 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028792119999999999 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.317555+0000", 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012334970000000001 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328022+0000", 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010466392999999999 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328092+0000", 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9900000000000005e-05 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328138+0000", 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6060000000000003e-05 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75538 7.0 7:0739b178:::benchmark_data_smithi066_76917_object75537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 158.08678556300001, 2024-03-26T03:04:28.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78758199900000003, 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75538 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547488+0000", 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0359999999999998e-06 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547625+0000", 2024-03-26T03:04:28.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013731600000000001 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547645+0000", 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9944000000000001e-05 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547656+0000", 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1610000000000001e-05 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547729+0000", 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2249999999999994e-05 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547738+0000", 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5040000000000008e-06 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315069+0000", 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.76733058799999998 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315202+0000", 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132769 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315298+0000", 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6464000000000003e-05 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.316448+0000", 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011497969999999999 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.319454+0000", 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003006075 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.334937+0000", 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015482902 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.335019+0000", 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1930999999999999e-05 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.335066+0000", 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7812999999999998e-05 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:111974 7.0 7:06bf99cb:::benchmark_data_smithi066_76917_object11791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 97.654145577999998, 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14525418500000001, 2024-03-26T03:04:28.744 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111974 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980127+0000", 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0849999999999999e-06 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980134+0000", 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1810000000000004e-06 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980149+0000", 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5702999999999999e-05 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980159+0000", 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7969999999999995e-06 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980225+0000", 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5648000000000006e-05 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980350+0000", 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125227 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980649+0000", 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029928300000000001 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.982219+0000", 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001569916 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.983019+0000", 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00079931200000000005 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125310+0000", 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14229117699999999 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125347+0000", 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7212000000000002e-05 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125379+0000", 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1643999999999997e-05 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112720 7.0 7:054d20dd:::benchmark_data_smithi066_76917_object12537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.118665264000001, 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13726976599999999, 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.751 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112720 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515607+0000", 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1119999999999999e-06 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515613+0000", 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1709999999999999e-06 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515629+0000", 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5914e-05 2024-03-26T03:04:28.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515638+0000", 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6910000000000003e-06 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515696+0000", 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7989999999999999e-05 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515813+0000", 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116997 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.516136+0000", 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032339800000000002 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.518072+0000", 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001936173 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.521492+0000", 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034199230000000001 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652808+0000", 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.131315819 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652843+0000", 2024-03-26T03:04:28.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4805000000000003e-05 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652875+0000", 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1773000000000002e-05 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:142785 7.0 7:05427235:::benchmark_data_smithi066_76917_object42602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.188019748000002, 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.121159576, 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:28.757 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 142785 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446253+0000", 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3099999999999999e-06 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446259+0000", 2024-03-26T03:04:28.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2999999999999998e-06 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446275+0000", 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6047999999999999e-05 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446284+0000", 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:28.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446354+0000", 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0160999999999995e-05 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446480+0000", 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000126036 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446868+0000", 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038751299999999997 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:28.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.449292+0000", 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024239219999999998 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.452247+0000", 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029555039999999999 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567305+0000", 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.115057495 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567363+0000", 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8637000000000001e-05 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567410+0000", 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6641000000000003e-05 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:28.763 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:28.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:28.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:28.889 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:28.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:28.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:29.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:29.026 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:04:29.049 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:29.151 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:29.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:29.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:04:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:04:29.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:29 smithi192 ceph-mon[33110]: pgmap v229: 158 pgs: 158 active+clean; 244 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-26T03:04:29.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:29 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/516725435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:04:29.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:29 smithi192 ceph-mon[33110]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:04:29.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:29.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:29.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:29 smithi066 ceph-mon[37920]: pgmap v229: 158 pgs: 158 active+clean; 244 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-26T03:04:29.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:29 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/516725435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:04:29.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:29 smithi066 ceph-mon[37920]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:04:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:29 smithi066 ceph-mon[29666]: pgmap v229: 158 pgs: 158 active+clean; 244 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-26T03:04:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:29 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/516725435' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:04:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:29 smithi066 ceph-mon[29666]: from='client.24880 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:04:29.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:29.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:04:29.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:29.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:29.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:29.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:29.730 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:29.786 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:29.815 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:29.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:29.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:29.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:30.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:30.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:30.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:30.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[29666]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:04:30.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[29666]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T03:04:30.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:30.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:30.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[37920]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:04:30.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[37920]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T03:04:30.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:30.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:30 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:30.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:30 smithi192 ceph-mon[33110]: from='client.24880 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:04:30.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:30 smithi192 ceph-mon[33110]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T03:04:30.126 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:30.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:30.296 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:30.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:30.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:30.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:30.554 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:30.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:30.687 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:30.720 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:30.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:30.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:30.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:30.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:30.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:30.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:31.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:31.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:31.142 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:04:31.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:31.227 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:31.266 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:04:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:04:31.266 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:31.275 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:31 smithi192 ceph-mon[33110]: pgmap v231: 158 pgs: 158 active+clean; 44 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T03:04:31.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:31.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:31.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:31 smithi066 ceph-mon[29666]: pgmap v231: 158 pgs: 158 active+clean; 44 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T03:04:31.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:31 smithi066 ceph-mon[37920]: pgmap v231: 158 pgs: 158 active+clean; 44 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T03:04:31.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:31.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:31.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:31.607 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:31.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:31.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:31.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:31.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:31.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:04:31.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:04:31.960 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:31.990 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:31.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:32.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:32.128 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:32.142 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:32.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:32.257 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:32.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:32.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:32.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:32.508 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:32.610 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:32.639 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:32.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:32.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:32.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:32.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:32.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:32.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:33.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:33.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:33.153 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:33.223 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:33.223 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:33.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:33 smithi192 ceph-mon[33110]: pgmap v232: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 871 op/s 2024-03-26T03:04:33.255 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:33.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:33.347 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:33 smithi066 ceph-mon[29666]: pgmap v232: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 871 op/s 2024-03-26T03:04:33.347 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:33 smithi066 ceph-mon[37920]: pgmap v232: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 871 op/s 2024-03-26T03:04:33.385 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:33.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:33.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:33.542 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:33.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:33.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:33.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:33.799 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:33.828 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:04:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:04:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:04:33.828 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:33.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:33.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:33.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:33.988 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:04:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:04:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:04:34.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:34.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:34.097 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 1 pg_num to 11 2024-03-26T03:04:34.152 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:34.180 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:34.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:34.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:04:34.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:04:34.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1148364610' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T03:04:34.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[29666]: 7.e scrub starts 2024-03-26T03:04:34.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[29666]: 7.e scrub ok 2024-03-26T03:04:34.290 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:04:34.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:04:34.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1148364610' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T03:04:34.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[37920]: 7.e scrub starts 2024-03-26T03:04:34.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:34 smithi066 ceph-mon[37920]: 7.e scrub ok 2024-03-26T03:04:34.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:34.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:34.435 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:34 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:04:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:34 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:04:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1148364610' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T03:04:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:34 smithi192 ceph-mon[33110]: 7.e scrub starts 2024-03-26T03:04:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:34 smithi192 ceph-mon[33110]: 7.e scrub ok 2024-03-26T03:04:34.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:04:34.536 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:34.564 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:34.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:34.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:34.766 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:34.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:34.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:34.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:34.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:34.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:35.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:35.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:35.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: pgmap v233: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 B/s wr, 485 op/s 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1148364610' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:04:35.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: pgmap v233: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 B/s wr, 485 op/s 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1148364610' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:04:35.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:35 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:04:35.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: pgmap v233: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 B/s wr, 485 op/s 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1148364610' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:04:35.173 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:35 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:04:35.197 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:35.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:35.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:35.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:35.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:35.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:35.578 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:35.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:35.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:35.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:35.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:35.934 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:35.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:35.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:36.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:36.090 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:36.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:36.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:36.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:36.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:36.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 6.4 scrub starts 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 6.4 scrub ok 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 6.4 scrub starts 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 6.4 scrub ok 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:04:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:04:36.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:04:36.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:04:36.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:04:36.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:36 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T03:04:36.431 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-26T03:04:36.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-26T03:04:36.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:36.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 6.4 scrub starts 2024-03-26T03:04:36.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 6.4 scrub ok 2024-03-26T03:04:36.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:04:36.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:04:36.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:04:36.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:04:36.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:04:36.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:04:36.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:36 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T03:04:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.501 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:36.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:36.601 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:36.601 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:04:36.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:04:36.601 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:04:36.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.601 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75523 7.7 7:e4bf8bb6:::benchmark_data_smithi066_76917_object75522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.308141+0000", 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 166.29037684400001, 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0242819809999999, 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75523 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308141+0000", 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308141+0000", 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308144+0000", 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2849999999999999e-06 2024-03-26T03:04:36.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308199+0000", 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4549000000000002e-05 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308223+0000", 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4646000000000001e-05 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308240+0000", 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6317e-05 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370555+0000", 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.062315191999999998 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370566+0000", 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0849000000000001e-05 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.306034+0000", 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93546880300000002 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.306851+0000", 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00081647499999999999 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332390+0000", 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025538693000000001 2024-03-26T03:04:36.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332423+0000", 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3172e-05 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75529 7.7 7:e0f45fed:::benchmark_data_smithi066_76917_object75528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.308269+0000", 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 166.29024886799999, 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.024087451, 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.607 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75529 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308269+0000", 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308269+0000", 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308270+0000", 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8700000000000004e-07 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308307+0000", 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7537000000000003e-05 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308315+0000", 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0870000000000007e-06 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308322+0000", 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6390000000000003e-06 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370590+0000", 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.062267956999999999 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370598+0000", 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5460000000000003e-06 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.307310+0000", 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93671239100000003 2024-03-26T03:04:36.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.307993+0000", 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00068283600000000003 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332308+0000", 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024315666 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332356+0000", 2024-03-26T03:04:36.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7805000000000001e-05 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75541 7.7 7:e067e745:::benchmark_data_smithi066_76917_object75540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.567207+0000", 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 166.03131113800001, 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.765059932, 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.613 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75541 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567207+0000", 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567207+0000", 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567210+0000", 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1e-06 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567274+0000", 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4264999999999995e-05 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567292+0000", 2024-03-26T03:04:36.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8331000000000001e-05 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567303+0000", 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0502999999999999e-05 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567369+0000", 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5870000000000005e-05 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567379+0000", 2024-03-26T03:04:36.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0186999999999999e-05 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.308049+0000", 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.74067035299999995 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.308219+0000", 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000169674 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3", 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.308417+0000", 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019777400000000001 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.310073+0000", 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001656554 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.330525+0000", 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020451608 2024-03-26T03:04:36.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332103+0000", 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001578491 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332166+0000", 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2803000000000002e-05 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332266+0000", 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100419 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76595 7.12 7.448c6a52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.299275+0000", 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 164.29924267800001, 2024-03-26T03:04:36.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98390260200000002, 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76595 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299275+0000", 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299275+0000", 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299277+0000", 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0020000000000001e-06 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299337+0000", 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9997999999999997e-05 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299350+0000", 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3040999999999999e-05 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299359+0000", 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8070000000000006e-06 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299430+0000", 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0667000000000006e-05 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299438+0000", 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0360000000000006e-06 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283166+0000", 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.983728622 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283178+0000", 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1429e-05 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76610 7.19 7.a1cdf4d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.322165+0000", 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 164.276352707, 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96114181099999996, 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:36.625 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76610 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322165+0000", 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322165+0000", 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322175+0000", 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7710000000000001e-06 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322210+0000", 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5701999999999998e-05 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322224+0000", 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3427e-05 2024-03-26T03:04:36.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322236+0000", 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2529e-05 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322294+0000", 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7765e-05 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322304+0000", 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7759999999999992e-06 2024-03-26T03:04:36.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283298+0000", 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96099361900000002 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283307+0000", 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2220000000000008e-06 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112085 7.12 7:493de57c:::benchmark_data_smithi066_76917_object11902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:04:36.631 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.319349698, 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.024736326, 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112085 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279170+0000", 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.593e-06 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279174+0000", 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8470000000000003e-06 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279184+0000", 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6080000000000002e-06 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279193+0000", 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7409999999999998e-06 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279262+0000", 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9488999999999998e-05 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279273+0000", 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.08e-05 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.291772+0000", 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0124987969999999 2024-03-26T03:04:36.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.291923+0000", 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015105899999999999 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.292236+0000", 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031284800000000001 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.296791+0000", 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045547749999999996 2024-03-26T03:04:36.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.297868+0000", 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010770319999999999 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303845+0000", 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059774659999999999 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303877+0000", 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1480999999999999e-05 2024-03-26T03:04:36.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303904+0000", 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.779e-05 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112092 7.12 7:4c17d0e1:::benchmark_data_smithi066_76917_object11909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.31202091, 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0192903719999999, 2024-03-26T03:04:36.638 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112092 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286499+0000", 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.717e-06 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286504+0000", 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286516+0000", 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1919999999999999e-05 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286525+0000", 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0359999999999995e-06 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286577+0000", 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2420000000000003e-05 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286590+0000", 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2469999999999999e-05 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.293804+0000", 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0072144949999999 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.293919+0000", 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115445 2024-03-26T03:04:36.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.294102+0000", 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000182339 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298885+0000", 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047836770000000001 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303949+0000", 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005063115 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305695+0000", 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017461379999999999 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305746+0000", 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0810000000000003e-05 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305787+0000", 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-05 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112198 7.12 7:4c79c881:::benchmark_data_smithi066_76917_object12015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.101099545, 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.81316842600000006, 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112198 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497420+0000", 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5859999999999999e-06 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497425+0000", 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2039999999999996e-06 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497438+0000", 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2672e-05 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497445+0000", 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8129999999999996e-06 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497501+0000", 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5503e-05 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497509+0000", 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6899999999999992e-06 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.294932+0000", 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79742365100000001 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295019+0000", 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7113000000000004e-05 2024-03-26T03:04:36.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295174+0000", 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000155014 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298890+0000", 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037151950000000001 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305657+0000", 2024-03-26T03:04:36.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0067669499999999999 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310476+0000", 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048193130000000004 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310539+0000", 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3424999999999997e-05 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310587+0000", 2024-03-26T03:04:36.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7296999999999998e-05 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112201 7.12 7:4b665aee:::benchmark_data_smithi066_76917_object12018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.094603159, 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80677220100000002, 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.652 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112201 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503916+0000", 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0890000000000001e-06 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503920+0000", 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.89e-06 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503929+0000", 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7229999999999999e-06 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503937+0000", 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0080000000000002e-06 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.504005+0000", 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7787000000000007e-05 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.504018+0000", 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.252e-05 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295896+0000", 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79187867700000003 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295978+0000", 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.174e-05 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.296092+0000", 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114016 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298893+0000", 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028010299999999999 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305824+0000", 2024-03-26T03:04:36.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069314609999999999 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310625+0000", 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004800861 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310658+0000", 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2891000000000001e-05 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310687+0000", 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8507999999999999e-05 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157251 7.19 7:9b7ef1c8:::benchmark_data_smithi066_76917_object57068:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.629850769999997, 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.025284095, 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157251 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968672+0000", 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6670000000000004e-06 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968678+0000", 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5309999999999998e-06 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968695+0000", 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7135000000000001e-05 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968705+0000", 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4280000000000002e-06 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.969069+0000", 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036450099999999999 2024-03-26T03:04:36.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.969082+0000", 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2761e-05 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.987851+0000", 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0187692269999999 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.987998+0000", 2024-03-26T03:04:36.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000146957 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.988367+0000", 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000368885 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992132+0000", 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037647359999999999 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993640+0000", 2024-03-26T03:04:36.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001508604 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993833+0000", 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000192404 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993891+0000", 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8391999999999999e-05 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993951+0000", 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9867000000000001e-05 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157261 7.19 7:989eae58:::benchmark_data_smithi066_76917_object57078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.623097465999997, 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.026782426, 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157261 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975421+0000", 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7999999999999993e-07 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975426+0000", 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4510000000000002e-06 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975436+0000", 2024-03-26T03:04:36.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0815999999999999e-05 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975443+0000", 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975508+0000", 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4856000000000004e-05 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975521+0000", 2024-03-26T03:04:36.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2813000000000001e-05 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990406+0000", 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.014885641 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990530+0000", 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012369900000000001 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990814+0000", 2024-03-26T03:04:36.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000283503 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995533+0000", 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047195300000000004 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997610+0000", 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020769709999999999 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002088+0000", 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004477895 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002142+0000", 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4333000000000003e-05 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002203+0000", 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0304999999999999e-05 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157262 7.19 7:9be70472:::benchmark_data_smithi066_76917_object57079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.620609915999999, 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244733150000001, 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157262 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977909+0000", 2024-03-26T03:04:36.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977915+0000", 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3040000000000004e-06 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977925+0000", 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0761000000000001e-05 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977933+0000", 2024-03-26T03:04:36.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2690000000000002e-06 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.978012+0000", 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9157000000000001e-05 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.978024+0000", 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2398999999999999e-05 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992170+0000", 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.014145802 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992257+0000", 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6532000000000004e-05 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992467+0000", 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021020199999999999 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995538+0000", 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003071514 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997648+0000", 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021097339999999998 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002278+0000", 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046300810000000003 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002337+0000", 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9058000000000003e-05 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002381+0000", 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3902000000000001e-05 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157326 7.19 7:9db2bad9:::benchmark_data_smithi066_76917_object57143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.679 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.562055995000001, 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96608487600000004, 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157326 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:04:36.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036464+0000", 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.486e-06 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036474+0000", 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244999999999999e-05 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036490+0000", 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5928000000000001e-05 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036499+0000", 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6670000000000002e-06 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036567+0000", 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8338999999999997e-05 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036578+0000", 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0682999999999999e-05 2024-03-26T03:04:36.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994011+0000", 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95743293299999999 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994083+0000", 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2106000000000004e-05 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994259+0000", 2024-03-26T03:04:36.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017634500000000001 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997687+0000", 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034280309999999998 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002229+0000", 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045413550000000004 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002484+0000", 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025479300000000001 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002515+0000", 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1798999999999998e-05 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002547+0000", 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1165999999999999e-05 2024-03-26T03:04:36.686 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157394 7.19 7:9d416a04:::benchmark_data_smithi066_76917_object57211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.503994210999998, 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90827457, 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.687 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157394 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094525+0000", 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1820000000000001e-06 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094531+0000", 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6420000000000001e-06 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094543+0000", 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1473e-05 2024-03-26T03:04:36.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094549+0000", 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3749999999999999e-06 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094591+0000", 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2333999999999998e-05 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094599+0000", 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1740000000000003e-06 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995590+0000", 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.900991404 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995703+0000", 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112657 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995979+0000", 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027668000000000001 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002238+0000", 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006258581 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002425+0000", 2024-03-26T03:04:36.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018703599999999999 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002629+0000", 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000204358 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002755+0000", 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012557900000000001 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002798+0000", 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3095e-05 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157412 7.19 7:99cf3667:::benchmark_data_smithi066_76917_object57229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.489535146999998, 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89402950199999998, 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.694 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157412 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108984+0000", 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.06e-06 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108988+0000", 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108996+0000", 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999992e-06 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109003+0000", 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0099999999999998e-06 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109077+0000", 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4035000000000005e-05 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109090+0000", 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2683e-05 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997735+0000", 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88864550600000003 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997869+0000", 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013324800000000001 2024-03-26T03:04:36.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.998178+0000", 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030980100000000001 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002441+0000", 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042626239999999996 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002581+0000", 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014011500000000001 2024-03-26T03:04:36.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002909+0000", 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032837200000000001 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002971+0000", 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1194999999999999e-05 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003012+0000", 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1405000000000001e-05 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157451 7.19 7:9ac14505:::benchmark_data_smithi066_76917_object57268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.461582601000003, 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87424788600000003, 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.702 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157451 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136936+0000", 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3880000000000001e-06 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136941+0000", 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9099999999999996e-06 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136956+0000", 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4181e-05 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136971+0000", 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5316999999999999e-05 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.137210+0000", 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023869700000000001 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.137224+0000", 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4207e-05 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.999829+0000", 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86260561300000005 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.999973+0000", 2024-03-26T03:04:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014313800000000001 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.000235+0000", 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026196699999999999 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002445+0000", 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022107519999999999 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002855+0000", 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040992999999999997 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011105+0000", 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082499259999999994 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011144+0000", 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8408000000000002e-05 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011183+0000", 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9452000000000001e-05 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157623 7.19 7:9ae45d3c:::benchmark_data_smithi066_76917_object57440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.297849902000003, 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.71074928199999998, 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157623 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:04:36.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300670+0000", 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300674+0000", 2024-03-26T03:04:36.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4469999999999999e-06 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300678+0000", 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.163e-06 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300717+0000", 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300724+0000", 2024-03-26T03:04:36.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0079999999999996e-06 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003059+0000", 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70233499399999999 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003162+0000", 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103136 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003429+0000", 2024-03-26T03:04:36.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000266395 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011275+0000", 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078460179999999997 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011290+0000", 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5318000000000001e-05 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011330+0000", 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9536999999999998e-05 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011379+0000", 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9342000000000003e-05 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011417+0000", 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8158999999999998e-05 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157629 7.19 7:9d222119:::benchmark_data_smithi066_76917_object57446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.291820164999997, 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70924085000000003, 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157629 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306699+0000", 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.702e-06 2024-03-26T03:04:36.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306705+0000", 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1109999999999996e-06 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306718+0000", 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2378000000000001e-05 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306725+0000", 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2250000000000003e-06 2024-03-26T03:04:36.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306795+0000", 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0272999999999996e-05 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306806+0000", 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0883000000000001e-05 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005375+0000", 2024-03-26T03:04:36.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69856929300000004 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005481+0000", 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105763 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005761+0000", 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027942199999999998 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011295+0000", 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055347560000000001 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011461+0000", 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000166088 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015840+0000", 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043782969999999997 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015892+0000", 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2256999999999998e-05 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015938+0000", 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6402000000000001e-05 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157655 7.19 7:9b64c26d:::benchmark_data_smithi066_76917_object57472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.722 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.250967971999998, 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67836821199999997, 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157655 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:04:36.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347551+0000", 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.85e-07 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347554+0000", 2024-03-26T03:04:36.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1889999999999999e-06 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347564+0000", 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8030000000000008e-06 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347568+0000", 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9520000000000002e-06 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347621+0000", 2024-03-26T03:04:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2417000000000003e-05 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347628+0000", 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5120000000000002e-06 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.008556+0000", 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66092736900000004 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.008679+0000", 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012314300000000001 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.009015+0000", 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033638899999999998 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011425+0000", 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00241004 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015997+0000", 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045720439999999999 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025826+0000", 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0098289510000000007 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025878+0000", 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1316000000000001e-05 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025918+0000", 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0302000000000001e-05 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157664 7.19 7:9fd64715:::benchmark_data_smithi066_76917_object57481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.237410826999998, 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66498246000000005, 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:36.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157664 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:36.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361108+0000", 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6899999999999997e-07 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361110+0000", 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9270000000000001e-06 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:36.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361117+0000", 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9659999999999999e-06 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361122+0000", 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2459999999999997e-06 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361169+0000", 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6907999999999999e-05 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:36.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361176+0000", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5129999999999999e-06 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011489+0000", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65031320299999995 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011597+0000", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107469 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011895+0000", 2024-03-26T03:04:36.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029828400000000001 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015953+0000", 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040577169999999997 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.016041+0000", 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8707000000000003e-05 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:36.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026023+0000", 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0099818719999999993 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026054+0000", 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0925e-05 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026089+0000", 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5054e-05 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.735 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:36.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:36.736 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:36.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:36.838 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:36.865 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:36.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:36.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:36.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:37.091 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:37.092 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:37.129 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:37.191 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: 6.c scrub starts 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: 6.c scrub ok 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: pgmap v235: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 487 op/s 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: 5.a scrub starts 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[37920]: 5.a scrub ok 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: 6.c scrub starts 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: 6.c scrub ok 2024-03-26T03:04:37.192 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: pgmap v235: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 487 op/s 2024-03-26T03:04:37.193 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T03:04:37.193 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T03:04:37.193 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:04:37.193 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:04:37.193 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: 5.a scrub starts 2024-03-26T03:04:37.193 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:37 smithi066 ceph-mon[29666]: 5.a scrub ok 2024-03-26T03:04:37.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:37.257 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:37.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:37.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:37.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: 6.c scrub starts 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: 6.c scrub ok 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: pgmap v235: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 487 op/s 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:04:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: 5.a scrub starts 2024-03-26T03:04:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:37 smithi192 ceph-mon[33110]: 5.a scrub ok 2024-03-26T03:04:37.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:37.556 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:04:37.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:04:37.575 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:04:37.594 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:04:37.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:37.613 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:04:37.633 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:04:37.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:37.653 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:04:37.675 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:04:37.695 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:04:37.696 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:04:37.696 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:04:37.696 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:04:37.696 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:04:37.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:04:37.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:04:37.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:04:37.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:04:37.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.771 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:37.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:37.817 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:37.872 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:37.899 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:37.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:38.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:38.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:38.114 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:04:38.127 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:38.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:38.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 3.1 scrub starts 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 3.1 scrub ok 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1273606362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1446013569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:38.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T03:04:38.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:38.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:38.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:04:38.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 3.1 scrub starts 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 3.1 scrub ok 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1273606362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1446013569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:04:38.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 3.1 scrub starts 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 3.1 scrub ok 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1273606362' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1446013569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:38.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T03:04:38.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:38.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:38.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:38.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:38.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.664 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:38.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:38.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:38.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:38.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:38.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:38.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:39.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:39.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:39.143 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:39.170 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:39.199 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:04:39.199 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:04:39.199 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: pgmap v238: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:39.199 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:04:39.199 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:04:39.199 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1273606362' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:04:39.200 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-mon[33110]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T03:04:39.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:39.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:39.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: pgmap v238: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1273606362' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:04:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[37920]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: pgmap v238: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1273606362' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:04:39.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:04:39.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:39 smithi066 ceph-mon[29666]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T03:04:39.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:39.425 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:39.438 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:39.438 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:39.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:04:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:04:39.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:39.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:39.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:39.679 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:39.781 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:39.808 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:39.910 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:39.937 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:39.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:40.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:40.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:40.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:04:40.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:40.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:40.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:40.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:40 smithi192 ceph-mon[33110]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T03:04:40.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:40.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:40.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:40.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:40 smithi066 ceph-mon[29666]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T03:04:40.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:40.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:40 smithi066 ceph-mon[37920]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T03:04:40.360 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:40.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:40.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:40.462 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:04:40.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:40.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:40.671 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:40.700 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:40.730 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:40.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:40.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:40.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:40.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:40.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:40.956 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_0' removed 2024-03-26T03:04:41.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:41.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:41.119 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:04:41.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:41.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:41.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:41.348 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 7.c deep-scrub starts 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 7.c deep-scrub ok 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: pgmap v242: 168 pgs: 4 remapped+peering, 6 unknown, 158 active+clean; 216 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 2.12 deep-scrub starts 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 2.12 deep-scrub ok 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 5.10 deep-scrub starts 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: 5.10 deep-scrub ok 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/377069535' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/377069535' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:04:41.349 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:41 smithi192 ceph-mon[33110]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T03:04:41.357 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 7.c deep-scrub starts 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 7.c deep-scrub ok 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: pgmap v242: 168 pgs: 4 remapped+peering, 6 unknown, 158 active+clean; 216 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 2.12 deep-scrub starts 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 2.12 deep-scrub ok 2024-03-26T03:04:41.414 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 5.10 deep-scrub starts 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: 5.10 deep-scrub ok 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/377069535' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/377069535' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[29666]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 7.c deep-scrub starts 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 7.c deep-scrub ok 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: pgmap v242: 168 pgs: 4 remapped+peering, 6 unknown, 158 active+clean; 216 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:41.415 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 2.12 deep-scrub starts 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 2.12 deep-scrub ok 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 5.10 deep-scrub starts 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: 5.10 deep-scrub ok 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/377069535' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/377069535' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:04:41.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:41 smithi066 ceph-mon[37920]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T03:04:41.432 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:04:41.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:04:41.432 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T03:04:41.433 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-03-26T03:04:41.433 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-26T03:04:41.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:41.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:41.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:41.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:41.730 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:41.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:41.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:41.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:41.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:41.939 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:41.939 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:41.986 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:42.014 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:42.115 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:42.143 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:42.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: 2.19 scrub starts 2024-03-26T03:04:42.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: 2.19 scrub ok 2024-03-26T03:04:42.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: 4.1b scrub starts 2024-03-26T03:04:42.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: 4.1b scrub ok 2024-03-26T03:04:42.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: 6.8 scrub starts 2024-03-26T03:04:42.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: 6.8 scrub ok 2024-03-26T03:04:42.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:42 smithi192 ceph-mon[33110]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T03:04:42.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:42.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:42.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:42.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:42.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:42.525 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:42.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:04:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: 2.19 scrub starts 2024-03-26T03:04:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: 2.19 scrub ok 2024-03-26T03:04:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: 4.1b scrub starts 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: 4.1b scrub ok 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: 6.8 scrub starts 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: 6.8 scrub ok 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[29666]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: 2.19 scrub starts 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: 2.19 scrub ok 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: 4.1b scrub starts 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: 4.1b scrub ok 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: 6.8 scrub starts 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: 6.8 scrub ok 2024-03-26T03:04:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:42 smithi066 ceph-mon[37920]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T03:04:42.627 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:42.654 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:42.739 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:04:42.740 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:04:42.740 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-26T03:04:42.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-26T03:04:42.756 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:42.759 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:42.786 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:04:42.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:42.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:42.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:42.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:43.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:43.079 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:43.180 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:43.208 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:43.213 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 1.5 scrub starts 2024-03-26T03:04:43.213 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 1.5 scrub ok 2024-03-26T03:04:43.213 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: pgmap v245: 142 pgs: 2 activating+undersized+degraded+remapped, 4 remapped+peering, 2 active+undersized+remapped, 2 activating+undersized+remapped, 132 active+clean; 530 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3/690 objects degraded (0.435%); 2/690 objects misplaced (0.290%) 2024-03-26T03:04:43.213 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:04:43.214 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:04:43.214 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 1.9 scrub starts 2024-03-26T03:04:43.214 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 1.9 scrub ok 2024-03-26T03:04:43.214 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:04:43.214 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:04:43.214 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:43 smithi192 ceph-mon[33110]: Health check failed: Degraded data redundancy: 3/690 objects degraded (0.435%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 1.5 scrub starts 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 1.5 scrub ok 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: pgmap v245: 142 pgs: 2 activating+undersized+degraded+remapped, 4 remapped+peering, 2 active+undersized+remapped, 2 activating+undersized+remapped, 132 active+clean; 530 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3/690 objects degraded (0.435%); 2/690 objects misplaced (0.290%) 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 1.9 scrub starts 2024-03-26T03:04:43.294 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 1.9 scrub ok 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[29666]: Health check failed: Degraded data redundancy: 3/690 objects degraded (0.435%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 1.5 scrub starts 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 1.5 scrub ok 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: pgmap v245: 142 pgs: 2 activating+undersized+degraded+remapped, 4 remapped+peering, 2 active+undersized+remapped, 2 activating+undersized+remapped, 132 active+clean; 530 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3/690 objects degraded (0.435%); 2/690 objects misplaced (0.290%) 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 1.9 scrub starts 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 1.9 scrub ok 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:04:43.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:04:43.296 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-mon[37920]: Health check failed: Degraded data redundancy: 3/690 objects degraded (0.435%), 2 pgs degraded (PG_DEGRADED) 2024-03-26T03:04:43.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:43.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:43.341 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:04:43.360 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:04:43.379 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:04:43.399 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:04:43.419 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:04:43.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:43.441 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:04:43.465 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:04:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:43.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:43.484 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:04:43.485 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:04:43.486 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:04:43.567 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:43.594 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:43.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:43.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:43.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:43.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:43.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:43.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:43.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:44.009 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:04:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:04:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:04:44.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:44.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:44.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:44.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:27526 7.5 7:a2f1966b:::benchmark_data_smithi066_76917_object27525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:14.791040+0000", 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 209.56261381100001, 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18028749399999999, 2024-03-26T03:04:44.356 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 27526 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791040+0000", 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791040+0000", 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791042+0000", 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8610000000000001e-06 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791106+0000", 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3446000000000002e-05 2024-03-26T03:04:44.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791116+0000", 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0883999999999999e-05 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791122+0000", 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.711e-06 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791168+0000", 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5958000000000003e-05 2024-03-26T03:04:44.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791260+0000", 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2096999999999996e-05 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791357+0000", 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6366999999999994e-05 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.793034+0000", 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016775449999999999 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797742+0000", 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047082690000000002 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.971228+0000", 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.173485168 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.971292+0000", 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4892000000000001e-05 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.971328+0000", 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5296000000000002e-05 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:27528 7.5 7:acac9c12:::benchmark_data_smithi066_76917_object27527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:14.791328+0000", 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 209.562326399, 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.180093482, 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 27528 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791328+0000", 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791328+0000", 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791328+0000", 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8700000000000005e-07 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791399+0000", 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0508999999999999e-05 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791405+0000", 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0800000000000002e-06 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.791409+0000", 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2470000000000002e-06 2024-03-26T03:04:44.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.792124+0000", 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000714709 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.792239+0000", 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114613 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.792321+0000", 2024-03-26T03:04:44.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2429000000000006e-05 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.793153+0000", 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00083237800000000005 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797794+0000", 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046411020000000002 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.971360+0000", 2024-03-26T03:04:44.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.173565209 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.971395+0000", 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5342999999999997e-05 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.971421+0000", 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6176000000000001e-05 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30125 7.5 7:a05450a2:::benchmark_data_smithi066_76917_object30124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.591941+0000", 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 207.76171340900001, 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16553153800000001, 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30125 2024-03-26T03:04:44.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591941+0000", 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591941+0000", 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591942+0000", 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.274e-06 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591997+0000", 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4965e-05 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592009+0000", 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1815e-05 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592014+0000", 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6160000000000001e-06 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592064+0000", 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9252999999999998e-05 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592198+0000", 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013461500000000001 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592354+0000", 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015609099999999999 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.594795+0000", 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024406369999999998 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.594809+0000", 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3723999999999999e-05 2024-03-26T03:04:44.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757298+0000", 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16248926599999999 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757389+0000", 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1459000000000006e-05 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757472+0000", 2024-03-26T03:04:44.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2823000000000003e-05 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30135 7.5 7:a0d7ca16:::benchmark_data_smithi066_76917_object30134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.596049+0000", 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 207.757604934, 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16158143799999999, 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.375 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30135 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596049+0000", 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596049+0000", 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596055+0000", 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3339999999999998e-06 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596104+0000", 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8191e-05 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596113+0000", 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7480000000000005e-06 2024-03-26T03:04:44.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596119+0000", 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8069999999999996e-06 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596159+0000", 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9968000000000001e-05 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596256+0000", 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.713e-05 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.596354+0000", 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7981999999999998e-05 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.597204+0000", 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00084963499999999995 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.597457+0000", 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025306999999999999 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757529+0000", 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16007226299999999 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757592+0000", 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2546999999999996e-05 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757631+0000", 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8763000000000001e-05 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31286 7.5 7:a7556d43:::benchmark_data_smithi066_76917_object31285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.738365+0000", 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 206.61528866699999, 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14416979099999999, 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.381 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31286 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738365+0000", 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738365+0000", 2024-03-26T03:04:44.383 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.4 scrub starts 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.4 scrub ok 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.7 scrub starts 2024-03-26T03:04:44.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.7 scrub ok 2024-03-26T03:04:44.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1966591118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:44.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:04:44.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:04:44.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.8 scrub starts 2024-03-26T03:04:44.385 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[37920]: 1.8 scrub ok 2024-03-26T03:04:44.385 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.4 scrub starts 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.4 scrub ok 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.7 scrub starts 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.7 scrub ok 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1966591118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.8 scrub starts 2024-03-26T03:04:44.386 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:44 smithi066 ceph-mon[29666]: 1.8 scrub ok 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738368+0000", 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5179999999999999e-06 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738422+0000", 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4412e-05 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738433+0000", 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0947999999999999e-05 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738441+0000", 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1079999999999993e-06 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738522+0000", 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0955e-05 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738654+0000", 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013184199999999999 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738810+0000", 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015606999999999999 2024-03-26T03:04:44.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741214+0000", 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024038729999999999 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741254+0000", 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0312999999999998e-05 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.882470+0000", 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14121599500000001 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.882506+0000", 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5459000000000001e-05 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.882535+0000", 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9298e-05 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.391 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:64284 7.5 7:a5be7971:::benchmark_data_smithi066_76917_object64283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:40.119248+0000", 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 184.23440591100001, 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.150313118, 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.392 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 64284 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119248+0000", 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119248+0000", 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119250+0000", 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9999999999999999e-06 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119331+0000", 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1014999999999996e-05 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119344+0000", 2024-03-26T03:04:44.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3050999999999999e-05 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119350+0000", 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2219999999999999e-06 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119408+0000", 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7667000000000002e-05 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119534+0000", 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125764 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.119700+0000", 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016594299999999999 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124072+0000", 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043719559999999998 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124108+0000", 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6161999999999997e-05 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269440+0000", 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.145331817 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269509+0000", 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9497000000000002e-05 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269561+0000", 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2024000000000002e-05 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:64287 7.5 7:ac7e86f4:::benchmark_data_smithi066_76917_object64286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:40.120347+0000", 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 184.23330705399999, 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14932589900000001, 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.398 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 64287 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.120347+0000", 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.120347+0000", 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.120348+0000", 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4139999999999999e-06 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.120483+0000", 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013419500000000001 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.120497+0000", 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4197e-05 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.120505+0000", 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8220000000000007e-06 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.121851+0000", 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013465040000000001 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.121978+0000", 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012717499999999999 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.122105+0000", 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127162 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124191+0000", 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020851419999999999 2024-03-26T03:04:44.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.129852+0000", 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056613649999999998 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269609+0000", 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13975707000000001 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269645+0000", 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5706e-05 2024-03-26T03:04:44.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269673+0000", 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8147e-05 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:64291 7.5 7:af27228d:::benchmark_data_smithi066_76917_object64290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:40.124179+0000", 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 184.229475154, 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14562641800000001, 2024-03-26T03:04:44.404 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 64291 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124179+0000", 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124179+0000", 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124181+0000", 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3e-06 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124299+0000", 2024-03-26T03:04:44.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117337 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124311+0000", 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2558000000000001e-05 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124318+0000", 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.584e-06 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124363+0000", 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4888000000000001e-05 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124458+0000", 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5305999999999999e-05 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.124558+0000", 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100517 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.126017+0000", 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014591059999999999 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.129906+0000", 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038887840000000002 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269716+0000", 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13981016299999999 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269769+0000", 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2337999999999999e-05 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:40.269805+0000", 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6536999999999999e-05 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.410 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75530 7.5 7:ae8760c0:::benchmark_data_smithi066_76917_object75529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.284446+0000", 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 174.06920803, 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12066189500000001, 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75530 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284446+0000", 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284446+0000", 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284450+0000", 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8999999999999999e-06 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284490+0000", 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9694000000000003e-05 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284514+0000", 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4206000000000002e-05 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284527+0000", 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3317e-05 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284657+0000", 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012984699999999999 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284810+0000", 2024-03-26T03:04:44.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:04:44.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:04:44.414 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.4 scrub starts 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.4 scrub ok 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.7 scrub starts 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.7 scrub ok 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1966591118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.8 scrub starts 2024-03-26T03:04:44.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:44 smithi192 ceph-mon[33110]: 1.8 scrub ok 2024-03-26T03:04:44.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015290799999999999 2024-03-26T03:04:44.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.284983+0000", 2024-03-26T03:04:44.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000173 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.287987+0000", 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030040459999999998 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.405091+0000", 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.117104374 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.405108+0000", 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6603e-05 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75530 7.15 7:ae8760c0:::benchmark_data_smithi066_76917_object75529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.308783+0000", 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 174.04487114599999, 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0285267060000001, 2024-03-26T03:04:44.418 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75530 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308783+0000", 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308783+0000", 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308785+0000", 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1320000000000001e-06 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308823+0000", 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8393000000000003e-05 2024-03-26T03:04:44.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308836+0000", 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2809000000000001e-05 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308845+0000", 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0080000000000008e-06 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.405271+0000", 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.096426019000000002 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.405282+0000", 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1187e-05 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.331876+0000", 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92659350600000001 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332602+0000", 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00072563100000000002 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.337254+0000", 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004652303 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.337310+0000", 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5717999999999997e-05 2024-03-26T03:04:44.423 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76576 7.17 7.8d247657 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.289324+0000", 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 172.06432965900001, 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99226051000000004, 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.424 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76576 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289324+0000", 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289324+0000", 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289326+0000", 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.725e-06 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289373+0000", 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999997e-05 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289384+0000", 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1353e-05 2024-03-26T03:04:44.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289391+0000", 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7479999999999996e-06 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289482+0000", 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1595999999999995e-05 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289495+0000", 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2132e-05 2024-03-26T03:04:44.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281571+0000", 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99207615000000005 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281585+0000", 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4185999999999999e-05 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.428 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76577 7.17 7.e186eaf7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.289404+0000", 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 172.064250254, 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99222399800000005, 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76577 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289404+0000", 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289404+0000", 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289405+0000", 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4000000000000001e-07 2024-03-26T03:04:44.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289438+0000", 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3522000000000002e-05 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289445+0000", 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3089999999999998e-06 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289449+0000", 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1010000000000002e-06 2024-03-26T03:04:44.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289523+0000", 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3114999999999994e-05 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289528+0000", 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3460000000000001e-06 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281617+0000", 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99208908500000004 2024-03-26T03:04:44.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281628+0000", 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.078e-05 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76589 7.17 7.fbb76497 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.289456+0000", 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 172.064198331, 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99220566700000001, 2024-03-26T03:04:44.433 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76589 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289456+0000", 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289456+0000", 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289456+0000", 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4000000000000001e-07 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289488+0000", 2024-03-26T03:04:44.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1158999999999998e-05 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289494+0000", 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1480000000000003e-06 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289498+0000", 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6e-06 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289552+0000", 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4021999999999999e-05 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.289559+0000", 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3459999999999999e-06 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281654+0000", 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99209541899999998 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281661+0000", 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.233e-06 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76591 7.11 7.7e39b7d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.296459+0000", 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 172.05719515000001, 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98504224799999995, 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76591 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296459+0000", 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296459+0000", 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296460+0000", 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.128e-06 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296521+0000", 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0575999999999997e-05 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296529+0000", 2024-03-26T03:04:44.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0749999999999998e-06 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296535+0000", 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2729999999999999e-06 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296585+0000", 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0238999999999998e-05 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.296591+0000", 2024-03-26T03:04:44.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1299999999999998e-06 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281490+0000", 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98489873299999997 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281501+0000", 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1094000000000001e-05 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76620 7.15 7:adffade7:::benchmark_data_smithi066_76917_object76619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.342684+0000", 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 172.01096957799999, 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.090062579, 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76620 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.443 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342684+0000", 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342684+0000", 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342692+0000", 2024-03-26T03:04:44.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7889999999999995e-06 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342728+0000", 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5933999999999998e-05 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342737+0000", 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7539999999999995e-06 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342744+0000", 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.922e-06 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342792+0000", 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8566999999999998e-05 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.342801+0000", 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0210000000000005e-06 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281611+0000", 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.938809703 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281622+0000", 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0738e-05 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.413490+0000", 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1318681719999999 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.413646+0000", 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000156107 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.413870+0000", 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000223935 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.416369+0000", 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024991380000000001 2024-03-26T03:04:44.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.419200+0000", 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028310340000000001 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.430893+0000", 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011692352 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.430976+0000", 2024-03-26T03:04:44.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3634000000000006e-05 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:54.432747+0000", 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017707790000000001 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76628 7.17 7.1e883bf7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.356805+0000", 2024-03-26T03:04:44.450 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.99684863100001, 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92488745000000006, 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76628 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.451 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.356805+0000", 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.356805+0000", 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.356808+0000", 2024-03-26T03:04:44.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1210000000000001e-06 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.356877+0000", 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9717999999999999e-05 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.356889+0000", 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1249e-05 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.356897+0000", 2024-03-26T03:04:44.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1880000000000002e-06 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.357000+0000", 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103761 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.357010+0000", 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8309999999999996e-06 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281685+0000", 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92467486200000004 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281693+0000", 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7200000000000006e-06 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76630 7.17 7.e7c84397 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:04:44.455 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.422234+0000", 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.931420024, 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.85950340199999997, 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76630 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422234+0000", 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422234+0000", 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422236+0000", 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9939999999999999e-06 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422291+0000", 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4709000000000001e-05 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422300+0000", 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.431e-06 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422306+0000", 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8980000000000001e-06 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422372+0000", 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6157999999999999e-05 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.422382+0000", 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3109999999999995e-06 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281729+0000", 2024-03-26T03:04:44.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.85934741599999998 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.281737+0000", 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4849999999999992e-06 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:110053 7.15 7:a8af5a7e:::benchmark_data_smithi066_76917_object9870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:48.326565+0000", 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 116.02708911400001, 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11038972900000001, 2024-03-26T03:04:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 110053 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326565+0000", 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326565+0000", 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326567+0000", 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9800000000000001e-06 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326573+0000", 2024-03-26T03:04:44.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9939999999999997e-06 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326585+0000", 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2328e-05 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326594+0000", 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3790000000000006e-06 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326652+0000", 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.791e-05 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.326779+0000", 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127704 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.327051+0000", 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027174499999999999 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.328176+0000", 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001124945 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.328532+0000", 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035634300000000001 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.436882+0000", 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10834959199999999 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.436914+0000", 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1851999999999999e-05 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:48.436955+0000", 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0957000000000001e-05 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.466 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:142783 7.5 7:a52f4218:::benchmark_data_smithi066_76917_object42600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:29.445649+0000", 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 74.908005548999995, 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13952932400000001, 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 142783 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445649+0000", 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445649+0000", 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445650+0000", 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9350000000000001e-06 2024-03-26T03:04:44.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445657+0000", 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.173e-06 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445672+0000", 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5764e-05 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445681+0000", 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4509999999999992e-06 2024-03-26T03:04:44.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445767+0000", 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6478999999999997e-05 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.445948+0000", 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018031199999999999 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446321+0000", 2024-03-26T03:04:44.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000373488 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.450922+0000", 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046007670000000004 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.454037+0000", 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00311547 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.585081+0000", 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13104406199999999 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.585135+0000", 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3754000000000001e-05 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.585178+0000", 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2669000000000001e-05 2024-03-26T03:04:44.472 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:142789 7.5 7:a6d1a351:::benchmark_data_smithi066_76917_object42606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:29.447753+0000", 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 74.905901384999993, 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13753670600000001, 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:04:44.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 142789 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.447753+0000", 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.447753+0000", 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.447755+0000", 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3590000000000002e-06 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.447769+0000", 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4036000000000001e-05 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.447781+0000", 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2224e-05 2024-03-26T03:04:44.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.447790+0000", 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5909999999999996e-06 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.448430+0000", 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064039800000000003 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.448547+0000", 2024-03-26T03:04:44.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117037 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.448874+0000", 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032688299999999998 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.451117+0000", 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022424490000000001 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.456866+0000", 2024-03-26T03:04:44.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057497729999999997 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.585215+0000", 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.128348668 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.585258+0000", 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3297000000000002e-05 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:04:44.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.585289+0000", 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0991000000000002e-05 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:04:44.479 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:04:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.481 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:44.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:44.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:44.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:44.608 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:44.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:44.734 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:44.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:44.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:44.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:44.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:44.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:45.083 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:45.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:45.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:45.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:04:45.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:45.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[29666]: 1.6 scrub starts 2024-03-26T03:04:45.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[29666]: 1.6 scrub ok 2024-03-26T03:04:45.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[29666]: pgmap v246: 142 pgs: 2 activating+undersized+degraded+remapped, 1 active+undersized+remapped, 2 activating+undersized+remapped, 137 active+clean; 268 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3/687 objects degraded (0.437%); 2/687 objects misplaced (0.291%) 2024-03-26T03:04:45.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[29666]: 4.1c scrub starts 2024-03-26T03:04:45.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[29666]: 4.1c scrub ok 2024-03-26T03:04:45.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T03:04:45.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[37920]: 1.6 scrub starts 2024-03-26T03:04:45.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[37920]: 1.6 scrub ok 2024-03-26T03:04:45.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[37920]: pgmap v246: 142 pgs: 2 activating+undersized+degraded+remapped, 1 active+undersized+remapped, 2 activating+undersized+remapped, 137 active+clean; 268 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3/687 objects degraded (0.437%); 2/687 objects misplaced (0.291%) 2024-03-26T03:04:45.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[37920]: 4.1c scrub starts 2024-03-26T03:04:45.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[37920]: 4.1c scrub ok 2024-03-26T03:04:45.300 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:45 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T03:04:45.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:45.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:45.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:45.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:45 smithi192 ceph-mon[33110]: 1.6 scrub starts 2024-03-26T03:04:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:45 smithi192 ceph-mon[33110]: 1.6 scrub ok 2024-03-26T03:04:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:45 smithi192 ceph-mon[33110]: pgmap v246: 142 pgs: 2 activating+undersized+degraded+remapped, 1 active+undersized+remapped, 2 activating+undersized+remapped, 137 active+clean; 268 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3/687 objects degraded (0.437%); 2/687 objects misplaced (0.291%) 2024-03-26T03:04:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:45 smithi192 ceph-mon[33110]: 4.1c scrub starts 2024-03-26T03:04:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:45 smithi192 ceph-mon[33110]: 4.1c scrub ok 2024-03-26T03:04:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:45 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-26T03:04:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:45.589 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:45.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:45.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:45.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:45.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:45.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:45.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:45.994 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:45.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:46.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:46.125 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:46.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:46.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:46.353 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:46.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 5.14 scrub starts 2024-03-26T03:04:46.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 5.14 scrub ok 2024-03-26T03:04:46.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:04:46.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:04:46.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 5.14 scrub starts 2024-03-26T03:04:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 5.14 scrub ok 2024-03-26T03:04:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:04:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:04:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:04:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:04:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:04:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:04:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:04:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:46 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:04:46.381 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:46.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 5.14 scrub starts 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 5.14 scrub ok 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:04:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:04:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:04:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:46 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:04:46.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:46.608 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:46.638 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:46.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:46.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:46.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:46.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:46.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:46.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:47.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:47.043 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:04:47.116 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:47.147 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:47.248 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:47.276 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[37920]: pgmap v247: 142 pgs: 142 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[37920]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/687 objects degraded (0.437%), 2 pgs degraded) 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[37920]: 2.13 deep-scrub starts 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[29666]: pgmap v247: 142 pgs: 142 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[29666]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/687 objects degraded (0.437%), 2 pgs degraded) 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:04:47.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:47 smithi066 ceph-mon[29666]: 2.13 deep-scrub starts 2024-03-26T03:04:47.378 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:47.406 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:47 smithi192 ceph-mon[33110]: pgmap v247: 142 pgs: 142 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:47 smithi192 ceph-mon[33110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/687 objects degraded (0.437%), 2 pgs degraded) 2024-03-26T03:04:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:47 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:04:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:47 smithi192 ceph-mon[33110]: 2.13 deep-scrub starts 2024-03-26T03:04:47.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:47.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:47.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:47.661 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:47.763 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:47.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:47.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:47.920 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:47.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:48.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:48.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:48.143 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-26T03:04:48.144 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:04:48.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:48.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:48.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:48.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:48.412 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.442 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:48.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 2.13 deep-scrub ok 2024-03-26T03:04:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2861636906' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 2.c deep-scrub starts 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 2.c deep-scrub ok 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2528690139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:48 smithi192 ceph-mon[33110]: from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:48.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:48.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 2.13 deep-scrub ok 2024-03-26T03:04:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2861636906' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 2.c deep-scrub starts 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 2.c deep-scrub ok 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2528690139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[29666]: from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 2.13 deep-scrub ok 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2861636906' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 2.c deep-scrub starts 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 2.c deep-scrub ok 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2528690139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:48 smithi066 ceph-mon[37920]: from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:04:48.681 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:48.710 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:48.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:48.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:48.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:48.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:48.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:49.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:49.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:49.207 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:49.234 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:49.297 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: pgmap v248: 142 pgs: 142 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2861636906' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: 5.9 deep-scrub starts 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[37920]: 5.9 deep-scrub ok 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: pgmap v248: 142 pgs: 142 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2861636906' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:04:49.298 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:04:49.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T03:04:49.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: 5.9 deep-scrub starts 2024-03-26T03:04:49.299 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:49 smithi066 ceph-mon[29666]: 5.9 deep-scrub ok 2024-03-26T03:04:49.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:49.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:49.467 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:49.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:04:49.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: pgmap v248: 142 pgs: 142 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:04:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2861636906' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:04:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:04:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T03:04:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: 5.9 deep-scrub starts 2024-03-26T03:04:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:49 smithi192 ceph-mon[33110]: 5.9 deep-scrub ok 2024-03-26T03:04:49.495 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:49.597 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:49.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:49.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:49.754 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:49.783 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:49.783 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:49.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:49.883 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:49.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:49.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:50.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:50.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:50.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:50.238 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:50 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:04:50.238 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:50 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:50.238 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:50 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:04:50.239 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:50 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:50.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:50.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:50.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:50.402 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:50 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:04:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:50 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:04:50.504 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:50.531 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:50.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:50.659 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:50.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:50.790 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:50.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:50.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:50.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:51.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:51.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:51.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:51.161 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_1' created 2024-03-26T03:04:51.183 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:51.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:51.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:51.390 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:51 smithi066 ceph-mon[29666]: pgmap v250: 142 pgs: 142 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:51.390 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:51 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/226687996' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:04:51.390 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:51 smithi066 ceph-mon[29666]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:04:51.390 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:51 smithi066 ceph-mon[37920]: pgmap v250: 142 pgs: 142 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:51.391 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:51 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/226687996' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:04:51.391 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:51 smithi066 ceph-mon[37920]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:04:51.415 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.445 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:51.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:51 smithi192 ceph-mon[33110]: pgmap v250: 142 pgs: 142 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:51.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:51 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/226687996' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:04:51.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:51 smithi192 ceph-mon[33110]: from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:04:51.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:51.574 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:51.676 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:51.704 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:51.735 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:04:51.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:51.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:51.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:51.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:51.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:51.987 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-26T03:04:52.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:52.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:52.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:52.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:52.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:52.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:52.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:52 smithi066 ceph-mon[29666]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T03:04:52.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:52 smithi066 ceph-mon[29666]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T03:04:52.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:52 smithi066 ceph-mon[37920]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T03:04:52.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:52 smithi066 ceph-mon[37920]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T03:04:52.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:52.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:52 smithi192 ceph-mon[33110]: from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T03:04:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:52 smithi192 ceph-mon[33110]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T03:04:52.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:52.604 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:52.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:52.733 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:52.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:52.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:52.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:52.982 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:52.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:52.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:53.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:53.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:53.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:53.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:53.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-mon[29666]: pgmap v252: 158 pgs: 16 unknown, 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:04:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-mon[29666]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T03:04:53.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-mon[37920]: pgmap v252: 158 pgs: 16 unknown, 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:53.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:04:53.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-mon[37920]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T03:04:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:53.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:53.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:53 smithi192 ceph-mon[33110]: pgmap v252: 158 pgs: 16 unknown, 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-03-26T03:04:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:53 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:04:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:53 smithi192 ceph-mon[33110]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T03:04:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.501 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:53.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:53.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:53.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:53.730 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:53.760 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:53.856 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:04:53.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:53.876 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:04:53.876 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:04:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:04:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:04:53.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:53.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:53.900 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:04:53.919 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:04:53.939 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:04:53.961 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:04:53.982 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:04:53.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:54.002 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:04:54.003 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:04:54.003 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:04:54.003 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:04:54.003 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:04:54.003 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:04:54.004 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:04:54.004 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:04:54.004 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:04:54.015 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:04:54.015 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:04:54.015 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T03:04:54.015 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:04:54.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:54.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:54.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:54.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:54.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:54.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:54.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:54.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[29666]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T03:04:54.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:54.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:54.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1118476361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:54.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[37920]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T03:04:54.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:54.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:54.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:54 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1118476361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:54 smithi192 ceph-mon[33110]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T03:04:54.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:04:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:04:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:54 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1118476361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:04:54.512 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:54.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:54.547 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:04:54.547 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:04:54.644 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:54.671 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:54.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:54.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:54.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:54.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:54.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:55.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:55.059 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:55.086 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:55.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.189 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:55.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:55.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:55.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:55.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:55 smithi066 ceph-mon[29666]: pgmap v255: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-26T03:04:55.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:55 smithi066 ceph-mon[37920]: pgmap v255: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-26T03:04:55.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:55.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:55 smithi192 ceph-mon[33110]: pgmap v255: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-03-26T03:04:55.549 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:55.577 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:55.679 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:55.706 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:55.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:55.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:55.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:55.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:55.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:56.067 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:56.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:56.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:56.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:56.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:56.355 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:56.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:56.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:56.536 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:04:56.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:56.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:56.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:56.743 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:04:56.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:56.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:56.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:56.975 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.003 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:57.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:57.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:04:57.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:57.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:57.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:57.263 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:57.364 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:57 smithi066 ceph-mon[29666]: pgmap v256: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:57.364 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:57 smithi066 ceph-mon[37920]: pgmap v256: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:57.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:57.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:57 smithi192 ceph-mon[33110]: pgmap v256: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:57.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:57.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:57.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:57.652 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:57.655 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:57.697 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:04:57.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:57.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:57.884 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:57.912 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:57.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:58.014 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:58.041 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:58.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:58.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:58.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:58.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:58.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:58.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:58 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:04:58.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:58 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:04:58.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:58 smithi192 ceph-mon[33110]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T03:04:58.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:58.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:58.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:58 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:04:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:58 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:04:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:58 smithi066 ceph-mon[29666]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:58 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:58 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:04:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:58 smithi066 ceph-mon[37920]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T03:04:58.653 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:58.682 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:58.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:04:58.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:04:58.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:58.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:58.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:59.043 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:04:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.072 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:59.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:04:59.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:04:59.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:04:59.276 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:59 smithi066 ceph-mon[29666]: pgmap v258: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:59.276 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:04:59 smithi066 ceph-mon[29666]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T03:04:59.277 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:59 smithi066 ceph-mon[37920]: pgmap v258: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:59.277 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:04:59 smithi066 ceph-mon[37920]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T03:04:59.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:04:59.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:04:59.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:59.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:59.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:59 smithi192 ceph-mon[33110]: pgmap v258: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:04:59.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:04:59 smithi192 ceph-mon[33110]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T03:04:59.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:04:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:04:59.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:04:59.588 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:04:59.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:04:59.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:04:59.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:04:59.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:04:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:04:59.948 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:04:59.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:04:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:04:59.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:00.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:00.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:00.187 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:05:00.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:00.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:00.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/332640242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:00 smithi066 ceph-mon[29666]: from='client.34901 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/332640242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:00 smithi066 ceph-mon[37920]: from='client.34901 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:00.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:00.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/332640242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:00 smithi192 ceph-mon[33110]: from='client.34901 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:00.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:00.598 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:00.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:00.727 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:00.756 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:00.858 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:00.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:00.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:00.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:01.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:01.117 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:01.145 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:01.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.275 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:01.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:01.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:01.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[29666]: pgmap v260: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[29666]: from='client.34901 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[29666]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[37920]: pgmap v260: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[37920]: from='client.34901 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:01.481 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:01 smithi066 ceph-mon[37920]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T03:05:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:01 smithi192 ceph-mon[33110]: pgmap v260: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:05:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:01 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:01.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:01 smithi192 ceph-mon[33110]: from='client.34901 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:01.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:01 smithi192 ceph-mon[33110]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T03:05:01.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:01.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:01.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:01.663 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:01.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:01.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:01.812 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:05:01.812 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:05:01.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:01.927 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:01.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:02.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:02.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:02.158 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.186 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:02.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:02.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:02.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:02.417 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.446 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:02.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:02 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:02.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:02 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:02.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:02 smithi192 ceph-mon[33110]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T03:05:02.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:02 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:02.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:02 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:02.515 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:02 smithi066 ceph-mon[29666]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T03:05:02.515 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:02 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:02.515 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:02 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:02.516 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:02 smithi066 ceph-mon[37920]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T03:05:02.542 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:05:02.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:02.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:02.677 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:02.704 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:02.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:02.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:02.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:02.963 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:02.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:03.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:03.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:03.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:03.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:03.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:05:03.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:03.328 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:03.349 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:03 smithi066 ceph-mon[29666]: pgmap v263: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:05:03.350 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:03 smithi066 ceph-mon[37920]: pgmap v263: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:05:03.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:03.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:03.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:03 smithi192 ceph-mon[33110]: pgmap v263: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:05:03.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:03.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:03.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:03.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:03.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:03.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:05:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:05:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:05:03.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:03.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:03.955 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-26T03:05:03.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:03.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:04.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:04.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:04.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:04.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:04.360 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.388 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:04.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1211249488' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:05:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1211249488' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:05:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:04 smithi192 ceph-mon[33110]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T03:05:04.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:04.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:04.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1211249488' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:05:04.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1211249488' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:05:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:04 smithi066 ceph-mon[37920]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T03:05:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1211249488' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:05:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1211249488' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:05:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:04 smithi066 ceph-mon[29666]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T03:05:04.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:04.647 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:04.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:04.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:04.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:04.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:04.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:04.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:04.987 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:05:04.987 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:05:04.987 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 179, 2024-03-26T03:05:04.987 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:05:04.987 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:05:04.987 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:05:03.950057+0000", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T02:58:43.841339+0000", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5806080, 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "nodeep-scrub", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "noscrub", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:05:04.988 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 8, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:05:04.989 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:04.990 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 11, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 11, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 11, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 11, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 11, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "166", 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "164", 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:04.992 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:04.993 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:04.994 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.1800000667572021, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.1800000667572021, 2024-03-26T03:05:04.995 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:05:04.996 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:05:04.997 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:04.998 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:04.999 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:05.000 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.75, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.75, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:05:05.001 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:05.002 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:05.003 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:05.004 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:05.004 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:05.004 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:05.005 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:05.006 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:05.007 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6500000953674316, 2024-03-26T03:05:05.008 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6500000953674316, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:05:05.009 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:05.010 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:05.011 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:05:05.012 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:05.013 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:05.014 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:05.015 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.25, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.25, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.25, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.25, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.016 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:05.017 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:05.018 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:05:05.019 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:05.020 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:05.021 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:05.022 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.25, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.25, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.25, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.25, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:05:05.023 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:05:05.024 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:05.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:05.026 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:05.027 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.028 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.75, 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.75, 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:05:05.029 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 8, 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_1", 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:04:50.578884+0000", 2024-03-26T03:05:05.030 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 16, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 16, 2024-03-26T03:05:05.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 16, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 16, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 16, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "179", 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:05:05.032 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:05:05.033 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:05:05.034 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:05:05.035 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 1, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:05:05.036 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.037 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.038 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:05:05.039 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:05:05.040 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.041 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.042 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:05:05.043 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:05:05.044 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 169, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.045 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.047 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:05:05.050 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.053 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:05:05.054 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.057 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.058 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:05:05.059 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:05:05.059 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.059 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:05:05.060 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:05:05.061 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.062 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:05:05.064 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:05:05.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:05:05.066 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.067 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.068 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:05:05.070 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:05:00.820527+0000", 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.071 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:03:48.539871+0000", 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:04:08.741223+0000", 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.072 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:04:25.921561+0000", 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.073 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:04:57.151379+0000", 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:04:12.849070+0000", 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:04:38.436214+0000", 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:05:05.075 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:03:47.887673+0000", 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [], 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:05:05.076 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:05:05.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:05.078 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [], 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:05:05.079 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:05:05.080 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:05.081 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:05:05.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:05.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:05.239 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:05.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:05.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:05.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:05 smithi066 ceph-mon[29666]: pgmap v265: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-26T03:05:05.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:05 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:05:05.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:05 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2342553128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:05:05.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:05 smithi066 ceph-mon[37920]: pgmap v265: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-26T03:05:05.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:05 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:05:05.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:05 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2342553128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:05:05.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:05.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:05 smithi192 ceph-mon[33110]: pgmap v265: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-26T03:05:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:05 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:05:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:05 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2342553128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:05:05.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:05.599 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:05.627 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:05.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:05.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:05.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:05.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:05.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:05.987 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:06.015 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:06.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:06.123 INFO:tasks.thrashosds.thrasher:j is {'epoch': 179, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:05:03.950057+0000', 'last_up_change': '2024-03-26T02:58:43.841339+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '166', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '164', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-26T03:04:50.578884+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 172, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:05:00.820527+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:03:48.539871+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:04:08.741223+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:04:25.921561+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:04:57.151379+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:04:12.849070+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:04:38.436214+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:03:47.887673+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:05:06.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:05:06.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:06.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:06.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:06.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:06.404 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:06.406 DEBUG:teuthology.orchestra.run.smithi066:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-26T03:05:06.450 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:05:06.451 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:05:06.499 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:hints = 1 2024-03-26T03:05:06.499 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:05:06.499 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object prefix: benchmark_data_smithi066_100612 2024-03-26T03:05:06.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:05:06.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:05:06.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:06.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:06.644 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:06.700 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:06.787 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:06.792 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:06.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:06.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:06.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:06.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:06.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:07.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:07.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:07.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[29666]: pgmap v266: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[37920]: pgmap v266: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:05:07.247 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:07.248 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:07 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:07.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:07.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.403 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:07.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:07 smithi192 ceph-mon[33110]: pgmap v266: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:05:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:07 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:05:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:07 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:07 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:07.499 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 1 16 3087 3071 191.925 191.938 0.00760091 0.00519272 2024-03-26T03:05:07.505 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:07.551 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:07.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:07.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:07.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:07.849 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:07.951 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:07.996 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:07.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:08.098 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:08.146 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:08.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:08.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:08.282 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:08.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:08.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:08.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:08.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:08.499 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 2 16 6270 6254 195.421 198.938 0.00602713 0.00510731 2024-03-26T03:05:08.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:08.567 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:08.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:08.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:09.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:09.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:05:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:05:09.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:09 smithi192 ceph-mon[33110]: pgmap v267: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-03-26T03:05:09.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:09.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:09.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:09 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1224067540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:09.492 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:09 smithi192 ceph-mon[33110]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:05:09.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:05:09.494 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:09.499 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 3 16 9028 9012 187.725 172.375 0.00377472 0.00531681 2024-03-26T03:05:09.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[29666]: pgmap v267: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-03-26T03:05:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1224067540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[29666]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[37920]: pgmap v267: 158 pgs: 158 active+clean; 582 KiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1224067540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:09 smithi066 ceph-mon[37920]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T03:05:09.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:09.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:09.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:09.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:09.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:09.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:09.890 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:09.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:09.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:10.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:10.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:10.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:10.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:10.260 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:10.260 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:10.260 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:10.261 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:10.261 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:10.261 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:10.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:10.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:10.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:10.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 4 16 11472 11456 178.978 152.75 0.00887877 0.00557656 2024-03-26T03:05:10.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:10.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:10.659 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:10.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:10.809 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:10.857 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:10.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:10.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:10.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:10.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:11.099 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.144 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:11.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:11.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:11.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:11.285 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:11 smithi066 ceph-mon[29666]: pgmap v269: 158 pgs: 158 active+clean; 424 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s 2024-03-26T03:05:11.285 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3740801598' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:05:11.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:11 smithi066 ceph-mon[37920]: pgmap v269: 158 pgs: 158 active+clean; 424 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s 2024-03-26T03:05:11.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3740801598' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:05:11.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:11.416 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:11.460 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:05:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:11 smithi192 ceph-mon[33110]: pgmap v269: 158 pgs: 158 active+clean; 424 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s 2024-03-26T03:05:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3740801598' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:05:11.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 5 16 14186 14170 177.104 169.625 0.00253666 0.00563765 2024-03-26T03:05:11.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:11.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:11.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:11.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:11.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:11.836 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:11.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:11.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:11.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:12.075 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:12.122 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:12.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.271 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:12.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:12.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:12.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:12 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3740801598' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:05:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:12 smithi192 ceph-mon[33110]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T03:05:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:05:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:05:12.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:12.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 6 16 16765 16749 174.449 161.188 0.00540462 0.00572415 2024-03-26T03:05:12.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:12.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3740801598' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:05:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[29666]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3740801598' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[37920]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:05:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:12.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:12.692 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:12.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:12.839 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:12.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:12.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:12.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:13.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:13.124 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:13.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:13.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:13.287 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:05:13.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:13.427 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:13.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:13 smithi192 ceph-mon[33110]: pgmap v271: 158 pgs: 158 active+clean; 543 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-26T03:05:13.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 7 16 19327 19311 172.398 160.125 0.00643223 0.00579291 2024-03-26T03:05:13.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:13.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:13 smithi066 ceph-mon[29666]: pgmap v271: 158 pgs: 158 active+clean; 543 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-26T03:05:13.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:13 smithi066 ceph-mon[37920]: pgmap v271: 158 pgs: 158 active+clean; 543 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-26T03:05:13.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:13.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:13.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:13.801 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:13.841 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:13.943 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:13.992 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:13.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:14.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:14.097 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:05:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:05:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:05:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:14.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:14.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:14.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:14.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:14.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:14.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:14.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:14 smithi066 ceph-mon[29666]: 2.9 scrub starts 2024-03-26T03:05:14.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:14 smithi066 ceph-mon[29666]: 2.9 scrub ok 2024-03-26T03:05:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:14 smithi066 ceph-mon[37920]: 2.9 scrub starts 2024-03-26T03:05:14.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:14 smithi066 ceph-mon[37920]: 2.9 scrub ok 2024-03-26T03:05:14.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:14.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:14.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:14.487 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:05:14.487 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:05:14.487 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:05:14.487 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:05:14.487 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75525 7.4 7:229e4d32:::benchmark_data_smithi066_76917_object75524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.308357+0000", 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 204.17417908499999, 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.010940199, 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75525 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.488 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308357+0000", 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308357+0000", 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308360+0000", 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6729999999999998e-06 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308401+0000", 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1057999999999999e-05 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308412+0000", 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0577000000000001e-05 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308421+0000", 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8289999999999997e-06 2024-03-26T03:05:14.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495402+0000", 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.18698124099999999 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495411+0000", 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.285e-06 2024-03-26T03:05:14.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.312028+0000", 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.81661686300000003 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.314010+0000", 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0019819579999999998 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.319214+0000", 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.005204017 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.319297+0000", 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2698e-05 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.493 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75527 7.16 7:68ebb84d:::benchmark_data_smithi066_76917_object75526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.308433+0000", 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 204.17410218500001, 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.057818116, 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75527 2024-03-26T03:05:14.494 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308433+0000", 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308433+0000", 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308434+0000", 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2200000000000003e-07 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308458+0000", 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3436999999999999e-05 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308464+0000", 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.404e-06 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308470+0000", 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4179999999999996e-06 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.517883+0000", 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20941363299999999 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.517904+0000", 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0769999999999999e-05 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.351182+0000", 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.83327831100000005 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.355526+0000", 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043432890000000002 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.366164+0000", 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010638409 2024-03-26T03:05:14.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.366252+0000", 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7622999999999998e-05 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75535 7.14 7:296efe21:::benchmark_data_smithi066_76917_object75534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.322254+0000", 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 204.160282147, 2024-03-26T03:05:14.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.013142381, 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75535 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322254+0000", 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322254+0000", 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322257+0000", 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8410000000000002e-06 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.501 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:14 smithi192 ceph-mon[33110]: 2.9 scrub starts 2024-03-26T03:05:14.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:14 smithi192 ceph-mon[33110]: 2.9 scrub ok 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322294+0000", 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6766e-05 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322313+0000", 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9043999999999999e-05 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322328+0000", 2024-03-26T03:05:14.503 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4816e-05 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495341+0000", 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.17301283000000001 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495351+0000", 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0040999999999999e-05 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.504 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.325257+0000", 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.82990634299999999 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.325442+0000", 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000184842 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.325617+0000", 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000175184 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.334181+0000", 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0085639180000000002 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.334235+0000", 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3418999999999998e-05 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.335090+0000", 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00085500099999999998 2024-03-26T03:05:14.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.335349+0000", 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025913800000000001 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.335396+0000", 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7197999999999998e-05 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.507 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76596 7.18 7.dbf6958 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:52.302604+0000", 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 202.179931463, 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99306915299999998, 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76596 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.508 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302604+0000", 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302604+0000", 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302607+0000", 2024-03-26T03:05:14.509 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3630000000000001e-06 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302631+0000", 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4168999999999998e-05 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302644+0000", 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3159999999999999e-05 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.510 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302655+0000", 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.066e-05 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302731+0000", 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6691000000000001e-05 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302740+0000", 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1570000000000006e-06 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.511 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295660+0000", 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99291966399999998 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295673+0000", 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3288999999999999e-05 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76601 7.18 7.a9915598 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:05:14.512 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:52.309002+0000", 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 202.173533443, 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98671072100000001, 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76601 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.513 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309002+0000", 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309002+0000", 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309004+0000", 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0080000000000001e-06 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309028+0000", 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3490999999999999e-05 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309038+0000", 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0372e-05 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309046+0000", 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0590000000000003e-06 2024-03-26T03:05:14.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309124+0000", 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.805e-05 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309134+0000", 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0190000000000001e-05 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295703+0000", 2024-03-26T03:05:14.516 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98656848200000002 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295713+0000", 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0069e-05 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76606 7.18 7.ca314098 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:52.313649+0000", 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 202.168886421, 2024-03-26T03:05:14.517 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98228015700000004, 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76606 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313649+0000", 2024-03-26T03:05:14.518 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313649+0000", 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313652+0000", 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.317e-06 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.519 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313672+0000", 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9953e-05 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313689+0000", 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7540000000000001e-05 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313696+0000", 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5479999999999998e-06 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.520 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313741+0000", 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5008e-05 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313747+0000", 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9039999999999997e-06 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295917+0000", 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.982170816 2024-03-26T03:05:14.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295929+0000", 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2071e-05 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76573 7.14 7:2ce71561:::benchmark_data_smithi066_76917_object76572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289155+0000", 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.19338064799999, 2024-03-26T03:05:14.522 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.463067562, 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76573 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289155+0000", 2024-03-26T03:05:14.523 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289155+0000", 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289157+0000", 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.79e-06 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289186+0000", 2024-03-26T03:05:14.524 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.9400999999999999e-05 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289197+0000", 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0499000000000001e-05 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289207+0000", 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0324e-05 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289718+0000", 2024-03-26T03:05:14.525 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00051074300000000003 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289726+0000", 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8490000000000006e-06 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.294959+0000", 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0052334759999999999 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.526 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.294968+0000", 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7739999999999993e-06 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.675546+0000", 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3805785420000001 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.676666+0000", 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001119461 2024-03-26T03:05:14.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752119+0000", 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.075453352000000001 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752223+0000", 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010335099999999999 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76576 7.17 7:ea6e24b1:::benchmark_data_smithi066_76917_object76575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289318+0000", 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.193217664, 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.528118935, 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76576 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.529 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289318+0000", 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289318+0000", 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289319+0000", 2024-03-26T03:05:14.530 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8299999999999995e-07 2024-03-26T03:05:14.531 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.531 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.531 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.531 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289351+0000", 2024-03-26T03:05:14.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 8 15 21592 21577 168.55 141.625 0.0107999 0.00592611 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2016e-05 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289360+0000", 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4590000000000004e-06 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.532 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289368+0000", 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8369999999999997e-06 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290510+0000", 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0011418000000000001 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290517+0000", 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8959999999999997e-06 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293379+0000", 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002861948 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293388+0000", 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9959999999999999e-06 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.801084+0000", 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.507696285 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.802350+0000", 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00126542 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.817272+0000", 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.01492253 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.817437+0000", 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016476500000000001 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76577 7.17 7:ef576187:::benchmark_data_smithi066_76917_object76576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289379+0000", 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.193156555, 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.536755898, 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.536 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76577 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289379+0000", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289379+0000", 2024-03-26T03:05:14.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289380+0000", 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8800000000000001e-07 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289406+0000", 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6115e-05 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289414+0000", 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5669999999999997e-06 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289421+0000", 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3370000000000002e-06 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290544+0000", 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001122861 2024-03-26T03:05:14.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290549+0000", 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3990000000000003e-06 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293410+0000", 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028609740000000001 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.540 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293417+0000", 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9650000000000002e-06 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.802413+0000", 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.508995978 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.803792+0000", 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001378659 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.825676+0000", 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.021884596999999999 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.826135+0000", 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00045855800000000002 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76589 7.17 7:e926eddf:::benchmark_data_smithi066_76917_object76588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.542 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289429+0000", 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.19310636599999, 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5438101280000001, 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76589 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289429+0000", 2024-03-26T03:05:14.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289429+0000", 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289430+0000", 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7400000000000002e-07 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289455+0000", 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4635999999999999e-05 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289462+0000", 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9290000000000001e-06 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289469+0000", 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9770000000000003e-06 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290574+0000", 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0011057549999999999 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290579+0000", 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7600000000000002e-06 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293441+0000", 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002861363 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293448+0000", 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.187e-06 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.803843+0000", 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5103952920000001 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.805763+0000", 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00192022 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.832867+0000", 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.027103308 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833239+0000", 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000372927 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.548 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76605 7.14 7:2d4fabea:::benchmark_data_smithi066_76917_object76604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289476+0000", 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.19305941100001, 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.476329665, 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.549 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76605 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289476+0000", 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289476+0000", 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.550 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289477+0000", 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8000000000000005e-07 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289503+0000", 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5534999999999999e-05 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.551 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289510+0000", 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2790000000000001e-06 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289516+0000", 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2740000000000004e-06 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289771+0000", 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025493100000000002 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:05:14.552 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289776+0000", 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7539999999999997e-06 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.298866+0000", 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0090900359999999993 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.298874+0000", 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3480000000000004e-06 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.553 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752310+0000", 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4534362270000001 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752461+0000", 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000150419 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.753330+0000", 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00086932899999999996 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.554 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759667+0000", 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0063369380000000003 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759732+0000", 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4414999999999999e-05 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759810+0000", 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8844000000000006e-05 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.555 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759830+0000", 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.963e-05 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.760377+0000", 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00054731500000000004 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.765806+0000", 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0054286109999999999 2024-03-26T03:05:14.556 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76610 7.19 7:9b2fb385:::benchmark_data_smithi066_76917_object76609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289524+0000", 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.193011273, 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4842921849999999, 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.557 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76610 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289524+0000", 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289524+0000", 2024-03-26T03:05:14.558 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289525+0000", 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3399999999999998e-07 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289581+0000", 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6276999999999997e-05 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289590+0000", 2024-03-26T03:05:14.559 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0149999999999992e-06 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289598+0000", 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4760000000000001e-06 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290791+0000", 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001193295 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.560 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290798+0000", 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7519999999999999e-06 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.292610+0000", 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018123169999999999 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.292620+0000", 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7599999999999997e-06 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.664560+0000", 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3719395089999999 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.664739+0000", 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017908299999999999 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,6,7", 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.664948+0000", 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000209796 2024-03-26T03:05:14.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771112+0000", 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.106163639 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771767+0000", 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00065495799999999997 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771816+0000", 2024-03-26T03:05:14.563 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9298000000000001e-05 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771862+0000", 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6029000000000003e-05 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771981+0000", 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011827 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.773817+0000", 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018359769999999999 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76628 7.17 7:efdc1178:::benchmark_data_smithi066_76917_object76627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.290595+0000", 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.191940738, 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5449126049999999, 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.565 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76628 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290595+0000", 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.566 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290595+0000", 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290596+0000", 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.392e-06 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290660+0000", 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3312999999999995e-05 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290668+0000", 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0469999999999994e-06 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290676+0000", 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.259e-06 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291402+0000", 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00072557899999999996 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291407+0000", 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8470000000000001e-06 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293472+0000", 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020646879999999999 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293479+0000", 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5169999999999996e-06 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.569 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.805808+0000", 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.512329349 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.806173+0000", 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00036477100000000001 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5,7", 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.806375+0000", 2024-03-26T03:05:14.570 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020274600000000001 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.813734+0000", 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0073585120000000002 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833675+0000", 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019941496999999999 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.571 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833728+0000", 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2692000000000003e-05 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833774+0000", 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5426000000000002e-05 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833903+0000", 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000129529 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.572 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835508+0000", 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0016044410000000001 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76630 7.17 7:e9c213e7:::benchmark_data_smithi066_76917_object76629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.290970+0000", 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.191566102, 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.559619444, 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.573 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76630 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290970+0000", 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.574 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290970+0000", 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290971+0000", 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0839999999999999e-06 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291026+0000", 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5686999999999997e-05 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.575 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291036+0000", 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.594e-06 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291044+0000", 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1610000000000003e-06 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291608+0000", 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00056420000000000005 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291615+0000", 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3790000000000003e-06 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293502+0000", 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018876349999999999 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293509+0000", 2024-03-26T03:05:14.577 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5270000000000004e-06 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835554+0000", 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5420450429999999 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835739+0000", 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018524399999999999 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.578 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5,7", 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835962+0000", 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00022237799999999999 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.839842+0000", 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038806880000000002 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.839859+0000", 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7227e-05 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.844365+0000", 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0045059039999999998 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.848761+0000", 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043960520000000001 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.848823+0000", 2024-03-26T03:05:14.580 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1501999999999995e-05 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.850589+0000", 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017661389999999999 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:112611 7.13 7:c88931eb:::benchmark_data_smithi066_76917_object12428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:05:14.581 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 142.119010062, 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.85905683200000005, 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 112611 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.582 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6499999999999999e-07 2024-03-26T03:05:14.583 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363529+0000", 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7949999999999999e-06 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363534+0000", 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1560000000000003e-06 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363539+0000", 2024-03-26T03:05:14.584 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2329999999999997e-06 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363678+0000", 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000138412 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363710+0000", 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.1860999999999999e-05 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.585 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.169363+0000", 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.80565370300000005 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.169491+0000", 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000127441 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.170058+0000", 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00056694500000000001 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.586 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.587 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.177633+0000", 2024-03-26T03:05:14.587 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0075750779999999998 2024-03-26T03:05:14.587 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.181265+0000", 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003632298 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.588 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.181318+0000", 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2575999999999999e-05 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.222533+0000", 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.041215214 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.222582+0000", 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9555000000000002e-05 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.589 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142695 7.18 7:1a9a85f2:::benchmark_data_smithi066_76917_object42512:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.371156+0000", 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 105.111379755, 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0339076030000001, 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.590 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142695 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371156+0000", 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.591 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371156+0000", 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371157+0000", 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4389999999999999e-06 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371162+0000", 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3810000000000001e-06 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.592 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371170+0000", 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.038e-06 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371176+0000", 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3609999999999997e-06 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371422+0000", 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000245386 2024-03-26T03:05:14.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371428+0000", 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2110000000000003e-06 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.374547+0000", 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031191000000000001 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.374558+0000", 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0951000000000001e-05 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405007+0000", 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.030449669 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405064+0000", 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6066999999999997e-05 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.595 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142780 7.18 7:1931f3fd:::benchmark_data_smithi066_76917_object42597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.442439+0000", 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 105.040096858, 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97209616700000001, 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.596 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142780 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442439+0000", 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442439+0000", 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442441+0000", 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7409999999999999e-06 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442447+0000", 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4930000000000003e-06 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442457+0000", 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0236e-05 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442466+0000", 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.354e-06 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442563+0000", 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7499999999999998e-05 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442574+0000", 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0658e-05 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405672+0000", 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96309829800000002 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405817+0000", 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000145026 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.406214+0000", 2024-03-26T03:05:14.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000396473 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414323+0000", 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0081098179999999995 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414379+0000", 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5500000000000001e-05 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.601 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414425+0000", 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5986000000000003e-05 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414494+0000", 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8732000000000005e-05 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414535+0000", 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1352e-05 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.602 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142786 7.18 7:18a9b3ae:::benchmark_data_smithi066_76917_object42603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.446346+0000", 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 105.03618966000001, 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98757069600000003, 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.603 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142786 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446346+0000", 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446346+0000", 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446348+0000", 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0320000000000002e-06 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446354+0000", 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6389999999999997e-06 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446365+0000", 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1208e-05 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446374+0000", 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9870000000000005e-06 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446439+0000", 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4834999999999998e-05 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446451+0000", 2024-03-26T03:05:14.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.222e-05 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.408158+0000", 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96170670800000002 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.408280+0000", 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012194 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.408566+0000", 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00028633599999999999 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414573+0000", 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0060071839999999996 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414703+0000", 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000129443 2024-03-26T03:05:14.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433781+0000", 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019078504 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433855+0000", 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.436e-05 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433917+0000", 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1299999999999999e-05 2024-03-26T03:05:14.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142796 7.18 7:1c624d3a:::benchmark_data_smithi066_76917_object42613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.453769+0000", 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 105.028766404, 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98029207799999996, 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.610 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142796 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453769+0000", 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453769+0000", 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453771+0000", 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0949999999999998e-06 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453776+0000", 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.216e-06 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453783+0000", 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4510000000000003e-06 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453789+0000", 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8730000000000004e-06 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.454575+0000", 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00078626300000000004 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.454586+0000", 2024-03-26T03:05:14.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.044e-05 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.412024+0000", 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.957438758 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.412162+0000", 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013784299999999999 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:05:14.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.412494+0000", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000332076 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414745+0000", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002250232 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414782+0000", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7766999999999999e-05 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433978+0000", 2024-03-26T03:05:14.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019195796000000001 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434018+0000", 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0212000000000001e-05 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434061+0000", 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3056000000000002e-05 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.616 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142818 7.18 7:1ebdfa01:::benchmark_data_smithi066_76917_object42635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 104.895715049, 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.84743222200000001, 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142818 2024-03-26T03:05:14.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8999999999999996e-07 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586824+0000", 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.9119999999999998e-06 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586830+0000", 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6559999999999997e-06 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586835+0000", 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5809999999999996e-06 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586975+0000", 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013987400000000001 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586984+0000", 2024-03-26T03:05:14.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8950000000000004e-06 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.419246+0000", 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.83226167200000001 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.419365+0000", 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118733 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:05:14.621 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.419698+0000", 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00033390999999999998 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.421858+0000", 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0021593250000000001 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.421901+0000", 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2904000000000001e-05 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.622 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434168+0000", 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.012267659 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434214+0000", 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5793e-05 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434253+0000", 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8717999999999998e-05 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.623 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.624 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.624 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:14.624 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:14.624 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:05:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:14.668 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:14.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:14.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:14.816 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:14.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:14.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:14.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:14.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:15.051 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:15.092 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:15.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:15.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:15.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:15.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:15.383 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:05:15.408 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:05:15.414 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:05:15.458 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:15 smithi192 ceph-mon[33110]: pgmap v272: 158 pgs: 158 active+clean; 845 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-26T03:05:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:15 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:05:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:15 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:05:15.500 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 9 16 23554 23538 163.435 122.562 0.00411783 0.00611122 2024-03-26T03:05:15.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:15 smithi066 ceph-mon[29666]: pgmap v272: 158 pgs: 158 active+clean; 845 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-26T03:05:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:15 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:05:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:15 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:05:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:15 smithi066 ceph-mon[37920]: pgmap v272: 158 pgs: 158 active+clean; 845 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-26T03:05:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:15 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:05:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:15 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:05:15.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:15.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:15.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:15.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:15.888 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:15.941 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:15.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:15.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:16.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:16.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:16.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:16.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:16.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:16.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:16.294 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:05:16.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:16.402 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:16.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:16 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:05:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:16 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:05:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:16 smithi192 ceph-mon[33110]: from='client.24973 v1:172.21.15.66:0/1585015294' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:05:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:16 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:05:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:16 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:05:16.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 10 16 25710 25694 160.562 134.75 0.00648193 0.00622149 2024-03-26T03:05:16.503 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:16.552 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:16.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:16.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[37920]: from='client.24973 v1:172.21.15.66:0/1585015294' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[29666]: from='client.24973 v1:172.21.15.66:0/1585015294' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:05:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:16 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:05:16.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:16.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:16.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:16.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:16.803 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:16.866 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:16.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:16.873 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-03-26T03:05:14.040828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":886512040,"num_objects":13748,"num_object_clones":0,"num_object_copies":41244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13748,"num_whiteouts":0,"num_read":1372,"num_read_kb":1092,"num_write":14146,"num_write_kb":865197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15650,"ondisk_log_size":15650,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3890092,"kb_used_data":2560616,"kb_used_omap":12,"kb_used_meta":1329395,"kb_avail":746071124,"statfs":{"total":767960285184,"available":763976830976,"internally_reserved":0,"allocated":2622070784,"data_stored":2618455037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1361301040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":885915648,"num_objects":13518,"num_object_clones":0,"num_object_copies":40554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13518,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":13518,"num_write_kb":865152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002928"},"pg_stats":[{"pgid":"8.4","version":"181'1297","reported_seq":1325,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:13.952298+0000","last_change":"2024-03-26T03:04:52.197394+0000","last_active":"2024-03-26T03:05:13.952298+0000","last_peered":"2024-03-26T03:05:13.952298+0000","last_clean":"2024-03-26T03:05:13.952298+0000","last_became_active":"2024-03-26T03:04:52.197080+0000","last_became_peered":"2024-03-26T03:04:52.197080+0000","last_unstale":"2024-03-26T03:05:13.952298+0000","last_undegraded":"2024-03-26T03:05:13.952298+0000","last_fullsized":"2024-03-26T03:05:13.952298+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":1297,"log_dups_size":0,"ondisk_log_size":1297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:04.134916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85000192,"num_objects":1297,"num_object_clones":0,"num_object_copies":3891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1297,"num_write_kb":83008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":348,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464747+0000","last_change":"2024-03-26T03:04:48.638112+0000","last_active":"2024-03-26T03:05:11.464747+0000","last_peered":"2024-03-26T03:05:11.464747+0000","last_clean":"2024-03-26T03:05:11.464747+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:05:11.464747+0000","last_undegraded":"2024-03-26T03:05:11.464747+0000","last_fullsized":"2024-03-26T03:05:11.464747+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:04:48.638002+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:09.353317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":315,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459081+0000","last_change":"2024-03-26T03:04:38.309217+0000","last_active":"2024-03-26T03:05:11.459081+0000","last_peered":"2024-03-26T03:05:11.459081+0000","last_clean":"2024-03-26T03:05:11.459081+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:05:11.459081+0000","last_undegraded":"2024-03-26T03:05:11.459081+0000","last_fullsized":"2024-03-26T03:05:11.459081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:04:38.309114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:38.309114+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:01.503706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.960512+0000","last_change":"2024-03-26T03:01:48.444274+0000","last_active":"2024-03-26T03:05:08.960512+0000","last_peered":"2024-03-26T03:05:08.960512+0000","last_clean":"2024-03-26T03:05:08.960512+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:05:08.960512+0000","last_undegraded":"2024-03-26T03:05:08.960512+0000","last_fullsized":"2024-03-26T03:05:08.960512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:48.444147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:48.444147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.477112+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":64,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.460483+0000","last_change":"2024-03-26T03:04:37.863032+0000","last_active":"2024-03-26T03:05:11.460483+0000","last_peered":"2024-03-26T03:05:11.460483+0000","last_clean":"2024-03-26T03:05:11.460483+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:05:11.460483+0000","last_undegraded":"2024-03-26T03:05:11.460483+0000","last_fullsized":"2024-03-26T03:05:11.460483+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:37.862940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:37.862940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.217505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"180'511","reported_seq":537,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.170250+0000","last_change":"2024-03-26T03:04:52.164503+0000","last_active":"2024-03-26T03:05:09.170250+0000","last_peered":"2024-03-26T03:05:09.170250+0000","last_clean":"2024-03-26T03:05:09.170250+0000","last_became_active":"2024-03-26T03:04:52.164179+0000","last_became_peered":"2024-03-26T03:04:52.164179+0000","last_unstale":"2024-03-26T03:05:09.170250+0000","last_undegraded":"2024-03-26T03:05:09.170250+0000","last_fullsized":"2024-03-26T03:05:09.170250+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:08.819705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":511,"num_write_kb":32704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":290,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962662+0000","last_change":"2024-03-26T03:04:06.968953+0000","last_active":"2024-03-26T03:05:08.962662+0000","last_peered":"2024-03-26T03:05:08.962662+0000","last_clean":"2024-03-26T03:05:08.962662+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:05:08.962662+0000","last_undegraded":"2024-03-26T03:05:08.962662+0000","last_fullsized":"2024-03-26T03:05:08.962662+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_clean_scrub_stamp":"2024-03-26T03:04:06.968735+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:17.994202+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968791+0000","last_change":"2024-03-26T03:03:59.711554+0000","last_active":"2024-03-26T03:05:08.968791+0000","last_peered":"2024-03-26T03:05:08.968791+0000","last_clean":"2024-03-26T03:05:08.968791+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:05:08.968791+0000","last_undegraded":"2024-03-26T03:05:08.968791+0000","last_fullsized":"2024-03-26T03:05:08.968791+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:03:59.710985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:25.977819+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.958305+0000","last_change":"2024-03-26T03:04:36.837815+0000","last_active":"2024-03-26T03:05:08.958305+0000","last_peered":"2024-03-26T03:05:08.958305+0000","last_clean":"2024-03-26T03:05:08.958305+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:05:08.958305+0000","last_undegraded":"2024-03-26T03:05:08.958305+0000","last_fullsized":"2024-03-26T03:05:08.958305+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:02.467182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962557+0000","last_change":"2024-03-26T03:04:47.630302+0000","last_active":"2024-03-26T03:05:08.962557+0000","last_peered":"2024-03-26T03:05:08.962557+0000","last_clean":"2024-03-26T03:05:08.962557+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:05:08.962557+0000","last_undegraded":"2024-03-26T03:05:08.962557+0000","last_fullsized":"2024-03-26T03:05:08.962557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:47.630187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:04:47.630187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:59.447692+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"180'630","reported_seq":656,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:10.087052+0000","last_change":"2024-03-26T03:04:52.165009+0000","last_active":"2024-03-26T03:05:10.087052+0000","last_peered":"2024-03-26T03:05:10.087052+0000","last_clean":"2024-03-26T03:05:10.087052+0000","last_became_active":"2024-03-26T03:04:52.164468+0000","last_became_peered":"2024-03-26T03:04:52.164468+0000","last_unstale":"2024-03-26T03:05:10.087052+0000","last_undegraded":"2024-03-26T03:05:10.087052+0000","last_fullsized":"2024-03-26T03:05:10.087052+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:04.943306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41287680,"num_objects":630,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":40320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":330,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.460845+0000","last_change":"2024-03-26T03:01:54.088999+0000","last_active":"2024-03-26T03:05:11.460845+0000","last_peered":"2024-03-26T03:05:11.460845+0000","last_clean":"2024-03-26T03:05:11.460845+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:05:11.460845+0000","last_undegraded":"2024-03-26T03:05:11.460845+0000","last_fullsized":"2024-03-26T03:05:11.460845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:01:54.088880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:54.088880+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":269,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.959673+0000","last_change":"2024-03-26T03:04:47.600713+0000","last_active":"2024-03-26T03:05:08.959673+0000","last_peered":"2024-03-26T03:05:08.959673+0000","last_clean":"2024-03-26T03:05:08.959673+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:05:08.959673+0000","last_undegraded":"2024-03-26T03:05:08.959673+0000","last_fullsized":"2024-03-26T03:05:08.959673+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:04:47.600600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.828347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.958885+0000","last_change":"2024-03-26T03:04:45.653970+0000","last_active":"2024-03-26T03:05:08.958885+0000","last_peered":"2024-03-26T03:05:08.958885+0000","last_clean":"2024-03-26T03:05:08.958885+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:05:08.958885+0000","last_undegraded":"2024-03-26T03:05:08.958885+0000","last_fullsized":"2024-03-26T03:05:08.958885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:45.653866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:04:45.653866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:55.060254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464369+0000","last_change":"2024-03-26T03:04:41.640595+0000","last_active":"2024-03-26T03:05:11.464369+0000","last_peered":"2024-03-26T03:05:11.464369+0000","last_clean":"2024-03-26T03:05:11.464369+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:05:11.464369+0000","last_undegraded":"2024-03-26T03:05:11.464369+0000","last_fullsized":"2024-03-26T03:05:11.464369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:47.003467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"180'517","reported_seq":543,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.167251+0000","last_change":"2024-03-26T03:04:52.197465+0000","last_active":"2024-03-26T03:05:09.167251+0000","last_peered":"2024-03-26T03:05:09.167251+0000","last_clean":"2024-03-26T03:05:09.167251+0000","last_became_active":"2024-03-26T03:04:52.196952+0000","last_became_peered":"2024-03-26T03:04:52.196952+0000","last_unstale":"2024-03-26T03:05:09.167251+0000","last_undegraded":"2024-03-26T03:05:09.167251+0000","last_fullsized":"2024-03-26T03:05:09.167251+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.864582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33882112,"num_objects":517,"num_object_clones":0,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"153'25","reported_seq":294,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.456055+0000","last_change":"2024-03-26T03:03:57.378897+0000","last_active":"2024-03-26T03:05:11.456055+0000","last_peered":"2024-03-26T03:05:11.456055+0000","last_clean":"2024-03-26T03:05:11.456055+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:05:11.456055+0000","last_undegraded":"2024-03-26T03:05:11.456055+0000","last_fullsized":"2024-03-26T03:05:11.456055+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'25","last_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:03:57.378721+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846823+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464832+0000","last_change":"2024-03-26T03:01:33.183481+0000","last_active":"2024-03-26T03:05:11.464832+0000","last_peered":"2024-03-26T03:05:11.464832+0000","last_clean":"2024-03-26T03:05:11.464832+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:05:11.464832+0000","last_undegraded":"2024-03-26T03:05:11.464832+0000","last_fullsized":"2024-03-26T03:05:11.464832+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.183362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.183362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":245,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968451+0000","last_change":"2024-03-26T03:00:45.397106+0000","last_active":"2024-03-26T03:05:08.968451+0000","last_peered":"2024-03-26T03:05:08.968451+0000","last_clean":"2024-03-26T03:05:08.968451+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:05:08.968451+0000","last_undegraded":"2024-03-26T03:05:08.968451+0000","last_fullsized":"2024-03-26T03:05:08.968451+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.397023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.963332+0000","last_change":"2024-03-26T03:03:55.839464+0000","last_active":"2024-03-26T03:05:08.963332+0000","last_peered":"2024-03-26T03:05:08.963332+0000","last_clean":"2024-03-26T03:05:08.963332+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:05:08.963332+0000","last_undegraded":"2024-03-26T03:05:08.963332+0000","last_fullsized":"2024-03-26T03:05:08.963332+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:55.839275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:55.839275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:25.215925+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":334,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.957411+0000","last_change":"2024-03-26T03:04:43.715167+0000","last_active":"2024-03-26T03:05:08.957411+0000","last_peered":"2024-03-26T03:05:08.957411+0000","last_clean":"2024-03-26T03:05:08.957411+0000","last_became_active":"2024-03-26T03:04:41.967552+0000","last_became_peered":"2024-03-26T03:04:41.967552+0000","last_unstale":"2024-03-26T03:05:08.957411+0000","last_undegraded":"2024-03-26T03:05:08.957411+0000","last_fullsized":"2024-03-26T03:05:08.957411+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:49.647595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"180'599","reported_seq":625,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.739753+0000","last_change":"2024-03-26T03:04:52.165355+0000","last_active":"2024-03-26T03:05:09.739753+0000","last_peered":"2024-03-26T03:05:09.739753+0000","last_clean":"2024-03-26T03:05:09.739753+0000","last_became_active":"2024-03-26T03:04:52.164397+0000","last_became_peered":"2024-03-26T03:04:52.164397+0000","last_unstale":"2024-03-26T03:05:09.739753+0000","last_undegraded":"2024-03-26T03:05:09.739753+0000","last_fullsized":"2024-03-26T03:05:09.739753+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:10.262037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":38336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":527,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464507+0000","last_change":"2024-03-26T03:04:42.630937+0000","last_active":"2024-03-26T03:05:11.464507+0000","last_peered":"2024-03-26T03:05:11.464507+0000","last_clean":"2024-03-26T03:05:11.464507+0000","last_became_active":"2024-03-26T03:04:41.962917+0000","last_became_peered":"2024-03-26T03:04:41.962917+0000","last_unstale":"2024-03-26T03:05:11.464507+0000","last_undegraded":"2024-03-26T03:05:11.464507+0000","last_fullsized":"2024-03-26T03:05:11.464507+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:43.292148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":273,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455514+0000","last_change":"2024-03-26T03:03:48.428215+0000","last_active":"2024-03-26T03:05:11.455514+0000","last_peered":"2024-03-26T03:05:11.455514+0000","last_clean":"2024-03-26T03:05:11.455514+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:05:11.455514+0000","last_undegraded":"2024-03-26T03:05:11.455514+0000","last_fullsized":"2024-03-26T03:05:11.455514+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:03:48.427958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:48.427958+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846862+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969422+0000","last_change":"2024-03-26T03:01:50.576964+0000","last_active":"2024-03-26T03:05:08.969422+0000","last_peered":"2024-03-26T03:05:08.969422+0000","last_clean":"2024-03-26T03:05:08.969422+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:05:08.969422+0000","last_undegraded":"2024-03-26T03:05:08.969422+0000","last_fullsized":"2024-03-26T03:05:08.969422+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:50.576769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.576769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":337,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.960920+0000","last_change":"2024-03-26T03:01:50.516513+0000","last_active":"2024-03-26T03:05:08.960920+0000","last_peered":"2024-03-26T03:05:08.960920+0000","last_clean":"2024-03-26T03:05:08.960920+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:05:08.960920+0000","last_undegraded":"2024-03-26T03:05:08.960920+0000","last_fullsized":"2024-03-26T03:05:08.960920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:01:50.516336+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.516336+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.477196+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":242,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455626+0000","last_change":"2024-03-26T03:03:32.396450+0000","last_active":"2024-03-26T03:05:11.455626+0000","last_peered":"2024-03-26T03:05:11.455626+0000","last_clean":"2024-03-26T03:05:11.455626+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:05:11.455626+0000","last_undegraded":"2024-03-26T03:05:11.455626+0000","last_fullsized":"2024-03-26T03:05:11.455626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:32.396317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:32.396317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846790+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"180'527","reported_seq":553,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.166929+0000","last_change":"2024-03-26T03:04:52.162802+0000","last_active":"2024-03-26T03:05:09.166929+0000","last_peered":"2024-03-26T03:05:09.166929+0000","last_clean":"2024-03-26T03:05:09.166929+0000","last_became_active":"2024-03-26T03:04:52.162480+0000","last_became_peered":"2024-03-26T03:04:52.162480+0000","last_unstale":"2024-03-26T03:05:09.166929+0000","last_undegraded":"2024-03-26T03:05:09.166929+0000","last_fullsized":"2024-03-26T03:05:09.166929+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.505434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":527,"num_object_clones":0,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":527,"num_write_kb":33728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":335,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.958229+0000","last_change":"2024-03-26T03:04:42.754060+0000","last_active":"2024-03-26T03:05:08.958229+0000","last_peered":"2024-03-26T03:05:08.958229+0000","last_clean":"2024-03-26T03:05:08.958229+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:05:08.958229+0000","last_undegraded":"2024-03-26T03:05:08.958229+0000","last_fullsized":"2024-03-26T03:05:08.958229+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.753882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.753882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:58.379811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":282,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968588+0000","last_change":"2024-03-26T03:04:05.712151+0000","last_active":"2024-03-26T03:05:08.968588+0000","last_peered":"2024-03-26T03:05:08.968588+0000","last_clean":"2024-03-26T03:05:08.968588+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:05:08.968588+0000","last_undegraded":"2024-03-26T03:05:08.968588+0000","last_fullsized":"2024-03-26T03:05:08.968588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:05.711979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.711979+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:33.910298+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968869+0000","last_change":"2024-03-26T03:00:52.439831+0000","last_active":"2024-03-26T03:05:08.968869+0000","last_peered":"2024-03-26T03:05:08.968869+0000","last_clean":"2024-03-26T03:05:08.968869+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:05:08.968869+0000","last_undegraded":"2024-03-26T03:05:08.968869+0000","last_fullsized":"2024-03-26T03:05:08.968869+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.439710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.439710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":338,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455573+0000","last_change":"2024-03-26T03:03:56.360899+0000","last_active":"2024-03-26T03:05:11.455573+0000","last_peered":"2024-03-26T03:05:11.455573+0000","last_clean":"2024-03-26T03:05:11.455573+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:05:11.455573+0000","last_undegraded":"2024-03-26T03:05:11.455573+0000","last_fullsized":"2024-03-26T03:05:11.455573+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:03:56.360747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:56.360747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:23.660024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968419+0000","last_change":"2024-03-26T03:04:14.431790+0000","last_active":"2024-03-26T03:05:08.968419+0000","last_peered":"2024-03-26T03:05:08.968419+0000","last_clean":"2024-03-26T03:05:08.968419+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:05:08.968419+0000","last_undegraded":"2024-03-26T03:05:08.968419+0000","last_fullsized":"2024-03-26T03:05:08.968419+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.305791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.305791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:29.825899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"181'1206","reported_seq":1234,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:13.930394+0000","last_change":"2024-03-26T03:04:52.165534+0000","last_active":"2024-03-26T03:05:13.930394+0000","last_peered":"2024-03-26T03:05:13.930394+0000","last_clean":"2024-03-26T03:05:13.930394+0000","last_became_active":"2024-03-26T03:04:52.165202+0000","last_became_peered":"2024-03-26T03:04:52.165202+0000","last_unstale":"2024-03-26T03:05:13.930394+0000","last_undegraded":"2024-03-26T03:05:13.930394+0000","last_fullsized":"2024-03-26T03:05:13.930394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":1206,"log_dups_size":0,"ondisk_log_size":1206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.699745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79036416,"num_objects":1206,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1206,"num_write_kb":77184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":289,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455174+0000","last_change":"2024-03-26T03:03:37.340501+0000","last_active":"2024-03-26T03:05:11.455174+0000","last_peered":"2024-03-26T03:05:11.455174+0000","last_clean":"2024-03-26T03:05:11.455174+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:05:11.455174+0000","last_undegraded":"2024-03-26T03:05:11.455174+0000","last_fullsized":"2024-03-26T03:05:11.455174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:03:37.340409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:37.340409+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464491+0000","last_change":"2024-03-26T03:01:47.284940+0000","last_active":"2024-03-26T03:05:11.464491+0000","last_peered":"2024-03-26T03:05:11.464491+0000","last_clean":"2024-03-26T03:05:11.464491+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:05:11.464491+0000","last_undegraded":"2024-03-26T03:05:11.464491+0000","last_fullsized":"2024-03-26T03:05:11.464491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:47.284840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.284840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605365+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":56,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455613+0000","last_change":"2024-03-26T03:04:14.416201+0000","last_active":"2024-03-26T03:05:11.455613+0000","last_peered":"2024-03-26T03:05:11.455613+0000","last_clean":"2024-03-26T03:05:11.455613+0000","last_became_active":"2024-03-26T03:04:14.415770+0000","last_became_peered":"2024-03-26T03:04:14.415770+0000","last_unstale":"2024-03-26T03:05:11.455613+0000","last_undegraded":"2024-03-26T03:05:11.455613+0000","last_fullsized":"2024-03-26T03:05:11.455613+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:00.224975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:00.224975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:20.942429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.964946+0000","last_change":"2024-03-26T03:04:35.720859+0000","last_active":"2024-03-26T03:05:08.964946+0000","last_peered":"2024-03-26T03:05:08.964946+0000","last_clean":"2024-03-26T03:05:08.964946+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:05:08.964946+0000","last_undegraded":"2024-03-26T03:05:08.964946+0000","last_fullsized":"2024-03-26T03:05:08.964946+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:02.340517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"181'1231","reported_seq":1259,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:13.998637+0000","last_change":"2024-03-26T03:04:52.197065+0000","last_active":"2024-03-26T03:05:13.998637+0000","last_peered":"2024-03-26T03:05:13.998637+0000","last_clean":"2024-03-26T03:05:13.998637+0000","last_became_active":"2024-03-26T03:04:52.196605+0000","last_became_peered":"2024-03-26T03:04:52.196605+0000","last_unstale":"2024-03-26T03:05:13.998637+0000","last_undegraded":"2024-03-26T03:05:13.998637+0000","last_fullsized":"2024-03-26T03:05:13.998637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":1231,"log_dups_size":0,"ondisk_log_size":1231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:13.687550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":3693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":78784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"91'9","reported_seq":330,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966304+0000","last_change":"2024-03-26T03:00:37.456923+0000","last_active":"2024-03-26T03:05:08.966304+0000","last_peered":"2024-03-26T03:05:08.966304+0000","last_clean":"2024-03-26T03:05:08.966304+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:05:08.966304+0000","last_undegraded":"2024-03-26T03:05:08.966304+0000","last_fullsized":"2024-03-26T03:05:08.966304+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:00:37.456826+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":292,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.454543+0000","last_change":"2024-03-26T03:03:35.345801+0000","last_active":"2024-03-26T03:05:11.454543+0000","last_peered":"2024-03-26T03:05:11.454543+0000","last_clean":"2024-03-26T03:05:11.454543+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:05:11.454543+0000","last_undegraded":"2024-03-26T03:05:11.454543+0000","last_fullsized":"2024-03-26T03:05:11.454543+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:03:35.345668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:35.345668+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.462068+0000","last_change":"2024-03-26T03:01:40.975997+0000","last_active":"2024-03-26T03:05:11.462068+0000","last_peered":"2024-03-26T03:05:11.462068+0000","last_clean":"2024-03-26T03:05:11.462068+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:05:11.462068+0000","last_undegraded":"2024-03-26T03:05:11.462068+0000","last_fullsized":"2024-03-26T03:05:11.462068+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:40.975886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.975886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465126+0000","last_change":"2024-03-26T03:04:34.494561+0000","last_active":"2024-03-26T03:05:11.465126+0000","last_peered":"2024-03-26T03:05:11.465126+0000","last_clean":"2024-03-26T03:05:11.465126+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:05:11.465126+0000","last_undegraded":"2024-03-26T03:05:11.465126+0000","last_fullsized":"2024-03-26T03:05:11.465126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.494481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.494481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:42.566729+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":335,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965212+0000","last_change":"2024-03-26T03:04:42.754550+0000","last_active":"2024-03-26T03:05:08.965212+0000","last_peered":"2024-03-26T03:05:08.965212+0000","last_clean":"2024-03-26T03:05:08.965212+0000","last_became_active":"2024-03-26T03:04:41.967378+0000","last_became_peered":"2024-03-26T03:04:41.967378+0000","last_unstale":"2024-03-26T03:05:08.965212+0000","last_undegraded":"2024-03-26T03:05:08.965212+0000","last_fullsized":"2024-03-26T03:05:08.965212+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.754402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.754402+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:02.232433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"180'643","reported_seq":669,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:10.069353+0000","last_change":"2024-03-26T03:04:52.164796+0000","last_active":"2024-03-26T03:05:10.069353+0000","last_peered":"2024-03-26T03:05:10.069353+0000","last_clean":"2024-03-26T03:05:10.069353+0000","last_became_active":"2024-03-26T03:04:52.164450+0000","last_became_peered":"2024-03-26T03:04:52.164450+0000","last_unstale":"2024-03-26T03:05:10.069353+0000","last_undegraded":"2024-03-26T03:05:10.069353+0000","last_fullsized":"2024-03-26T03:05:10.069353+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:18.393856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42139648,"num_objects":643,"num_object_clones":0,"num_object_copies":1929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":643,"num_write_kb":41152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":247,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.454832+0000","last_change":"2024-03-26T03:01:41.237613+0000","last_active":"2024-03-26T03:05:11.454832+0000","last_peered":"2024-03-26T03:05:11.454832+0000","last_clean":"2024-03-26T03:05:11.454832+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:05:11.454832+0000","last_undegraded":"2024-03-26T03:05:11.454832+0000","last_fullsized":"2024-03-26T03:05:11.454832+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:41.236598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:41.236598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846537+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":326,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966204+0000","last_change":"2024-03-26T03:01:36.769019+0000","last_active":"2024-03-26T03:05:08.966204+0000","last_peered":"2024-03-26T03:05:08.966204+0000","last_clean":"2024-03-26T03:05:08.966204+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:05:08.966204+0000","last_undegraded":"2024-03-26T03:05:08.966204+0000","last_fullsized":"2024-03-26T03:05:08.966204+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968341+0000","last_change":"2024-03-26T03:04:32.627947+0000","last_active":"2024-03-26T03:05:08.968341+0000","last_peered":"2024-03-26T03:05:08.968341+0000","last_clean":"2024-03-26T03:05:08.968341+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:05:08.968341+0000","last_undegraded":"2024-03-26T03:05:08.968341+0000","last_fullsized":"2024-03-26T03:05:08.968341+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:04:32.627849+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:04:32.627849+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:57.916085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":242,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455499+0000","last_change":"2024-03-26T03:03:29.432831+0000","last_active":"2024-03-26T03:05:11.455499+0000","last_peered":"2024-03-26T03:05:11.455499+0000","last_clean":"2024-03-26T03:05:11.455499+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:05:11.455499+0000","last_undegraded":"2024-03-26T03:05:11.455499+0000","last_fullsized":"2024-03-26T03:05:11.455499+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:29.432698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:29.432698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846606+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":337,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455880+0000","last_change":"2024-03-26T03:04:42.002277+0000","last_active":"2024-03-26T03:05:11.455880+0000","last_peered":"2024-03-26T03:05:11.455880+0000","last_clean":"2024-03-26T03:05:11.455880+0000","last_became_active":"2024-03-26T03:04:41.967264+0000","last_became_peered":"2024-03-26T03:04:41.967264+0000","last_unstale":"2024-03-26T03:05:11.455880+0000","last_undegraded":"2024-03-26T03:05:11.455880+0000","last_fullsized":"2024-03-26T03:05:11.455880+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.002145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.002145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:07.546577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"180'615","reported_seq":641,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.759741+0000","last_change":"2024-03-26T03:04:52.198930+0000","last_active":"2024-03-26T03:05:09.759741+0000","last_peered":"2024-03-26T03:05:09.759741+0000","last_clean":"2024-03-26T03:05:09.759741+0000","last_became_active":"2024-03-26T03:04:52.198593+0000","last_became_peered":"2024-03-26T03:04:52.198593+0000","last_unstale":"2024-03-26T03:05:09.759741+0000","last_undegraded":"2024-03-26T03:05:09.759741+0000","last_fullsized":"2024-03-26T03:05:09.759741+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:51.341071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":615,"num_object_clones":0,"num_object_copies":1845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":615,"num_write_kb":39360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.461038+0000","last_change":"2024-03-26T03:03:49.866282+0000","last_active":"2024-03-26T03:05:11.461038+0000","last_peered":"2024-03-26T03:05:11.461038+0000","last_clean":"2024-03-26T03:05:11.461038+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:05:11.461038+0000","last_undegraded":"2024-03-26T03:05:11.461038+0000","last_fullsized":"2024-03-26T03:05:11.461038+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:49.866190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:49.866190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785607+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":305,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962719+0000","last_change":"2024-03-26T03:01:47.898281+0000","last_active":"2024-03-26T03:05:08.962719+0000","last_peered":"2024-03-26T03:05:08.962719+0000","last_clean":"2024-03-26T03:05:08.962719+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:05:08.962719+0000","last_undegraded":"2024-03-26T03:05:08.962719+0000","last_fullsized":"2024-03-26T03:05:08.962719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:01:47.898135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:47.898135+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465463+0000","last_change":"2024-03-26T03:04:35.520145+0000","last_active":"2024-03-26T03:05:11.465463+0000","last_peered":"2024-03-26T03:05:11.465463+0000","last_clean":"2024-03-26T03:05:11.465463+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:05:11.465463+0000","last_undegraded":"2024-03-26T03:05:11.465463+0000","last_fullsized":"2024-03-26T03:05:11.465463+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.520064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.520064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:48.039960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969056+0000","last_change":"2024-03-26T03:04:04.660434+0000","last_active":"2024-03-26T03:05:08.969056+0000","last_peered":"2024-03-26T03:05:08.969056+0000","last_clean":"2024-03-26T03:05:08.969056+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:05:08.969056+0000","last_undegraded":"2024-03-26T03:05:08.969056+0000","last_fullsized":"2024-03-26T03:05:08.969056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.660320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.660320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:20.401750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":343,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.461755+0000","last_change":"2024-03-26T03:04:43.812006+0000","last_active":"2024-03-26T03:05:11.461755+0000","last_peered":"2024-03-26T03:05:11.461755+0000","last_clean":"2024-03-26T03:05:11.461755+0000","last_became_active":"2024-03-26T03:04:41.970404+0000","last_became_peered":"2024-03-26T03:04:41.970404+0000","last_unstale":"2024-03-26T03:05:11.461755+0000","last_undegraded":"2024-03-26T03:05:11.461755+0000","last_fullsized":"2024-03-26T03:05:11.461755+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:48.587501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"180'529","reported_seq":555,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.155374+0000","last_change":"2024-03-26T03:04:52.198694+0000","last_active":"2024-03-26T03:05:09.155374+0000","last_peered":"2024-03-26T03:05:09.155374+0000","last_clean":"2024-03-26T03:05:09.155374+0000","last_became_active":"2024-03-26T03:04:52.198372+0000","last_became_peered":"2024-03-26T03:04:52.198372+0000","last_unstale":"2024-03-26T03:05:09.155374+0000","last_undegraded":"2024-03-26T03:05:09.155374+0000","last_fullsized":"2024-03-26T03:05:09.155374+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:19.398452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":529,"num_write_kb":33856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459511+0000","last_change":"2024-03-26T03:04:05.281972+0000","last_active":"2024-03-26T03:05:11.459511+0000","last_peered":"2024-03-26T03:05:11.459511+0000","last_clean":"2024-03-26T03:05:11.459511+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:05:11.459511+0000","last_undegraded":"2024-03-26T03:05:11.459511+0000","last_fullsized":"2024-03-26T03:05:11.459511+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:05.281823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.281823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:34.500521+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":275,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.964811+0000","last_change":"2024-03-26T03:01:33.900763+0000","last_active":"2024-03-26T03:05:08.964811+0000","last_peered":"2024-03-26T03:05:08.964811+0000","last_clean":"2024-03-26T03:05:08.964811+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:05:08.964811+0000","last_undegraded":"2024-03-26T03:05:08.964811+0000","last_fullsized":"2024-03-26T03:05:08.964811+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:33.900630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.900630+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464441+0000","last_change":"2024-03-26T03:03:53.328993+0000","last_active":"2024-03-26T03:05:11.464441+0000","last_peered":"2024-03-26T03:05:11.464441+0000","last_clean":"2024-03-26T03:05:11.464441+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:05:11.464441+0000","last_undegraded":"2024-03-26T03:05:11.464441+0000","last_fullsized":"2024-03-26T03:05:11.464441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.328899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.328899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":269,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962191+0000","last_change":"2024-03-26T03:04:36.749372+0000","last_active":"2024-03-26T03:05:08.962191+0000","last_peered":"2024-03-26T03:05:08.962191+0000","last_clean":"2024-03-26T03:05:08.962191+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:05:08.962191+0000","last_undegraded":"2024-03-26T03:05:08.962191+0000","last_fullsized":"2024-03-26T03:05:08.962191+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.749289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.749289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:45.385780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968552+0000","last_change":"2024-03-26T03:04:34.648671+0000","last_active":"2024-03-26T03:05:08.968552+0000","last_peered":"2024-03-26T03:05:08.968552+0000","last_clean":"2024-03-26T03:05:08.968552+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:05:08.968552+0000","last_undegraded":"2024-03-26T03:05:08.968552+0000","last_fullsized":"2024-03-26T03:05:08.968552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.648562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.648562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:35.382356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":335,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.970067+0000","last_change":"2024-03-26T03:04:42.604828+0000","last_active":"2024-03-26T03:05:08.970067+0000","last_peered":"2024-03-26T03:05:08.970067+0000","last_clean":"2024-03-26T03:05:08.970067+0000","last_became_active":"2024-03-26T03:04:41.964807+0000","last_became_peered":"2024-03-26T03:04:41.964807+0000","last_unstale":"2024-03-26T03:05:08.970067+0000","last_undegraded":"2024-03-26T03:05:08.970067+0000","last_fullsized":"2024-03-26T03:05:08.970067+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.604727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.604727+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:52.917317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"181'1304","reported_seq":1332,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:14.005224+0000","last_change":"2024-03-26T03:04:52.166972+0000","last_active":"2024-03-26T03:05:14.005224+0000","last_peered":"2024-03-26T03:05:14.005224+0000","last_clean":"2024-03-26T03:05:14.005224+0000","last_became_active":"2024-03-26T03:04:52.166591+0000","last_became_peered":"2024-03-26T03:04:52.166591+0000","last_unstale":"2024-03-26T03:05:14.005224+0000","last_undegraded":"2024-03-26T03:05:14.005224+0000","last_fullsized":"2024-03-26T03:05:14.005224+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":1304,"log_dups_size":0,"ondisk_log_size":1304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:04.715629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85458944,"num_objects":1304,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1304,"num_write_kb":83456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465333+0000","last_change":"2024-03-26T03:01:48.249675+0000","last_active":"2024-03-26T03:05:11.465333+0000","last_peered":"2024-03-26T03:05:11.465333+0000","last_clean":"2024-03-26T03:05:11.465333+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:05:11.465333+0000","last_undegraded":"2024-03-26T03:05:11.465333+0000","last_fullsized":"2024-03-26T03:05:11.465333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:48.249567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:48.249567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":357,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.461797+0000","last_change":"2024-03-26T03:01:42.020043+0000","last_active":"2024-03-26T03:05:11.461797+0000","last_peered":"2024-03-26T03:05:11.461797+0000","last_clean":"2024-03-26T03:05:11.461797+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:05:11.461797+0000","last_undegraded":"2024-03-26T03:05:11.461797+0000","last_fullsized":"2024-03-26T03:05:11.461797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:01:42.019820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:42.019820+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785541+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":520,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464436+0000","last_change":"2024-03-26T03:04:37.514101+0000","last_active":"2024-03-26T03:05:11.464436+0000","last_peered":"2024-03-26T03:05:11.464436+0000","last_clean":"2024-03-26T03:05:11.464436+0000","last_became_active":"2024-03-26T03:04:37.115068+0000","last_became_peered":"2024-03-26T03:04:37.115068+0000","last_unstale":"2024-03-26T03:05:11.464436+0000","last_undegraded":"2024-03-26T03:05:11.464436+0000","last_fullsized":"2024-03-26T03:05:11.464436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:37.514019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:37.514019+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:06.164221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.458612+0000","last_change":"2024-03-26T03:04:34.328064+0000","last_active":"2024-03-26T03:05:11.458612+0000","last_peered":"2024-03-26T03:05:11.458612+0000","last_clean":"2024-03-26T03:05:11.458612+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:05:11.458612+0000","last_undegraded":"2024-03-26T03:05:11.458612+0000","last_fullsized":"2024-03-26T03:05:11.458612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.327936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.327936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:39.482498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":262,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965049+0000","last_change":"2024-03-26T03:00:45.454030+0000","last_active":"2024-03-26T03:05:08.965049+0000","last_peered":"2024-03-26T03:05:08.965049+0000","last_clean":"2024-03-26T03:05:08.965049+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:05:08.965049+0000","last_undegraded":"2024-03-26T03:05:08.965049+0000","last_fullsized":"2024-03-26T03:05:08.965049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:00:45.453925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:45.453925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459600+0000","last_change":"2024-03-26T03:04:36.275126+0000","last_active":"2024-03-26T03:05:11.459600+0000","last_peered":"2024-03-26T03:05:11.459600+0000","last_clean":"2024-03-26T03:05:11.459600+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:05:11.459600+0000","last_undegraded":"2024-03-26T03:05:11.459600+0000","last_fullsized":"2024-03-26T03:05:11.459600+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.985829+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"180'626","reported_seq":652,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:09.733433+0000","last_change":"2024-03-26T03:04:52.157100+0000","last_active":"2024-03-26T03:05:09.733433+0000","last_peered":"2024-03-26T03:05:09.733433+0000","last_clean":"2024-03-26T03:05:09.733433+0000","last_became_active":"2024-03-26T03:04:52.156769+0000","last_became_peered":"2024-03-26T03:04:52.156769+0000","last_unstale":"2024-03-26T03:05:09.733433+0000","last_undegraded":"2024-03-26T03:05:09.733433+0000","last_fullsized":"2024-03-26T03:05:09.733433+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:06.242188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41025536,"num_objects":626,"num_object_clones":0,"num_object_copies":1878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":626,"num_write_kb":40064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.960772+0000","last_change":"2024-03-26T03:01:44.370403+0000","last_active":"2024-03-26T03:05:08.960772+0000","last_peered":"2024-03-26T03:05:08.960772+0000","last_clean":"2024-03-26T03:05:08.960772+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:05:08.960772+0000","last_undegraded":"2024-03-26T03:05:08.960772+0000","last_fullsized":"2024-03-26T03:05:08.960772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:44.370278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:44.370278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.477081+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":296,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968782+0000","last_change":"2024-03-26T03:04:36.643519+0000","last_active":"2024-03-26T03:05:08.968782+0000","last_peered":"2024-03-26T03:05:08.968782+0000","last_clean":"2024-03-26T03:05:08.968782+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:05:08.968782+0000","last_undegraded":"2024-03-26T03:05:08.968782+0000","last_fullsized":"2024-03-26T03:05:08.968782+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:04:36.643371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.643371+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:51.493597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.959063+0000","last_change":"2024-03-26T03:01:45.357919+0000","last_active":"2024-03-26T03:05:08.959063+0000","last_peered":"2024-03-26T03:05:08.959063+0000","last_clean":"2024-03-26T03:05:08.959063+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:05:08.959063+0000","last_undegraded":"2024-03-26T03:05:08.959063+0000","last_fullsized":"2024-03-26T03:05:08.959063+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:45.357787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:45.357787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.476859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":635,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968506+0000","last_change":"2024-03-26T03:03:57.681467+0000","last_active":"2024-03-26T03:05:08.968506+0000","last_peered":"2024-03-26T03:05:08.968506+0000","last_clean":"2024-03-26T03:05:08.968506+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:05:08.968506+0000","last_undegraded":"2024-03-26T03:05:08.968506+0000","last_fullsized":"2024-03-26T03:05:08.968506+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:03:57.679470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:03:57.679470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:10.270071+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":363,"num_read_kb":358,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465309+0000","last_change":"2024-03-26T03:04:06.566392+0000","last_active":"2024-03-26T03:05:11.465309+0000","last_peered":"2024-03-26T03:05:11.465309+0000","last_clean":"2024-03-26T03:05:11.465309+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:05:11.465309+0000","last_undegraded":"2024-03-26T03:05:11.465309+0000","last_fullsized":"2024-03-26T03:05:11.465309+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:06.566290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:06.566290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":527,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464873+0000","last_change":"2024-03-26T03:04:43.651718+0000","last_active":"2024-03-26T03:05:11.464873+0000","last_peered":"2024-03-26T03:05:11.464873+0000","last_clean":"2024-03-26T03:05:11.464873+0000","last_became_active":"2024-03-26T03:04:41.965091+0000","last_became_peered":"2024-03-26T03:04:41.965091+0000","last_unstale":"2024-03-26T03:05:11.464873+0000","last_undegraded":"2024-03-26T03:05:11.464873+0000","last_fullsized":"2024-03-26T03:05:11.464873+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.651633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.651633+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:09.790183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"181'1305","reported_seq":1333,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:13.967868+0000","last_change":"2024-03-26T03:04:52.164646+0000","last_active":"2024-03-26T03:05:13.967868+0000","last_peered":"2024-03-26T03:05:13.967868+0000","last_clean":"2024-03-26T03:05:13.967868+0000","last_became_active":"2024-03-26T03:04:52.164220+0000","last_became_peered":"2024-03-26T03:04:52.164220+0000","last_unstale":"2024-03-26T03:05:13.967868+0000","last_undegraded":"2024-03-26T03:05:13.967868+0000","last_fullsized":"2024-03-26T03:05:13.967868+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":1305,"log_dups_size":0,"ondisk_log_size":1305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:18.073246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85524480,"num_objects":1305,"num_object_clones":0,"num_object_copies":3915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1305,"num_write_kb":83520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962457+0000","last_change":"2024-03-26T03:04:01.972787+0000","last_active":"2024-03-26T03:05:08.962457+0000","last_peered":"2024-03-26T03:05:08.962457+0000","last_clean":"2024-03-26T03:05:08.962457+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:05:08.962457+0000","last_undegraded":"2024-03-26T03:05:08.962457+0000","last_fullsized":"2024-03-26T03:05:08.962457+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.972671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.972671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:29.228532+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":403,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968088+0000","last_change":"2024-03-26T03:04:01.624276+0000","last_active":"2024-03-26T03:05:08.968088+0000","last_peered":"2024-03-26T03:05:08.968088+0000","last_clean":"2024-03-26T03:05:08.968088+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:05:08.968088+0000","last_undegraded":"2024-03-26T03:05:08.968088+0000","last_fullsized":"2024-03-26T03:05:08.968088+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:04:01.623984+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.623984+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:10.506596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969274+0000","last_change":"2024-03-26T03:04:14.420602+0000","last_active":"2024-03-26T03:05:08.969274+0000","last_peered":"2024-03-26T03:05:08.969274+0000","last_clean":"2024-03-26T03:05:08.969274+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:05:08.969274+0000","last_undegraded":"2024-03-26T03:05:08.969274+0000","last_fullsized":"2024-03-26T03:05:08.969274+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:06.319990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:06.319990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:25.788201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":241,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965743+0000","last_change":"2024-03-26T03:01:33.720224+0000","last_active":"2024-03-26T03:05:08.965743+0000","last_peered":"2024-03-26T03:05:08.965743+0000","last_clean":"2024-03-26T03:05:08.965743+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:05:08.965743+0000","last_undegraded":"2024-03-26T03:05:08.965743+0000","last_fullsized":"2024-03-26T03:05:08.965743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.719795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.719795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":341,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962025+0000","last_change":"2024-03-26T03:04:42.737754+0000","last_active":"2024-03-26T03:05:08.962025+0000","last_peered":"2024-03-26T03:05:08.962025+0000","last_clean":"2024-03-26T03:05:08.962025+0000","last_became_active":"2024-03-26T03:04:41.966771+0000","last_became_peered":"2024-03-26T03:04:41.966771+0000","last_unstale":"2024-03-26T03:05:08.962025+0000","last_undegraded":"2024-03-26T03:05:08.962025+0000","last_fullsized":"2024-03-26T03:05:08.962025+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.737606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.737606+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:06.784665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"180'638","reported_seq":664,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:10.069819+0000","last_change":"2024-03-26T03:04:52.197778+0000","last_active":"2024-03-26T03:05:10.069819+0000","last_peered":"2024-03-26T03:05:10.069819+0000","last_clean":"2024-03-26T03:05:10.069819+0000","last_became_active":"2024-03-26T03:04:52.197016+0000","last_became_peered":"2024-03-26T03:04:52.197016+0000","last_unstale":"2024-03-26T03:05:10.069819+0000","last_undegraded":"2024-03-26T03:05:10.069819+0000","last_fullsized":"2024-03-26T03:05:10.069819+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:07.883964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41811968,"num_objects":638,"num_object_clones":0,"num_object_copies":1914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":339,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962896+0000","last_change":"2024-03-26T03:01:40.859597+0000","last_active":"2024-03-26T03:05:08.962896+0000","last_peered":"2024-03-26T03:05:08.962896+0000","last_clean":"2024-03-26T03:05:08.962896+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:05:08.962896+0000","last_undegraded":"2024-03-26T03:05:08.962896+0000","last_fullsized":"2024-03-26T03:05:08.962896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:01:40.859466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.859466+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"92'22","reported_seq":282,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969614+0000","last_change":"2024-03-26T03:00:42.399658+0000","last_active":"2024-03-26T03:05:08.969614+0000","last_peered":"2024-03-26T03:05:08.969614+0000","last_clean":"2024-03-26T03:05:08.969614+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:05:08.969614+0000","last_undegraded":"2024-03-26T03:05:08.969614+0000","last_fullsized":"2024-03-26T03:05:08.969614+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:00:42.399566+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968217+0000","last_change":"2024-03-26T03:04:33.655187+0000","last_active":"2024-03-26T03:05:08.968217+0000","last_peered":"2024-03-26T03:05:08.968217+0000","last_clean":"2024-03-26T03:05:08.968217+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:05:08.968217+0000","last_undegraded":"2024-03-26T03:05:08.968217+0000","last_fullsized":"2024-03-26T03:05:08.968217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:33.655103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:33.655103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:35.914728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":250,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.961783+0000","last_change":"2024-03-26T03:03:53.959327+0000","last_active":"2024-03-26T03:05:08.961783+0000","last_peered":"2024-03-26T03:05:08.961783+0000","last_clean":"2024-03-26T03:05:08.961783+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:05:08.961783+0000","last_undegraded":"2024-03-26T03:05:08.961783+0000","last_fullsized":"2024-03-26T03:05:08.961783+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.959237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":343,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.461974+0000","last_change":"2024-03-26T03:04:42.846233+0000","last_active":"2024-03-26T03:05:11.461974+0000","last_peered":"2024-03-26T03:05:11.461974+0000","last_clean":"2024-03-26T03:05:11.461974+0000","last_became_active":"2024-03-26T03:04:41.969936+0000","last_became_peered":"2024-03-26T03:04:41.969936+0000","last_unstale":"2024-03-26T03:05:11.461974+0000","last_undegraded":"2024-03-26T03:05:11.461974+0000","last_fullsized":"2024-03-26T03:05:11.461974+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:55.734011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"181'1340","reported_seq":1368,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:14.004592+0000","last_change":"2024-03-26T03:04:52.164436+0000","last_active":"2024-03-26T03:05:14.004592+0000","last_peered":"2024-03-26T03:05:14.004592+0000","last_clean":"2024-03-26T03:05:14.004592+0000","last_became_active":"2024-03-26T03:04:52.164107+0000","last_became_peered":"2024-03-26T03:04:52.164107+0000","last_unstale":"2024-03-26T03:05:14.004592+0000","last_undegraded":"2024-03-26T03:05:14.004592+0000","last_fullsized":"2024-03-26T03:05:14.004592+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":1340,"log_dups_size":0,"ondisk_log_size":1340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:20.354862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1340,"num_write_kb":85760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.957757+0000","last_change":"2024-03-26T03:04:38.837560+0000","last_active":"2024-03-26T03:05:08.957757+0000","last_peered":"2024-03-26T03:05:08.957757+0000","last_clean":"2024-03-26T03:05:08.957757+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:05:08.957757+0000","last_undegraded":"2024-03-26T03:05:08.957757+0000","last_fullsized":"2024-03-26T03:05:08.957757+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:38.837480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:38.837480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:47.988811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":327,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.454691+0000","last_change":"2024-03-26T03:03:50.380060+0000","last_active":"2024-03-26T03:05:11.454691+0000","last_peered":"2024-03-26T03:05:11.454691+0000","last_clean":"2024-03-26T03:05:11.454691+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:05:11.454691+0000","last_undegraded":"2024-03-26T03:05:11.454691+0000","last_fullsized":"2024-03-26T03:05:11.454691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846896+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":280,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455449+0000","last_change":"2024-03-26T03:04:47.886299+0000","last_active":"2024-03-26T03:05:11.455449+0000","last_peered":"2024-03-26T03:05:11.455449+0000","last_clean":"2024-03-26T03:05:11.455449+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:05:11.455449+0000","last_undegraded":"2024-03-26T03:05:11.455449+0000","last_fullsized":"2024-03-26T03:05:11.455449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:04:47.886200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:47.886200+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:49.716601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":241,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966319+0000","last_change":"2024-03-26T03:00:35.424045+0000","last_active":"2024-03-26T03:05:08.966319+0000","last_peered":"2024-03-26T03:05:08.966319+0000","last_clean":"2024-03-26T03:05:08.966319+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:05:08.966319+0000","last_undegraded":"2024-03-26T03:05:08.966319+0000","last_fullsized":"2024-03-26T03:05:08.966319+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:35.423904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:35.423904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":297,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.964935+0000","last_change":"2024-03-26T03:03:59.969483+0000","last_active":"2024-03-26T03:05:08.964935+0000","last_peered":"2024-03-26T03:05:08.964935+0000","last_clean":"2024-03-26T03:05:08.964935+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:05:08.964935+0000","last_undegraded":"2024-03-26T03:05:08.964935+0000","last_fullsized":"2024-03-26T03:05:08.964935+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:03:59.969202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:59.969202+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:24.335149+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":267,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.957660+0000","last_change":"2024-03-26T03:01:46.408351+0000","last_active":"2024-03-26T03:05:08.957660+0000","last_peered":"2024-03-26T03:05:08.957660+0000","last_clean":"2024-03-26T03:05:08.957660+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:05:08.957660+0000","last_undegraded":"2024-03-26T03:05:08.957660+0000","last_fullsized":"2024-03-26T03:05:08.957660+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:01:46.408202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:46.408202+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.477039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966268+0000","last_change":"2024-03-26T03:01:38.802880+0000","last_active":"2024-03-26T03:05:08.966268+0000","last_peered":"2024-03-26T03:05:08.966268+0000","last_clean":"2024-03-26T03:05:08.966268+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:05:08.966268+0000","last_undegraded":"2024-03-26T03:05:08.966268+0000","last_fullsized":"2024-03-26T03:05:08.966268+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:38.802745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:38.802745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.959369+0000","last_change":"2024-03-26T03:01:52.467228+0000","last_active":"2024-03-26T03:05:08.959369+0000","last_peered":"2024-03-26T03:05:08.959369+0000","last_clean":"2024-03-26T03:05:08.959369+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:05:08.959369+0000","last_undegraded":"2024-03-26T03:05:08.959369+0000","last_fullsized":"2024-03-26T03:05:08.959369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:52.467134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:52.467134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.477229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":294,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965942+0000","last_change":"2024-03-26T03:01:50.876168+0000","last_active":"2024-03-26T03:05:08.965942+0000","last_peered":"2024-03-26T03:05:08.965942+0000","last_clean":"2024-03-26T03:05:08.965942+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:05:08.965942+0000","last_undegraded":"2024-03-26T03:05:08.965942+0000","last_fullsized":"2024-03-26T03:05:08.965942+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.460768+0000","last_change":"2024-03-26T03:01:32.933502+0000","last_active":"2024-03-26T03:05:11.460768+0000","last_peered":"2024-03-26T03:05:11.460768+0000","last_clean":"2024-03-26T03:05:11.460768+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:05:11.460768+0000","last_undegraded":"2024-03-26T03:05:11.460768+0000","last_fullsized":"2024-03-26T03:05:11.460768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:32.933389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:32.933389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459014+0000","last_change":"2024-03-26T03:03:56.206476+0000","last_active":"2024-03-26T03:05:11.459014+0000","last_peered":"2024-03-26T03:05:11.459014+0000","last_clean":"2024-03-26T03:05:11.459014+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:05:11.459014+0000","last_undegraded":"2024-03-26T03:05:11.459014+0000","last_fullsized":"2024-03-26T03:05:11.459014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:56.206361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:56.206361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:15.870388+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465183+0000","last_change":"2024-03-26T03:01:34.228140+0000","last_active":"2024-03-26T03:05:11.465183+0000","last_peered":"2024-03-26T03:05:11.465183+0000","last_clean":"2024-03-26T03:05:11.465183+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:05:11.465183+0000","last_undegraded":"2024-03-26T03:05:11.465183+0000","last_fullsized":"2024-03-26T03:05:11.465183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.227979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.227979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605338+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"151'29","reported_seq":303,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965389+0000","last_change":"2024-03-26T03:03:52.392314+0000","last_active":"2024-03-26T03:05:08.965389+0000","last_peered":"2024-03-26T03:05:08.965389+0000","last_clean":"2024-03-26T03:05:08.965389+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:05:08.965389+0000","last_undegraded":"2024-03-26T03:05:08.965389+0000","last_fullsized":"2024-03-26T03:05:08.965389+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:15.632896+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459180+0000","last_change":"2024-03-26T03:04:40.320823+0000","last_active":"2024-03-26T03:05:11.459180+0000","last_peered":"2024-03-26T03:05:11.459180+0000","last_clean":"2024-03-26T03:05:11.459180+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:05:11.459180+0000","last_undegraded":"2024-03-26T03:05:11.459180+0000","last_fullsized":"2024-03-26T03:05:11.459180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.320691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:40.531749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":338,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459141+0000","last_change":"2024-03-26T03:03:55.245804+0000","last_active":"2024-03-26T03:05:11.459141+0000","last_peered":"2024-03-26T03:05:11.459141+0000","last_clean":"2024-03-26T03:05:11.459141+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:05:11.459141+0000","last_undegraded":"2024-03-26T03:05:11.459141+0000","last_fullsized":"2024-03-26T03:05:11.459141+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:03:55.245713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:55.245713+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:24.948940+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.964700+0000","last_change":"2024-03-26T03:01:39.883186+0000","last_active":"2024-03-26T03:05:08.964700+0000","last_peered":"2024-03-26T03:05:08.964700+0000","last_clean":"2024-03-26T03:05:08.964700+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:05:08.964700+0000","last_undegraded":"2024-03-26T03:05:08.964700+0000","last_fullsized":"2024-03-26T03:05:08.964700+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:39.883059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:39.883059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":103,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465354+0000","last_change":"2024-03-26T03:04:14.416461+0000","last_active":"2024-03-26T03:05:11.465354+0000","last_peered":"2024-03-26T03:05:11.465354+0000","last_clean":"2024-03-26T03:05:11.465354+0000","last_became_active":"2024-03-26T03:04:14.415940+0000","last_became_peered":"2024-03-26T03:04:14.415940+0000","last_unstale":"2024-03-26T03:05:11.465354+0000","last_undegraded":"2024-03-26T03:05:11.465354+0000","last_fullsized":"2024-03-26T03:05:11.465354+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:24.578052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962565+0000","last_change":"2024-03-26T03:04:46.666940+0000","last_active":"2024-03-26T03:05:08.962565+0000","last_peered":"2024-03-26T03:05:08.962565+0000","last_clean":"2024-03-26T03:05:08.962565+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:05:08.962565+0000","last_undegraded":"2024-03-26T03:05:08.962565+0000","last_fullsized":"2024-03-26T03:05:08.962565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:05.225883+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":336,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965301+0000","last_change":"2024-03-26T03:04:44.796750+0000","last_active":"2024-03-26T03:05:08.965301+0000","last_peered":"2024-03-26T03:05:08.965301+0000","last_clean":"2024-03-26T03:05:08.965301+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:05:08.965301+0000","last_undegraded":"2024-03-26T03:05:08.965301+0000","last_fullsized":"2024-03-26T03:05:08.965301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.839204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455705+0000","last_change":"2024-03-26T03:01:53.315527+0000","last_active":"2024-03-26T03:05:11.455705+0000","last_peered":"2024-03-26T03:05:11.455705+0000","last_clean":"2024-03-26T03:05:11.455705+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:05:11.455705+0000","last_undegraded":"2024-03-26T03:05:11.455705+0000","last_fullsized":"2024-03-26T03:05:11.455705+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:01:53.315442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:53.315442+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846576+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":265,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969515+0000","last_change":"2024-03-26T03:01:40.604091+0000","last_active":"2024-03-26T03:05:08.969515+0000","last_peered":"2024-03-26T03:05:08.969515+0000","last_clean":"2024-03-26T03:05:08.969515+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:05:08.969515+0000","last_undegraded":"2024-03-26T03:05:08.969515+0000","last_fullsized":"2024-03-26T03:05:08.969515+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-26T03:01:40.603834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:40.603834+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.461271+0000","last_change":"2024-03-26T03:01:33.891331+0000","last_active":"2024-03-26T03:05:11.461271+0000","last_peered":"2024-03-26T03:05:11.461271+0000","last_clean":"2024-03-26T03:05:11.461271+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:05:11.461271+0000","last_undegraded":"2024-03-26T03:05:11.461271+0000","last_fullsized":"2024-03-26T03:05:11.461271+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.891247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.891247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966288+0000","last_change":"2024-03-26T03:03:44.286188+0000","last_active":"2024-03-26T03:05:08.966288+0000","last_peered":"2024-03-26T03:05:08.966288+0000","last_clean":"2024-03-26T03:05:08.966288+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:05:08.966288+0000","last_undegraded":"2024-03-26T03:05:08.966288+0000","last_fullsized":"2024-03-26T03:05:08.966288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":244,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962161+0000","last_change":"2024-03-26T03:04:51.152529+0000","last_active":"2024-03-26T03:05:08.962161+0000","last_peered":"2024-03-26T03:05:08.962161+0000","last_clean":"2024-03-26T03:05:08.962161+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:05:08.962161+0000","last_undegraded":"2024-03-26T03:05:08.962161+0000","last_fullsized":"2024-03-26T03:05:08.962161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:44.275533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:44.275533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455192+0000","last_change":"2024-03-26T03:04:44.941104+0000","last_active":"2024-03-26T03:05:11.455192+0000","last_peered":"2024-03-26T03:05:11.455192+0000","last_clean":"2024-03-26T03:05:11.455192+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:05:11.455192+0000","last_undegraded":"2024-03-26T03:05:11.455192+0000","last_fullsized":"2024-03-26T03:05:11.455192+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:44.940984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.940984+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:55.600513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968687+0000","last_change":"2024-03-26T03:04:02.578194+0000","last_active":"2024-03-26T03:05:08.968687+0000","last_peered":"2024-03-26T03:05:08.968687+0000","last_clean":"2024-03-26T03:05:08.968687+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:05:08.968687+0000","last_undegraded":"2024-03-26T03:05:08.968687+0000","last_fullsized":"2024-03-26T03:05:08.968687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.578096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.831727+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":245,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969738+0000","last_change":"2024-03-26T03:03:45.582599+0000","last_active":"2024-03-26T03:05:08.969738+0000","last_peered":"2024-03-26T03:05:08.969738+0000","last_clean":"2024-03-26T03:05:08.969738+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:05:08.969738+0000","last_undegraded":"2024-03-26T03:05:08.969738+0000","last_fullsized":"2024-03-26T03:05:08.969738+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:45.582463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:45.582463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:10.559400+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.964942+0000","last_change":"2024-03-26T03:03:54.412437+0000","last_active":"2024-03-26T03:05:08.964942+0000","last_peered":"2024-03-26T03:05:08.964942+0000","last_clean":"2024-03-26T03:05:08.964942+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:05:08.964942+0000","last_undegraded":"2024-03-26T03:05:08.964942+0000","last_fullsized":"2024-03-26T03:05:08.964942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:23.413024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":263,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966030+0000","last_change":"2024-03-26T03:01:37.776671+0000","last_active":"2024-03-26T03:05:08.966030+0000","last_peered":"2024-03-26T03:05:08.966030+0000","last_clean":"2024-03-26T03:05:08.966030+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:05:08.966030+0000","last_undegraded":"2024-03-26T03:05:08.966030+0000","last_fullsized":"2024-03-26T03:05:08.966030+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459261+0000","last_change":"2024-03-26T03:04:35.287576+0000","last_active":"2024-03-26T03:05:11.459261+0000","last_peered":"2024-03-26T03:05:11.459261+0000","last_clean":"2024-03-26T03:05:11.459261+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:05:11.459261+0000","last_undegraded":"2024-03-26T03:05:11.459261+0000","last_fullsized":"2024-03-26T03:05:11.459261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.287500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.287500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:50.321216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.460741+0000","last_change":"2024-03-26T03:04:04.867638+0000","last_active":"2024-03-26T03:05:11.460741+0000","last_peered":"2024-03-26T03:05:11.460741+0000","last_clean":"2024-03-26T03:05:11.460741+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:05:11.460741+0000","last_undegraded":"2024-03-26T03:05:11.460741+0000","last_fullsized":"2024-03-26T03:05:11.460741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.867528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.867528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464937+0000","last_change":"2024-03-26T03:01:44.279749+0000","last_active":"2024-03-26T03:05:11.464937+0000","last_peered":"2024-03-26T03:05:11.464937+0000","last_clean":"2024-03-26T03:05:11.464937+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:05:11.464937+0000","last_undegraded":"2024-03-26T03:05:11.464937+0000","last_fullsized":"2024-03-26T03:05:11.464937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:44.279630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:44.279630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"150'23","reported_seq":294,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965604+0000","last_change":"2024-03-26T03:03:50.334581+0000","last_active":"2024-03-26T03:05:08.965604+0000","last_peered":"2024-03-26T03:05:08.965604+0000","last_clean":"2024-03-26T03:05:08.965604+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:05:08.965604+0000","last_undegraded":"2024-03-26T03:05:08.965604+0000","last_fullsized":"2024-03-26T03:05:08.965604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:15.704178+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.462062+0000","last_change":"2024-03-26T03:04:34.772569+0000","last_active":"2024-03-26T03:05:11.462062+0000","last_peered":"2024-03-26T03:05:11.462062+0000","last_clean":"2024-03-26T03:05:11.462062+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:05:11.462062+0000","last_undegraded":"2024-03-26T03:05:11.462062+0000","last_fullsized":"2024-03-26T03:05:11.462062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.772488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.772488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:52.450698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464665+0000","last_change":"2024-03-26T03:04:40.594869+0000","last_active":"2024-03-26T03:05:11.464665+0000","last_peered":"2024-03-26T03:05:11.464665+0000","last_clean":"2024-03-26T03:05:11.464665+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:05:11.464665+0000","last_undegraded":"2024-03-26T03:05:11.464665+0000","last_fullsized":"2024-03-26T03:05:11.464665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.594788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.412549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":241,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966388+0000","last_change":"2024-03-26T03:01:34.770522+0000","last_active":"2024-03-26T03:05:08.966388+0000","last_peered":"2024-03-26T03:05:08.966388+0000","last_clean":"2024-03-26T03:05:08.966388+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:05:08.966388+0000","last_undegraded":"2024-03-26T03:05:08.966388+0000","last_fullsized":"2024-03-26T03:05:08.966388+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:34.770057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:34.770057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":285,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965159+0000","last_change":"2024-03-26T03:04:04.095435+0000","last_active":"2024-03-26T03:05:08.965159+0000","last_peered":"2024-03-26T03:05:08.965159+0000","last_clean":"2024-03-26T03:05:08.965159+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:05:08.965159+0000","last_undegraded":"2024-03-26T03:05:08.965159+0000","last_fullsized":"2024-03-26T03:05:08.965159+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:04:04.095182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.095182+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:33.443816+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.459198+0000","last_change":"2024-03-26T03:04:42.310863+0000","last_active":"2024-03-26T03:05:11.459198+0000","last_peered":"2024-03-26T03:05:11.459198+0000","last_clean":"2024-03-26T03:05:11.459198+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:05:11.459198+0000","last_undegraded":"2024-03-26T03:05:11.459198+0000","last_fullsized":"2024-03-26T03:05:11.459198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:42.310782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.310782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:10.649449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":342,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.462190+0000","last_change":"2024-03-26T03:04:02.892852+0000","last_active":"2024-03-26T03:05:11.462190+0000","last_peered":"2024-03-26T03:05:11.462190+0000","last_clean":"2024-03-26T03:05:11.462190+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:05:11.462190+0000","last_undegraded":"2024-03-26T03:05:11.462190+0000","last_fullsized":"2024-03-26T03:05:11.462190+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:02.892717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.892717+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465660+0000","last_change":"2024-03-26T03:04:45.715721+0000","last_active":"2024-03-26T03:05:11.465660+0000","last_peered":"2024-03-26T03:05:11.465660+0000","last_clean":"2024-03-26T03:05:11.465660+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:05:11.465660+0000","last_undegraded":"2024-03-26T03:05:11.465660+0000","last_fullsized":"2024-03-26T03:05:11.465660+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:04:45.715635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:45.715635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:58.465028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":291,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.462200+0000","last_change":"2024-03-26T03:03:51.933569+0000","last_active":"2024-03-26T03:05:11.462200+0000","last_peered":"2024-03-26T03:05:11.462200+0000","last_clean":"2024-03-26T03:05:11.462200+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:05:11.462200+0000","last_undegraded":"2024-03-26T03:05:11.462200+0000","last_fullsized":"2024-03-26T03:05:11.462200+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:03:51.933431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:51.933431+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.785626+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":262,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965222+0000","last_change":"2024-03-26T03:04:40.740150+0000","last_active":"2024-03-26T03:05:08.965222+0000","last_peered":"2024-03-26T03:05:08.965222+0000","last_clean":"2024-03-26T03:05:08.965222+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:05:08.965222+0000","last_undegraded":"2024-03-26T03:05:08.965222+0000","last_fullsized":"2024-03-26T03:05:08.965222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:07.702274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965264+0000","last_change":"2024-03-26T03:01:41.862376+0000","last_active":"2024-03-26T03:05:08.965264+0000","last_peered":"2024-03-26T03:05:08.965264+0000","last_clean":"2024-03-26T03:05:08.965264+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:05:08.965264+0000","last_undegraded":"2024-03-26T03:05:08.965264+0000","last_fullsized":"2024-03-26T03:05:08.965264+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:41.862250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:01:41.862250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":241,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968527+0000","last_change":"2024-03-26T03:03:53.661347+0000","last_active":"2024-03-26T03:05:08.968527+0000","last_peered":"2024-03-26T03:05:08.968527+0000","last_clean":"2024-03-26T03:05:08.968527+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:05:08.968527+0000","last_undegraded":"2024-03-26T03:05:08.968527+0000","last_fullsized":"2024-03-26T03:05:08.968527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.661258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.661258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:09.534161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":290,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.960025+0000","last_change":"2024-03-26T03:04:44.684514+0000","last_active":"2024-03-26T03:05:08.960025+0000","last_peered":"2024-03-26T03:05:08.960025+0000","last_clean":"2024-03-26T03:05:08.960025+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:05:08.960025+0000","last_undegraded":"2024-03-26T03:05:08.960025+0000","last_fullsized":"2024-03-26T03:05:08.960025+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:08.955521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":271,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.460948+0000","last_change":"2024-03-26T03:04:44.859471+0000","last_active":"2024-03-26T03:05:11.460948+0000","last_peered":"2024-03-26T03:05:11.460948+0000","last_clean":"2024-03-26T03:05:11.460948+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:05:11.460948+0000","last_undegraded":"2024-03-26T03:05:11.460948+0000","last_fullsized":"2024-03-26T03:05:11.460948+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:44.859394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.859394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:49.587400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":253,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.969014+0000","last_change":"2024-03-26T03:04:03.635491+0000","last_active":"2024-03-26T03:05:08.969014+0000","last_peered":"2024-03-26T03:05:08.969014+0000","last_clean":"2024-03-26T03:05:08.969014+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:05:08.969014+0000","last_undegraded":"2024-03-26T03:05:08.969014+0000","last_fullsized":"2024-03-26T03:05:08.969014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:03.634773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:03.634773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:09.151370+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455290+0000","last_change":"2024-03-26T03:03:40.345342+0000","last_active":"2024-03-26T03:05:11.455290+0000","last_peered":"2024-03-26T03:05:11.455290+0000","last_clean":"2024-03-26T03:05:11.455290+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:05:11.455290+0000","last_undegraded":"2024-03-26T03:05:11.455290+0000","last_fullsized":"2024-03-26T03:05:11.455290+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:40.345078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:40.345078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":292,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965412+0000","last_change":"2024-03-26T03:03:47.301632+0000","last_active":"2024-03-26T03:05:08.965412+0000","last_peered":"2024-03-26T03:05:08.965412+0000","last_clean":"2024-03-26T03:05:08.965412+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:05:08.965412+0000","last_undegraded":"2024-03-26T03:05:08.965412+0000","last_fullsized":"2024-03-26T03:05:08.965412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":261,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966103+0000","last_change":"2024-03-26T03:01:49.896560+0000","last_active":"2024-03-26T03:05:08.966103+0000","last_peered":"2024-03-26T03:05:08.966103+0000","last_clean":"2024-03-26T03:05:08.966103+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:05:08.966103+0000","last_undegraded":"2024-03-26T03:05:08.966103+0000","last_fullsized":"2024-03-26T03:05:08.966103+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455624+0000","last_change":"2024-03-26T03:03:53.357848+0000","last_active":"2024-03-26T03:05:11.455624+0000","last_peered":"2024-03-26T03:05:11.455624+0000","last_clean":"2024-03-26T03:05:11.455624+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:05:11.455624+0000","last_undegraded":"2024-03-26T03:05:11.455624+0000","last_fullsized":"2024-03-26T03:05:11.455624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:12.812262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":241,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966114+0000","last_change":"2024-03-26T03:00:52.383684+0000","last_active":"2024-03-26T03:05:08.966114+0000","last_peered":"2024-03-26T03:05:08.966114+0000","last_clean":"2024-03-26T03:05:08.966114+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:05:08.966114+0000","last_undegraded":"2024-03-26T03:05:08.966114+0000","last_fullsized":"2024-03-26T03:05:08.966114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:52.383607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:52.383607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"71'21","reported_seq":298,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455149+0000","last_change":"2024-03-26T03:03:34.444543+0000","last_active":"2024-03-26T03:05:11.455149+0000","last_peered":"2024-03-26T03:05:11.455149+0000","last_clean":"2024-03-26T03:05:11.455149+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:05:11.455149+0000","last_undegraded":"2024-03-26T03:05:11.455149+0000","last_fullsized":"2024-03-26T03:05:11.455149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:03:34.444409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:34.444409+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846762+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465177+0000","last_change":"2024-03-26T03:01:50.265230+0000","last_active":"2024-03-26T03:05:11.465177+0000","last_peered":"2024-03-26T03:05:11.465177+0000","last_clean":"2024-03-26T03:05:11.465177+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:05:11.465177+0000","last_undegraded":"2024-03-26T03:05:11.465177+0000","last_fullsized":"2024-03-26T03:05:11.465177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:50.265114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.265114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.458528+0000","last_change":"2024-03-26T03:04:37.286514+0000","last_active":"2024-03-26T03:05:11.458528+0000","last_peered":"2024-03-26T03:05:11.458528+0000","last_clean":"2024-03-26T03:05:11.458528+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:05:11.458528+0000","last_undegraded":"2024-03-26T03:05:11.458528+0000","last_fullsized":"2024-03-26T03:05:11.458528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:37.286411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:37.286411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:41.984154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":249,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.962394+0000","last_change":"2024-03-26T03:04:05.990912+0000","last_active":"2024-03-26T03:05:08.962394+0000","last_peered":"2024-03-26T03:05:08.962394+0000","last_clean":"2024-03-26T03:05:08.962394+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:05:08.962394+0000","last_undegraded":"2024-03-26T03:05:08.962394+0000","last_fullsized":"2024-03-26T03:05:08.962394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:05.990787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.990787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":280,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455341+0000","last_change":"2024-03-26T03:04:41.075472+0000","last_active":"2024-03-26T03:05:11.455341+0000","last_peered":"2024-03-26T03:05:11.455341+0000","last_clean":"2024-03-26T03:05:11.455341+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:05:11.455341+0000","last_undegraded":"2024-03-26T03:05:11.455341+0000","last_fullsized":"2024-03-26T03:05:11.455341+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:58.734700+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.465242+0000","last_change":"2024-03-26T03:04:04.517568+0000","last_active":"2024-03-26T03:05:11.465242+0000","last_peered":"2024-03-26T03:05:11.465242+0000","last_clean":"2024-03-26T03:05:11.465242+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:05:11.465242+0000","last_undegraded":"2024-03-26T03:05:11.465242+0000","last_fullsized":"2024-03-26T03:05:11.465242+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.517463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.517463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:26.130310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.464784+0000","last_change":"2024-03-26T03:01:39.191861+0000","last_active":"2024-03-26T03:05:11.464784+0000","last_peered":"2024-03-26T03:05:11.464784+0000","last_clean":"2024-03-26T03:05:11.464784+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:05:11.464784+0000","last_undegraded":"2024-03-26T03:05:11.464784+0000","last_fullsized":"2024-03-26T03:05:11.464784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:39.191709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:01:39.191709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.605451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":250,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.458386+0000","last_change":"2024-03-26T03:04:01.275984+0000","last_active":"2024-03-26T03:05:11.458386+0000","last_peered":"2024-03-26T03:05:11.458386+0000","last_clean":"2024-03-26T03:05:11.458386+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:05:11.458386+0000","last_undegraded":"2024-03-26T03:05:11.458386+0000","last_fullsized":"2024-03-26T03:05:11.458386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.275846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.275846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:19.631705+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.454329+0000","last_change":"2024-03-26T03:04:40.028444+0000","last_active":"2024-03-26T03:05:11.454329+0000","last_peered":"2024-03-26T03:05:11.454329+0000","last_clean":"2024-03-26T03:05:11.454329+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:05:11.454329+0000","last_undegraded":"2024-03-26T03:05:11.454329+0000","last_fullsized":"2024-03-26T03:05:11.454329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:40.028340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.028340+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:42.800908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":254,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.966026+0000","last_change":"2024-03-26T03:00:41.464704+0000","last_active":"2024-03-26T03:05:08.966026+0000","last_peered":"2024-03-26T03:05:08.966026+0000","last_clean":"2024-03-26T03:05:08.966026+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:05:08.966026+0000","last_undegraded":"2024-03-26T03:05:08.966026+0000","last_fullsized":"2024-03-26T03:05:08.966026+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:41.464590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:00:41.464590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":245,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.968941+0000","last_change":"2024-03-26T03:03:51.706901+0000","last_active":"2024-03-26T03:05:08.968941+0000","last_peered":"2024-03-26T03:05:08.968941+0000","last_clean":"2024-03-26T03:05:08.968941+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:05:08.968941+0000","last_undegraded":"2024-03-26T03:05:08.968941+0000","last_fullsized":"2024-03-26T03:05:08.968941+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:51.706771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:51.706771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:21.243226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":250,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455053+0000","last_change":"2024-03-26T03:04:01.340828+0000","last_active":"2024-03-26T03:05:11.455053+0000","last_peered":"2024-03-26T03:05:11.455053+0000","last_clean":"2024-03-26T03:05:11.455053+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:05:11.455053+0000","last_undegraded":"2024-03-26T03:05:11.455053+0000","last_fullsized":"2024-03-26T03:05:11.455053+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.340706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.340706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:13.841282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":323,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965056+0000","last_change":"2024-03-26T03:00:50.380208+0000","last_active":"2024-03-26T03:05:08.965056+0000","last_peered":"2024-03-26T03:05:08.965056+0000","last_clean":"2024-03-26T03:05:08.965056+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:05:08.965056+0000","last_undegraded":"2024-03-26T03:05:08.965056+0000","last_fullsized":"2024-03-26T03:05:08.965056+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:00:50.380106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:00:50.380106+0000","objects_scrubbed":11,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":262,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.964274+0000","last_change":"2024-03-26T03:01:49.986072+0000","last_active":"2024-03-26T03:05:08.964274+0000","last_peered":"2024-03-26T03:05:08.964274+0000","last_clean":"2024-03-26T03:05:08.964274+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:05:08.964274+0000","last_undegraded":"2024-03-26T03:05:08.964274+0000","last_fullsized":"2024-03-26T03:05:08.964274+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.985144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.985144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":254,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-26T03:05:11.455019+0000","last_change":"2024-03-26T03:03:27.420047+0000","last_active":"2024-03-26T03:05:11.455019+0000","last_peered":"2024-03-26T03:05:11.455019+0000","last_clean":"2024-03-26T03:05:11.455019+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:05:11.455019+0000","last_undegraded":"2024-03-26T03:05:11.455019+0000","last_fullsized":"2024-03-26T03:05:11.455019+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:27.419925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:27.419925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:14.846730+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":241,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-03-26T03:05:08.965985+0000","last_change":"2024-03-26T03:00:51.383778+0000","last_active":"2024-03-26T03:05:08.965985+0000","last_peered":"2024-03-26T03:05:08.965985+0000","last_clean":"2024-03-26T03:05:08.965985+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:05:08.965985+0000","last_undegraded":"2024-03-26T03:05:08.965985+0000","last_fullsized":"2024-03-26T03:05:08.965985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:00:51.383695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:00:51.383695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":885915648,"num_objects":13518,"num_object_clones":0,"num_object_copies":40554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":865152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2871918592,"data_stored":2871918592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13518,"ondisk_log_size":13518,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":363,"num_read_kb":358,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":641,"ondisk_log_size":641,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103248,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592816,"kb_used_data":395048,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93152336,"statfs":{"total":95995035648,"available":95387992064,"internally_reserved":0,"allocated":404529152,"data_stored":404079825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430292,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":566184,"kb_used_data":412832,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93178968,"statfs":{"total":95995035648,"available":95415263232,"internally_reserved":0,"allocated":422739968,"data_stored":422298215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":157022662},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299956,"kb_used_data":182700,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93445196,"statfs":{"total":95995035648,"available":95687880704,"internally_reserved":0,"allocated":187084800,"data_stored":186618809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440876,"kb_used_data":216740,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93304276,"statfs":{"total":95995035648,"available":95543578624,"internally_reserved":0,"allocated":221941760,"data_stored":221490297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411430,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349076,"kb_used_data":151308,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93396076,"statfs":{"total":95995035648,"available":95637581824,"internally_reserved":0,"allocated":154939392,"data_stored":154505950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367536,"kb_used_data":187816,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93377616,"statfs":{"total":95995035648,"available":95618678784,"internally_reserved":0,"allocated":192323584,"data_stored":191860842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":184023495},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986270,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":532732,"kb_used_data":402996,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93212420,"statfs":{"total":95995035648,"available":95449518080,"internally_reserved":0,"allocated":412667904,"data_stored":412206198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691930,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740916,"kb_used_data":611176,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93004236,"statfs":{"total":95995035648,"available":95236337664,"internally_reserved":0,"allocated":625844224,"data_stored":625394901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253231104,"data_stored":253231104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":236978176,"data_stored":236978176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212598784,"data_stored":212598784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":240779264,"data_stored":240779264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":419758080,"data_stored":419758080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":664535040,"data_stored":664535040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":426311680,"data_stored":426311680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":417726464,"data_stored":417726464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:05:16.880 INFO:tasks.thrashosds.thrasher:Setting 5.f to [4, 1, 7] 2024-03-26T03:05:16.880 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.f', '4', '1', '7'] 2024-03-26T03:05:16.880 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 5.f 4 1 7 2024-03-26T03:05:16.900 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:16.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:17.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:17.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:17.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:17.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:17.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:17.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 5.0 scrub starts 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 5.0 scrub ok 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: pgmap v273: 158 pgs: 158 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.66k op/s 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 2.1 scrub starts 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 2.1 scrub ok 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:05:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 5.0 scrub starts 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 5.0 scrub ok 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: pgmap v273: 158 pgs: 158 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.66k op/s 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 2.1 scrub starts 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 2.1 scrub ok 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:05:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:17 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:05:17.368 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:17.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:17.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 11 16 27592 27576 156.657 117.625 0.00213587 0.00635852 2024-03-26T03:05:17.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:17.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:17.665 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:17.712 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:05:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:05:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:05:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 5.0 scrub starts 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 5.0 scrub ok 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: pgmap v273: 158 pgs: 158 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.66k op/s 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 2.1 scrub starts 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 2.1 scrub ok 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:05:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:17 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:05:17.814 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.859 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:17.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:17.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:17.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:17.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:18.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:18.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:18.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:18.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:18.275 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:18.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 6.16 scrub starts 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 6.16 scrub ok 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:05:18.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 6.16 scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 6.16 scrub ok 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:18.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:18 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:18.378 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:18.422 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:18.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:18.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 12 16 29554 29538 153.821 122.625 0.00196573 0.0064799 2024-03-26T03:05:18.523 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:18.571 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:18.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:18.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:18.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:18.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:05:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:05:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 6.16 scrub starts 2024-03-26T03:05:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 6.16 scrub ok 2024-03-26T03:05:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:18 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:18.807 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:18.857 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:18.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:18.959 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.004 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:19.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:19.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.145 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:19.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:19.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:19.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:19.371 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:19.402 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:19.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 13 16 31020 31004 149.036 91.625 0.00238729 0.00669066 2024-03-26T03:05:19.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:19.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: 6.1b scrub starts 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: 6.1b scrub ok 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: pgmap v274: 158 pgs: 158 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[29666]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: 6.1b scrub starts 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: 6.1b scrub ok 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: pgmap v274: 158 pgs: 158 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:05:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:19 smithi066 ceph-mon[37920]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T03:05:19.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.675 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:19.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:19.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:05:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: 6.1b scrub starts 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: 6.1b scrub ok 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: pgmap v274: 158 pgs: 158 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:05:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:19 smithi192 ceph-mon[33110]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T03:05:19.777 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.824 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:19.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:19.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:19.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:19.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:20.059 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:20.089 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:20.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:20.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:20.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:20.363 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:20.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:20.492 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:20.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 14 16 32372 32356 144.425 84.5 0.00154193 0.0068651 2024-03-26T03:05:20.594 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:05:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:05:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:05:20.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:05:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:20 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:05:20.638 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:20 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:05:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:20 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:05:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:20 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:05:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:20 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:05:20.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:20.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:20.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:20.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:20.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:20.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:21.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:21.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:21.145 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:21.180 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:21.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:21.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:21.417 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:21.459 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:21.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 15 16 33727 33711 140.442 84.6875 0.0092972 0.00711483 2024-03-26T03:05:21.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:21.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:21.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:05:21.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:05:21.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: pgmap v277: 158 pgs: 158 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 5.1 scrub starts 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[29666]: 5.1 scrub ok 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: pgmap v277: 158 pgs: 158 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-03-26T03:05:21.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:05:21.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:05:21.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:05:21.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:05:21.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 5.1 scrub starts 2024-03-26T03:05:21.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:21 smithi066 ceph-mon[37920]: 5.1 scrub ok 2024-03-26T03:05:21.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:21.727 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: pgmap v277: 158 pgs: 158 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:05:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 5.1 scrub starts 2024-03-26T03:05:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:21 smithi192 ceph-mon[33110]: 5.1 scrub ok 2024-03-26T03:05:21.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:21.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:21.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:21.955 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.002 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:22.066 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:22.103 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.152 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:22.253 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:22.281 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:22.383 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:22.430 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:22.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 16 16 34971 34955 136.524 77.75 0.00536389 0.00731946 2024-03-26T03:05:22.532 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.568 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:22.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[29666]: 6.b scrub starts 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[29666]: 6.b scrub ok 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[37920]: 6.b scrub starts 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[37920]: 6.b scrub ok 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:05:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:22 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:05:22.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:22.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:22 smithi192 ceph-mon[33110]: 6.b scrub starts 2024-03-26T03:05:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:22 smithi192 ceph-mon[33110]: 6.b scrub ok 2024-03-26T03:05:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:22 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:05:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:22 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:05:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:22 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:05:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:22 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:05:22.804 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:22.848 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:22.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:22.992 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:22.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:23.094 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:23.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:23.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:23.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:23.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:23.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:23.501 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 17 16 36574 36558 134.386 100.188 0.00735326 0.0074355 2024-03-26T03:05:23.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:23.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:05:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:05:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[29666]: pgmap v278: 158 pgs: 158 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T03:05:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:05:23.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:05:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:05:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:05:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[37920]: pgmap v278: 158 pgs: 158 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T03:05:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:05:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:05:23.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:23.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:23 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:05:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:23 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:05:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:23 smithi192 ceph-mon[33110]: pgmap v278: 158 pgs: 158 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T03:05:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:23 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:05:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:23 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:05:23.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:23.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:23.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:23.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:23.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:24.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:24.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:05:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:05:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:05:24.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:24.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:24.330 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:24.387 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:24.489 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:24.504 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 18 15 38403 38388 133.252 114.375 0.00764752 0.0074991 2024-03-26T03:05:24.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:24.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 6.1f scrub starts 2024-03-26T03:05:24.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 6.1f scrub ok 2024-03-26T03:05:24.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 2.e scrub starts 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 2.e scrub ok 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 6.1f scrub starts 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 6.1f scrub ok 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 2.e scrub starts 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 2.e scrub ok 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:05:24.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:05:24.557 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:05:24.557 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:05:24.557 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:24.643 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:24.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 6.1f scrub starts 2024-03-26T03:05:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 6.1f scrub ok 2024-03-26T03:05:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 2.e scrub starts 2024-03-26T03:05:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 2.e scrub ok 2024-03-26T03:05:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:05:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:05:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:05:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:05:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:24.791 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:24.837 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:24.940 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:24.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:24.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:25.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:25.134 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:25.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:25.237 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:25.279 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:25.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:25.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:25.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:25.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:25.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 19 16 40266 40250 132.364 116.375 0.00330352 0.00754904 2024-03-26T03:05:25.515 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:25.549 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: 2.a scrub starts 2024-03-26T03:05:25.549 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: 2.a scrub ok 2024-03-26T03:05:25.549 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: pgmap v279: 158 pgs: 158 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: 6.0 scrub starts 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: 6.0 scrub ok 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: 2.a scrub starts 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: 2.a scrub ok 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: pgmap v279: 158 pgs: 158 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: 6.0 scrub starts 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: 6.0 scrub ok 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:05:25.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:25 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:05:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:25.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:25.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:25.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:25.719 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:25.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: 2.a scrub starts 2024-03-26T03:05:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: 2.a scrub ok 2024-03-26T03:05:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: pgmap v279: 158 pgs: 158 active+clean; 2.0 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-26T03:05:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: 6.0 scrub starts 2024-03-26T03:05:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: 6.0 scrub ok 2024-03-26T03:05:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:05:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:25 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:05:25.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:25.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:25.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:25.956 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:26.002 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:26.103 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:26.153 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:26.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.291 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:26.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:26.392 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:05:26.392 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:05:26.392 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[29666]: 6.2 scrub starts 2024-03-26T03:05:26.392 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[29666]: 6.2 scrub ok 2024-03-26T03:05:26.393 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:05:26.393 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:05:26.393 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[37920]: 6.2 scrub starts 2024-03-26T03:05:26.393 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:26 smithi066 ceph-mon[37920]: 6.2 scrub ok 2024-03-26T03:05:26.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:26.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:26.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:05:26.505196+0000 min lat: 0.00133267 max lat: 0.217836 avg lat: 0.00761456 2024-03-26T03:05:26.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:05:26.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 20 16 42023 42007 131.236 109.812 0.00431642 0.00761456 2024-03-26T03:05:26.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:26.583 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:26.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:26.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:26 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:05:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:26 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:05:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:26 smithi192 ceph-mon[33110]: 6.2 scrub starts 2024-03-26T03:05:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:26 smithi192 ceph-mon[33110]: 6.2 scrub ok 2024-03-26T03:05:26.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:26.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:26.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:26.953 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.000 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:27.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:27.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:27.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:27.208 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:27.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:27.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:27.371 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:27.429 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:27.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 21 16 43701 43685 129.98 104.875 0.0219676 0.00768714 2024-03-26T03:05:27.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:27.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:27.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 4.4 scrub starts 2024-03-26T03:05:27.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 4.4 scrub ok 2024-03-26T03:05:27.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 2.1e scrub starts 2024-03-26T03:05:27.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 2.1e scrub ok 2024-03-26T03:05:27.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: pgmap v280: 158 pgs: 158 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T03:05:27.589 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 4.4 scrub starts 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 4.4 scrub ok 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 2.1e scrub starts 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 2.1e scrub ok 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: pgmap v280: 158 pgs: 158 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:05:27.590 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:27 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:05:27.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:27.717 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 4.4 scrub starts 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 4.4 scrub ok 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 2.1e scrub starts 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 2.1e scrub ok 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: pgmap v280: 158 pgs: 158 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:05:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:05:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:05:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:27 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:05:27.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:27.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:27.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:27.979 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:05:27.996 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:27.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:28.036 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:05:28.079 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:05:28.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:28.118 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:05:28.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:28.160 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:05:28.199 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:05:28.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:28.244 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:05:28.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:28.284 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:05:28.289 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:05:28.290 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:05:28.336 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:05:28.364 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:28.411 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: 3.2 scrub starts 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: 3.2 scrub ok 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3301991821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3297604783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:28.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: 3.2 scrub starts 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: 3.2 scrub ok 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3301991821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3297604783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:28.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:28 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:28.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 22 16 45279 45263 128.555 98.625 0.00469813 0.00777376 2024-03-26T03:05:28.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:28.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:28.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:28.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:28.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: 3.2 scrub starts 2024-03-26T03:05:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: 3.2 scrub ok 2024-03-26T03:05:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:05:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:05:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3301991821' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3297604783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:28 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:28.796 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:28.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:28.942 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:28.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:28.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:29.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:29.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:29.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:29.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:29.358 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:29.408 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:29.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:29.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 23 16 47041 47025 127.753 110.125 0.00562375 0.0078218 2024-03-26T03:05:29.510 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:29.565 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:29.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: pgmap v281: 158 pgs: 158 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3301991821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:05:29.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[29666]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: pgmap v281: 158 pgs: 158 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3301991821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:29.580 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T03:05:29.581 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:05:29.581 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:05:29.581 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:29 smithi066 ceph-mon[37920]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T03:05:29.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:29.724 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:29.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:29.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:05:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:05:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:05:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: pgmap v281: 158 pgs: 158 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T03:05:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:05:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:05:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3301991821' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T03:05:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:05:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:05:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:29 smithi192 ceph-mon[33110]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T03:05:29.799 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:05:29.799 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:05:29.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:29.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:29.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:29.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:30.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:30.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:30.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:30.279 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[29666]: 6.f scrub starts 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[29666]: 6.f scrub ok 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[37920]: 6.f scrub starts 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[37920]: 6.f scrub ok 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:05:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:30 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:05:30.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:30.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:30.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 24 16 48596 48580 126.48 97.1875 0.0108207 0.00790088 2024-03-26T03:05:30.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:30.559 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:30.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:30 smithi192 ceph-mon[33110]: 6.f scrub starts 2024-03-26T03:05:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:30 smithi192 ceph-mon[33110]: 6.f scrub ok 2024-03-26T03:05:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:30 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:05:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:30 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:05:30.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:30.843 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:30.945 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:30.995 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:30.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:31.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:05:31.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.147 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:31.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:31.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:31.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:31.338 INFO:teuthology.orchestra.run.smithi066.stderr:set 5.f pg_upmap mapping to [4,1,7] 2024-03-26T03:05:31.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:31.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:31.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 25 15 50128 50113 125.253 95.8125 0.00590823 0.00797909 2024-03-26T03:05:31.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:31.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[29666]: 2.7 deep-scrub starts 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[29666]: 2.7 deep-scrub ok 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[29666]: pgmap v284: 158 pgs: 158 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/740133625' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [4, 1, 7]}]: dispatch 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[37920]: 2.7 deep-scrub starts 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[37920]: 2.7 deep-scrub ok 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[37920]: pgmap v284: 158 pgs: 158 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-26T03:05:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:31 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/740133625' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [4, 1, 7]}]: dispatch 2024-03-26T03:05:31.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:31.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:31 smithi192 ceph-mon[33110]: 2.7 deep-scrub starts 2024-03-26T03:05:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:31 smithi192 ceph-mon[33110]: 2.7 deep-scrub ok 2024-03-26T03:05:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:31 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:05:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:31 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:05:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:31 smithi192 ceph-mon[33110]: pgmap v284: 158 pgs: 158 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-26T03:05:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:31 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/740133625' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [4, 1, 7]}]: dispatch 2024-03-26T03:05:31.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:31.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:31.940 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:31.984 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:31.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:32.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:32.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:32.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:32.265 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:32.367 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:32.415 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:32.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 26 16 51769 51753 124.378 102.5 0.00770683 0.00803439 2024-03-26T03:05:32.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:32.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.562 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:32.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:32.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: 5.12 scrub starts 2024-03-26T03:05:32.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: 5.12 scrub ok 2024-03-26T03:05:32.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/740133625' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [4, 1, 7]}]': finished 2024-03-26T03:05:32.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[29666]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: 5.12 scrub starts 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: 5.12 scrub ok 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/740133625' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [4, 1, 7]}]': finished 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:05:32.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:32 smithi066 ceph-mon[37920]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T03:05:32.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:32.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: 5.12 scrub starts 2024-03-26T03:05:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: 5.12 scrub ok 2024-03-26T03:05:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/740133625' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [4, 1, 7]}]': finished 2024-03-26T03:05:32.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T03:05:32.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:05:32.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:05:32.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:32 smithi192 ceph-mon[33110]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T03:05:32.785 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:32.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:32.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:32.937 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:32.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:32.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:33.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:33.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:33.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:33.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:33.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:33.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:33.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:33.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 27 16 53242 53226 123.181 92.0625 0.00620255 0.00811272 2024-03-26T03:05:33.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:33.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:33.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:33 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:05:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:33 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:05:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:33 smithi192 ceph-mon[33110]: pgmap v286: 158 pgs: 158 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-26T03:05:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:33 smithi192 ceph-mon[33110]: 4.19 scrub starts 2024-03-26T03:05:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:33 smithi192 ceph-mon[33110]: 4.19 scrub ok 2024-03-26T03:05:33.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[29666]: pgmap v286: 158 pgs: 158 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[29666]: 4.19 scrub starts 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[29666]: 4.19 scrub ok 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:05:33.785 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:05:33.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[37920]: pgmap v286: 158 pgs: 158 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-26T03:05:33.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[37920]: 4.19 scrub starts 2024-03-26T03:05:33.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-mon[37920]: 4.19 scrub ok 2024-03-26T03:05:33.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:05:33.812 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:33.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:33.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:33.961 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:33.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:34.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:34.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:34.113 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:05:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:05:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:05:34.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:34.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:34.346 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:34.390 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:34.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:34.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 28 16 54732 54716 122.107 93.125 0.00430357 0.00818447 2024-03-26T03:05:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:34.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:34.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:34.659 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:05:34.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:34.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:34.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:34.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:34.964 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:34.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:35.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:35.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:35.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:35.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:35.323 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.371 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:35.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:35.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:35.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:35.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 29 16 55912 55896 120.439 73.75 0.0630028 0.00829687 2024-03-26T03:05:35.597 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:05:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:05:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: pgmap v288: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 893 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 5.17 scrub starts 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 5.17 scrub ok 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:05:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: pgmap v288: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 893 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 5.17 scrub starts 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 5.17 scrub ok 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:05:35.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:35 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:05:35.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:05:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: pgmap v288: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 893 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 5.17 scrub starts 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 5.17 scrub ok 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:05:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:35 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:05:35.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:35.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:35.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:35.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:35.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:36.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.071 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:36.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:36.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:36.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:36.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:36.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:36.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:36.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:36.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 30 16 57068 57052 118.833 72.25 0.00419645 0.0084101 2024-03-26T03:05:36.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:36.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:36.697 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[29666]: 4.16 scrub starts 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[29666]: 4.16 scrub ok 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[37920]: 4.16 scrub starts 2024-03-26T03:05:36.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:36 smithi066 ceph-mon[37920]: 4.16 scrub ok 2024-03-26T03:05:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:36 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:05:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:36 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:05:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:36 smithi192 ceph-mon[33110]: 4.16 scrub starts 2024-03-26T03:05:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:36 smithi192 ceph-mon[33110]: 4.16 scrub ok 2024-03-26T03:05:36.743 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:36.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:36.893 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:36.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:36.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:37.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:37.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:37.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:37.222 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:05:37.222 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:05:37.222 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T03:05:37.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:05:37.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.317 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:37.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:37.418 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:37.498 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:37.505 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:37.505 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 31 16 58355 58339 117.594 80.4375 0.0227313 0.00849897 2024-03-26T03:05:37.529 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:37.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[29666]: pgmap v289: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[29666]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[37920]: pgmap v289: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[37920]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:05:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:37 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:05:37.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:37.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:37 smithi192 ceph-mon[33110]: pgmap v289: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-03-26T03:05:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:37 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:37 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:37 smithi192 ceph-mon[33110]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T03:05:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:37 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:05:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:37 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:05:37.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:37.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:37.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:37.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:37.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:38.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:38.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:38.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:38.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:38.307 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:38.409 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:38.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:38.506 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 32 16 59752 59736 116.648 87.3125 0.00622359 0.0085679 2024-03-26T03:05:38.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:05:38.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:05:38.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[29666]: 4.1a deep-scrub starts 2024-03-26T03:05:38.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[29666]: 4.1a deep-scrub ok 2024-03-26T03:05:38.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:05:38.529 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:05:38.529 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[37920]: 4.1a deep-scrub starts 2024-03-26T03:05:38.529 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:38 smithi066 ceph-mon[37920]: 4.1a deep-scrub ok 2024-03-26T03:05:38.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.612 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:38.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:38.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:38 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:05:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:38 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:05:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:38 smithi192 ceph-mon[33110]: 4.1a deep-scrub starts 2024-03-26T03:05:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:38 smithi192 ceph-mon[33110]: 4.1a deep-scrub ok 2024-03-26T03:05:38.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:38.846 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:38.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:38.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:38.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:39.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:39.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.172 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:39.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:39.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:39.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:39.405 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:39.449 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:39.506 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 33 16 61248 61232 115.946 93.5 0.008663 0.00861917 2024-03-26T03:05:39.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:39.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:05:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:05:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: pgmap v291: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 762 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:05:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: 6.13 scrub starts 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: 6.13 scrub ok 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[29666]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: pgmap v291: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 762 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: 6.13 scrub starts 2024-03-26T03:05:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: 6.13 scrub ok 2024-03-26T03:05:39.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:39.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:39.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:39 smithi066 ceph-mon[37920]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T03:05:39.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:39.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:39.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: pgmap v291: 158 pgs: 158 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 762 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: 6.13 scrub starts 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: 6.13 scrub ok 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:39 smithi192 ceph-mon[33110]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T03:05:39.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:39.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:39.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:39.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:39.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:40.092 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:40.136 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:40.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:40.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:40.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:40.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:40.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:40.506 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 34 16 62684 62668 115.175 89.75 0.00651238 0.00867645 2024-03-26T03:05:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:40.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:40.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:40.658 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:40.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:40.693 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:40.693 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:40.778 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:40.812 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:05:40.812 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:05:40.827 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:40.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:40.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:40.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:41.078 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:41.109 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:41.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.255 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:41.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:41.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:41.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:41.492 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:41.506 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 35 16 64005 63989 114.244 82.5625 0.00695612 0.00874737 2024-03-26T03:05:41.534 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:41.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:41.692 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:41 smithi192 ceph-mon[33110]: pgmap v293: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-26T03:05:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:41 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:05:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:41 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:05:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:41 smithi192 ceph-mon[33110]: 2.3 deep-scrub starts 2024-03-26T03:05:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:41 smithi192 ceph-mon[33110]: 2.3 deep-scrub ok 2024-03-26T03:05:41.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.846 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:41.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:41.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[29666]: pgmap v293: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-26T03:05:41.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[29666]: 2.3 deep-scrub starts 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[29666]: 2.3 deep-scrub ok 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[37920]: pgmap v293: 158 pgs: 158 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 106 MiB/s wr, 1.69k op/s 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[37920]: 2.3 deep-scrub starts 2024-03-26T03:05:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:41 smithi066 ceph-mon[37920]: 2.3 deep-scrub ok 2024-03-26T03:05:41.947 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:41.992 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:41.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:42.094 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:42.137 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:42.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:42.238 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:42.285 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:42.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:42.387 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:42.433 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:42.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:42.506 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 36 16 65367 65351 113.435 85.125 0.00586779 0.00881077 2024-03-26T03:05:42.535 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:42.583 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:42.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:42.633 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:05:42.681 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:05:42.681 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:05:42.681 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:05:42.682 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:05:42.682 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:05:42.682 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:05:42.682 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:05:42.682 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:42 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:05:42.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:42.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:42.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:42 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:05:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:42 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:05:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:42 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:05:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:42 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:05:42.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:42.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:42.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:42.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:43.009 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:43.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:43.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:43.225 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:05:43.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:43.273 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:05:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:43.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:43.318 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:05:43.365 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:05:43.379 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:43.406 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:05:43.406 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:43.423 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:43.460 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:05:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.483 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:43.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:43.500 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:05:43.500 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: pgmap v294: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:05:43.501 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1824651175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: pgmap v294: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:05:43.502 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1824651175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:43.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 37 16 66525 66509 112.322 72.375 0.0215114 0.00889604 2024-03-26T03:05:43.547 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:05:43.564 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:05:43.585 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:43.617 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:43.718 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: pgmap v294: 158 pgs: 158 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.42k op/s 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T03:05:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:05:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:05:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:05:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:05:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:43 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1824651175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:43.766 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:43.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:43.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:43.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:43.979 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:05:44.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:44.034 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:44.085 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:05:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:05:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:05:44.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:44.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:44.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:44.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:44.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:44.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:44.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 38 16 67792 67776 111.45 79.1875 0.00821696 0.00896715 2024-03-26T03:05:44.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:44.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:44.693 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: 6.5 scrub starts 2024-03-26T03:05:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: 6.5 scrub ok 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2899648582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: from='client.25000 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T03:05:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:44 smithi192 ceph-mon[33110]: pgmap v297: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T03:05:44.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:44.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: 6.5 scrub starts 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: 6.5 scrub ok 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2899648582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: from='client.25000 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[29666]: pgmap v297: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T03:05:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: 6.5 scrub starts 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: 6.5 scrub ok 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2899648582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: from='client.25000 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: from='client.25000 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T03:05:44.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:44 smithi066 ceph-mon[37920]: pgmap v297: 158 pgs: 158 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T03:05:44.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:44.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:44.979 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:45.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:45.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:45.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:45.285 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:45.328 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:45.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:45.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:45.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 39 16 69185 69169 110.825 87.0625 0.0083272 0.00901834 2024-03-26T03:05:45.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:45.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[29666]: 4.1e scrub starts 2024-03-26T03:05:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[29666]: 4.1e scrub ok 2024-03-26T03:05:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:05:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:05:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[37920]: 4.1e scrub starts 2024-03-26T03:05:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[37920]: 4.1e scrub ok 2024-03-26T03:05:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:05:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:45 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:05:45.696 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:45 smithi192 ceph-mon[33110]: 4.1e scrub starts 2024-03-26T03:05:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:45 smithi192 ceph-mon[33110]: 4.1e scrub ok 2024-03-26T03:05:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:45 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:05:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:45 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:05:45.784 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:45.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:45.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:45.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:46.021 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:46.065 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:46.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:46.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:46.302 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:46.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:46.443 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:46 smithi066 ceph-mon[37920]: pgmap v298: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T03:05:46.457 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:46.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:46.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:05:46.507846+0000 min lat: 0.00133267 max lat: 0.217836 avg lat: 0.00907675 2024-03-26T03:05:46.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:05:46.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 40 16 70499 70483 110.107 82.125 0.0140652 0.00907675 2024-03-26T03:05:46.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:46.651 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:46 smithi192 ceph-mon[33110]: pgmap v298: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T03:05:46.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:46.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:46.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:46 smithi066 ceph-mon[29666]: pgmap v298: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-26T03:05:46.889 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:46.938 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:46.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:47.039 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:47.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:47.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:47.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:47.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:47.364 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:47.466 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:47.507 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 41 16 71832 71816 109.453 83.3125 0.00997648 0.00913053 2024-03-26T03:05:47.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:47 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:05:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:47 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:05:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:47 smithi066 ceph-mon[29666]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T03:05:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:47 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:05:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:47 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:05:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:47 smithi066 ceph-mon[37920]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T03:05:47.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:47.674 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:47.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:05:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:47 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:05:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:47 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:05:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:47 smithi192 ceph-mon[33110]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T03:05:47.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:47.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:47.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:47.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:47.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:48.040 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:48.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:48.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:48.184 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:05:48.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:48.279 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:48.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:48.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:48.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:48.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:48.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 42 16 72971 72955 108.538 71.1875 0.00797923 0.00920853 2024-03-26T03:05:48.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:48.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:48.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:48 smithi066 ceph-mon[29666]: pgmap v300: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-26T03:05:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:48 smithi066 ceph-mon[37920]: pgmap v300: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-26T03:05:48.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:48.674 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:48.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:48.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:48.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:48.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:48.909 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:48.947 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:05:48.947 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:05:48.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:05:48.947 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:05:48.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:70260 7.b 7:ded1957d:::benchmark_data_smithi066_76917_object70259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:45.085525+0000", 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 243.75395492499999, 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13351084699999999, 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:48.948 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 70260 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085525+0000", 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085525+0000", 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085527+0000", 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9819999999999998e-06 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085580+0000", 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3714000000000001e-05 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085608+0000", 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7223e-05 2024-03-26T03:05:48.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085616+0000", 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8100000000000004e-06 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.085710+0000", 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3510000000000004e-05 2024-03-26T03:05:48.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.086079+0000", 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000369284 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.086317+0000", 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00023795199999999999 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.092021+0000", 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0057039650000000001 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.092034+0000", 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2461e-05 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.218877+0000", 2024-03-26T03:05:48.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.126843228 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.218988+0000", 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000111605 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.219036+0000", 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7113000000000001e-05 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:48.955 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:75525 7.4 e110/96 7:229e4d32:::benchmark_data_smithi066_76917_object75524:head v 110'4631, mlcod=110'4629)", 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.281743+0000", 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 238.557736573, 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.24633327299999999, 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.281743+0000", 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.281743+0000", 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.281745+0000", 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.494e-06 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.957 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.281777+0000", 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2351999999999998e-05 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.281797+0000", 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9650999999999999e-05 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.281806+0000", 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6860000000000001e-06 2024-03-26T03:05:48.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.283073+0000", 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001266543 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.283096+0000", 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3417999999999998e-05 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.528076+0000", 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.24498012899999999 2024-03-26T03:05:48.959 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76595 7.12 7:4a563122:::benchmark_data_smithi066_76917_object76594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.282157+0000", 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 235.55732293700001, 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3286985920000001, 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76595 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282157+0000", 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282157+0000", 2024-03-26T03:05:48.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282159+0000", 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0370000000000001e-06 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282182+0000", 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2872000000000001e-05 2024-03-26T03:05:48.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282192+0000", 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0387999999999999e-05 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282201+0000", 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5760000000000006e-06 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282263+0000", 2024-03-26T03:05:48.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2346000000000002e-05 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.282273+0000", 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0259e-05 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290553+0000", 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0082802589999999999 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:48.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290564+0000", 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0210999999999999e-05 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.589430+0000", 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.298866509 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.589587+0000", 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015739399999999999 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2,6", 2024-03-26T03:05:48.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.589805+0000", 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021756 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.594996+0000", 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0051905950000000001 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.595012+0000", 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6192999999999999e-05 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.603614+0000", 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0086023000000000002 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.605223+0000", 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0016084739999999999 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.609460+0000", 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004237368 2024-03-26T03:05:48.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.610855+0000", 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013952509999999999 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76631 7.12 7:4c487067:::benchmark_data_smithi066_76917_object76630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T03:05:48.968 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:54.390525+0000", 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 234.44895439199999, 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.22170068800000001, 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76631 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.969 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.390525+0000", 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.970 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.390525+0000", 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.390529+0000", 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8530000000000002e-06 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.390570+0000", 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0873e-05 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.390594+0000", 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4451e-05 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.390610+0000", 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.579e-05 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.448298+0000", 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.057688021999999999 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.448311+0000", 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2751999999999999e-05 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.593031+0000", 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14472052799999999 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.593241+0000", 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000209576 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2,6", 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.593470+0000", 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000228748 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.595648+0000", 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0021782350000000002 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.601206+0000", 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0055577170000000002 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.604061+0000", 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028554940000000001 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.610918+0000", 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00685646 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.611204+0000", 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00028628699999999998 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.612226+0000", 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0010219020000000001 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.977 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:32363 8.1 8:87fc8ad4:::benchmark_data_smithi066_100612_object32362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:20.391847+0000", 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 28.447633037999999, 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19202301199999999, 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 32363 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.978 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391847+0000", 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391847+0000", 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391848+0000", 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3179999999999999e-06 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391872+0000", 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4352e-05 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391883+0000", 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0710000000000001e-05 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391903+0000", 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0075e-05 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.393374+0000", 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014705110000000001 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.393492+0000", 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118849 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.393622+0000", 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000130094 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.394813+0000", 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001190565 2024-03-26T03:05:48.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.395867+0000", 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001053663 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.583775+0000", 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.18790796400000001 2024-03-26T03:05:48.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.583829+0000", 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4098000000000003e-05 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.583870+0000", 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0812999999999997e-05 2024-03-26T03:05:48.984 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:33124 8.1 8:8362333d:::benchmark_data_smithi066_100612_object33123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.033247+0000", 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.806232907000002, 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14529449799999999, 2024-03-26T03:05:48.985 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 33124 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033247+0000", 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033247+0000", 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033249+0000", 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9259999999999999e-06 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033271+0000", 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2568000000000001e-05 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154308+0000", 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.12103662900000001 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154328+0000", 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0445000000000002e-05 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154565+0000", 2024-03-26T03:05:48.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000236266 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154771+0000", 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020644899999999999 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154944+0000", 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017347099999999999 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.156718+0000", 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017739450000000001 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.159798+0000", 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030792340000000001 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.178423+0000", 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.018625701000000001 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.178491+0000", 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8093e-05 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.178541+0000", 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9771000000000002e-05 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34158 8.7 8:e390f89f:::benchmark_data_smithi066_100612_object34157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:48.992 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.704399+0000", 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.135080771999998, 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21413618700000001, 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34158 2024-03-26T03:05:48.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704399+0000", 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:48.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704399+0000", 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704401+0000", 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.328e-06 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704429+0000", 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.8170999999999998e-05 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704441+0000", 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1956e-05 2024-03-26T03:05:48.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704450+0000", 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6119999999999999e-06 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704606+0000", 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015650700000000001 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704731+0000", 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012487800000000001 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704864+0000", 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000132325 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706325+0000", 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001461838 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707781+0000", 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014558590000000001 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:48.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708401+0000", 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00062018800000000003 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918486+0000", 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21008421499999999 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918535+0000", 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9310000000000001e-05 2024-03-26T03:05:49.000 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34164 8.a 8:5912da51:::benchmark_data_smithi066_100612_object34163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.706561+0000", 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.132918198999999, 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21261882400000001, 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.001 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34164 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706561+0000", 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.002 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706561+0000", 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706563+0000", 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.812e-06 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706587+0000", 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3898999999999999e-05 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706598+0000", 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1399e-05 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706607+0000", 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3480000000000004e-06 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706665+0000", 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8217999999999997e-05 2024-03-26T03:05:49.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706833+0000", 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016762299999999999 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706981+0000", 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014804900000000001 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919035+0000", 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21205386500000001 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919048+0000", 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3811e-05 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919086+0000", 2024-03-26T03:05:49.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7258999999999997e-05 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919135+0000", 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9626000000000002e-05 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919180+0000", 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4914999999999999e-05 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34166 8.3 8:ca06eeeb:::benchmark_data_smithi066_100612_object34165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.707740+0000", 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.131739464999999, 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.216887831, 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34166 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.008 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707740+0000", 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707740+0000", 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707742+0000", 2024-03-26T03:05:49.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5990000000000001e-06 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707765+0000", 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3765e-05 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707775+0000", 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4800000000000007e-06 2024-03-26T03:05:49.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707783+0000", 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7430000000000002e-06 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707839+0000", 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6372999999999997e-05 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707985+0000", 2024-03-26T03:05:49.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000145494 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708097+0000", 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011292 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919114+0000", 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21101609800000001 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919150+0000", 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6705000000000002e-05 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924521+0000", 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053703479999999996 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.013 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924585+0000", 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4406000000000006e-05 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924628+0000", 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2899999999999999e-05 2024-03-26T03:05:49.014 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.24961.0:34158 8.7 e183/172 ondisk, result = 0)", 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.708205+0000", 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.131274643000001, 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.210338894, 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.015 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708205+0000", 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708205+0000", 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708206+0000", 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2299999999999999e-07 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708210+0000", 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7519999999999998e-06 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708218+0000", 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7159999999999992e-06 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.018 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708225+0000", 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3890000000000002e-06 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708389+0000", 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016390900000000001 2024-03-26T03:05:49.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708396+0000", 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5340000000000002e-06 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918544+0000", 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21014767100000001 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.020 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34168 8.a 8:59dd35f8:::benchmark_data_smithi066_100612_object34167:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.708754+0000", 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.130725472999998, 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21735549000000001, 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.021 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34168 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708754+0000", 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.022 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708754+0000", 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708755+0000", 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2249999999999999e-06 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708775+0000", 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0165000000000001e-05 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918607+0000", 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20983168899999999 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918619+0000", 2024-03-26T03:05:49.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2215000000000001e-05 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919213+0000", 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00059358900000000005 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919358+0000", 2024-03-26T03:05:49.025 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014458599999999999 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919512+0000", 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000154382 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.026 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.923710+0000", 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041979529999999999 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.923752+0000", 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2240000000000002e-05 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.925962+0000", 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022098700000000001 2024-03-26T03:05:49.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.926065+0000", 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010254700000000001 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.926110+0000", 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5028999999999999e-05 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.028 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.24961.0:34164 8.a e183/172 ondisk, result = 0)", 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.708786+0000", 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.130693884999999, 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.210263589, 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.029 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708786+0000", 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.030 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708786+0000", 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708787+0000", 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.297e-06 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708792+0000", 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5510000000000001e-06 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918393+0000", 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.209601015 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918409+0000", 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6861e-05 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919020+0000", 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00061044400000000003 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919028+0000", 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2320000000000001e-06 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919049+0000", 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1189e-05 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.034 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.24961.0:34169 8.b e183/172 8:dfcca531:::benchmark_data_smithi066_100612_object34168:head v 183'2156, mlcod=183'2154)", 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.709192+0000", 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 27.130287288000002, 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.215641534, 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.035 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709192+0000", 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709192+0000", 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.036 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709194+0000", 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2270000000000001e-06 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709217+0000", 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.387e-05 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918416+0000", 2024-03-26T03:05:49.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20919812300000001 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918427+0000", 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1219999999999999e-05 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918621+0000", 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00019475800000000001 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918640+0000", 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8970999999999998e-05 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924640+0000", 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0059992149999999996 2024-03-26T03:05:49.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924834+0000", 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00019415 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34572 8.7 8:e0d7c130:::benchmark_data_smithi066_100612_object34571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.123979+0000", 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.715500318, 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.204696821, 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.040 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34572 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.123979+0000", 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.123979+0000", 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.123981+0000", 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5570000000000001e-06 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.042 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.124014+0000", 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2842999999999997e-05 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.124026+0000", 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2119999999999999e-05 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.124035+0000", 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9889999999999998e-06 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.125033+0000", 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00099777900000000007 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.125154+0000", 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012128100000000001 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.125285+0000", 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000131321 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.044 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127808+0000", 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0025223950000000002 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.130018+0000", 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022102939999999998 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328567+0000", 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19854961700000001 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328624+0000", 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6298000000000002e-05 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328676+0000", 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2327000000000002e-05 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34578 8.7 8:e32f7176:::benchmark_data_smithi066_100612_object34577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.127436+0000", 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.712043146999999, 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20137244400000001, 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.047 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34578 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127436+0000", 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127436+0000", 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127437+0000", 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3600000000000002e-07 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127461+0000", 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.391e-05 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127470+0000", 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8929999999999994e-06 2024-03-26T03:05:49.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127477+0000", 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1369999999999996e-06 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.128109+0000", 2024-03-26T03:05:49.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00063122000000000004 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.128234+0000", 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000125704 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.128368+0000", 2024-03-26T03:05:49.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000133843 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.130056+0000", 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001687926 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131128+0000", 2024-03-26T03:05:49.052 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001072143 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328724+0000", 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19759611599999999 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328771+0000", 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6535000000000001e-05 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328809+0000", 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8080999999999997e-05 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34584 8.7 8:e3ec79c0:::benchmark_data_smithi066_100612_object34583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.131279+0000", 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.70820075, 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20215682099999999, 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.054 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34584 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131279+0000", 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131279+0000", 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131281+0000", 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8390000000000001e-06 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131303+0000", 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.226e-05 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131314+0000", 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0624e-05 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131322+0000", 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0350000000000001e-06 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131403+0000", 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1310000000000006e-05 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131521+0000", 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118009 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131654+0000", 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013324300000000001 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330270+0000", 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.198616031 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330309+0000", 2024-03-26T03:05:49.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8905000000000001e-05 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333277+0000", 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0029678460000000001 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333374+0000", 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7459000000000003e-05 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333436+0000", 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1260000000000006e-05 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34585 8.3 8:c22d9924:::benchmark_data_smithi066_100612_object34584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.131968+0000", 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.707511778000001, 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20611238700000001, 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.061 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34585 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131968+0000", 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131968+0000", 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131969+0000", 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.282e-06 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132008+0000", 2024-03-26T03:05:49.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9110999999999999e-05 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132018+0000", 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6919999999999997e-06 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132026+0000", 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8660000000000006e-06 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132126+0000", 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100704 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330081+0000", 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.197954138 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330927+0000", 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00084675600000000005 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.335100+0000", 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041724580000000004 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.335144+0000", 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4280999999999998e-05 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.337992+0000", 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002848176 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.338040+0000", 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7685000000000002e-05 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.338080+0000", 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0238000000000003e-05 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34588 8.1 8:8ba52af6:::benchmark_data_smithi066_100612_object34587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.067 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.132044+0000", 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.707435522000001, 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20180536399999999, 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34588 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.068 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132044+0000", 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132044+0000", 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132045+0000", 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0400000000000005e-07 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132068+0000", 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3337e-05 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132076+0000", 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5129999999999999e-06 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132082+0000", 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5350000000000001e-06 2024-03-26T03:05:49.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132187+0000", 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000104338 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328422+0000", 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19623518100000001 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328587+0000", 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016529199999999999 2024-03-26T03:05:49.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.332138+0000", 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003551397 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.332175+0000", 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6486999999999998e-05 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333748+0000", 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001573292 2024-03-26T03:05:49.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333807+0000", 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8996000000000002e-05 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333849+0000", 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2191999999999999e-05 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.24961.0:34572 8.7 e183/172 ondisk, result = 0)", 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.132197+0000", 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.707283039, 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19648653499999999, 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132197+0000", 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132197+0000", 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132198+0000", 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3909999999999999e-06 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132203+0000", 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8930000000000004e-06 2024-03-26T03:05:49.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328492+0000", 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19628907800000001 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328502+0000", 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0404999999999999e-05 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328550+0000", 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8109999999999998e-05 2024-03-26T03:05:49.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328561+0000", 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0276e-05 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328683+0000", 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012238200000000001 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34594 8.7 8:e90646ed:::benchmark_data_smithi066_100612_object34593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.132813+0000", 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 26.706666378000001, 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20418934899999999, 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34594 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.078 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132813+0000", 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132813+0000", 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132815+0000", 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3069999999999999e-06 2024-03-26T03:05:49.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132837+0000", 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2121e-05 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328282+0000", 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19544577399999999 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328295+0000", 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2354999999999999e-05 2024-03-26T03:05:49.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330364+0000", 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020697240000000002 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330510+0000", 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014537299999999999 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.331832+0000", 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013220879999999999 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336747+0000", 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049151799999999999 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336817+0000", 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9748999999999999e-05 2024-03-26T03:05:49.082 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336910+0000", 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3179999999999999e-05 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336961+0000", 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1149000000000001e-05 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:05:49.083 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.337003+0000", 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1349e-05 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:05:49.084 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:05:49.085 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:48 smithi192 ceph-mon[33110]: pgmap v300: 158 pgs: 158 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-26T03:05:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.086 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:49.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:49.159 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:05:49.169 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:05:49.187 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.231 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:49.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:49.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:49.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:49.464 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:49.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 43 16 74093 74077 107.644 70.125 0.0169398 0.0092844 2024-03-26T03:05:49.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:49.614 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:49.662 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:49.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:05:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:05:49.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:49.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:49.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:49.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:49.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:50.037 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:50.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:50.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:49 smithi066 ceph-mon[29666]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T03:05:50.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:49 smithi066 ceph-mon[29666]: from='client.25006 v1:172.21.15.66:0/1399467552' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:05:50.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:49 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:49 smithi066 ceph-mon[37920]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T03:05:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:49 smithi066 ceph-mon[37920]: from='client.25006 v1:172.21.15.66:0/1399467552' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:05:50.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:49 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:50.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:50.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:49 smithi192 ceph-mon[33110]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T03:05:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:49 smithi192 ceph-mon[33110]: from='client.25006 v1:172.21.15.66:0/1399467552' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:05:50.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:49 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:05:50.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:50.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:50.460 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:50.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:05:50.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 44 16 75405 75389 107.061 82 0.0121948 0.00933497 2024-03-26T03:05:50.523 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:50.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.668 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:50.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:50.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.824 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:50.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:50.925 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:50.969 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:50.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:51.042 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-26T03:05:48.975420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4434172328,"num_objects":67881,"num_object_clones":0,"num_object_copies":203643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67881,"num_whiteouts":0,"num_read":1407,"num_read_kb":1127,"num_write":68279,"num_write_kb":4329709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69789,"ondisk_log_size":69789,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14426512,"kb_used_data":12849844,"kb_used_omap":12,"kb_used_meta":1576563,"kb_avail":735534704,"statfs":{"total":767960285184,"available":753187536896,"internally_reserved":0,"allocated":13158240256,"data_stored":13154190580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1614401072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":429391872,"num_objects":6552,"num_object_clones":0,"num_object_copies":19656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6552,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6552,"num_write_kb":419328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.507081"},"pg_stats":[{"pgid":"8.4","version":"190'4223","reported_seq":4271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971046+0000","last_change":"2024-03-26T03:04:52.197394+0000","last_active":"2024-03-26T03:05:43.971046+0000","last_peered":"2024-03-26T03:05:43.971046+0000","last_clean":"2024-03-26T03:05:43.971046+0000","last_became_active":"2024-03-26T03:04:52.197080+0000","last_became_peered":"2024-03-26T03:04:52.197080+0000","last_unstale":"2024-03-26T03:05:43.971046+0000","last_undegraded":"2024-03-26T03:05:43.971046+0000","last_fullsized":"2024-03-26T03:05:43.971046+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:04.134916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276758528,"num_objects":4223,"num_object_clones":0,"num_object_copies":12669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":270272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":375,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993393+0000","last_change":"2024-03-26T03:04:48.638112+0000","last_active":"2024-03-26T03:05:43.993393+0000","last_peered":"2024-03-26T03:05:43.993393+0000","last_clean":"2024-03-26T03:05:43.993393+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:05:43.993393+0000","last_undegraded":"2024-03-26T03:05:43.993393+0000","last_fullsized":"2024-03-26T03:05:43.993393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:04:48.638002+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:09.353317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":336,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975905+0000","last_change":"2024-03-26T03:04:38.309217+0000","last_active":"2024-03-26T03:05:43.975905+0000","last_peered":"2024-03-26T03:05:43.975905+0000","last_clean":"2024-03-26T03:05:43.975905+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:05:43.975905+0000","last_undegraded":"2024-03-26T03:05:43.975905+0000","last_fullsized":"2024-03-26T03:05:43.975905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:04:38.309114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:38.309114+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:01.503706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.987450+0000","last_change":"2024-03-26T03:05:23.378940+0000","last_active":"2024-03-26T03:05:43.987450+0000","last_peered":"2024-03-26T03:05:43.987450+0000","last_clean":"2024-03-26T03:05:43.987450+0000","last_became_active":"2024-03-26T02:58:56.927896+0000","last_became_peered":"2024-03-26T02:58:56.927896+0000","last_unstale":"2024-03-26T03:05:43.987450+0000","last_undegraded":"2024-03-26T03:05:43.987450+0000","last_fullsized":"2024-03-26T03:05:43.987450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:41.440579+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":84,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981369+0000","last_change":"2024-03-26T03:04:37.863032+0000","last_active":"2024-03-26T03:05:43.981369+0000","last_peered":"2024-03-26T03:05:43.981369+0000","last_clean":"2024-03-26T03:05:43.981369+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:05:43.981369+0000","last_undegraded":"2024-03-26T03:05:43.981369+0000","last_fullsized":"2024-03-26T03:05:43.981369+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:37.862940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:37.862940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.217505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"191'4265","reported_seq":4313,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.224751+0000","last_change":"2024-03-26T03:04:52.164503+0000","last_active":"2024-03-26T03:05:44.224751+0000","last_peered":"2024-03-26T03:05:44.224751+0000","last_clean":"2024-03-26T03:05:44.224751+0000","last_became_active":"2024-03-26T03:04:52.164179+0000","last_became_peered":"2024-03-26T03:04:52.164179+0000","last_unstale":"2024-03-26T03:05:44.224751+0000","last_undegraded":"2024-03-26T03:05:44.224751+0000","last_fullsized":"2024-03-26T03:05:44.224751+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:08.819705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279511040,"num_objects":4265,"num_object_clones":0,"num_object_copies":12795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4265,"num_write_kb":272960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":321,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.989348+0000","last_change":"2024-03-26T03:05:34.596907+0000","last_active":"2024-03-26T03:05:43.989348+0000","last_peered":"2024-03-26T03:05:43.989348+0000","last_clean":"2024-03-26T03:05:43.989348+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:05:43.989348+0000","last_undegraded":"2024-03-26T03:05:43.989348+0000","last_fullsized":"2024-03-26T03:05:43.989348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:05:34.596707+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.596707+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:03.068385+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978178+0000","last_change":"2024-03-26T03:03:59.711554+0000","last_active":"2024-03-26T03:05:43.978178+0000","last_peered":"2024-03-26T03:05:43.978178+0000","last_clean":"2024-03-26T03:05:43.978178+0000","last_became_active":"2024-03-26T02:58:56.930871+0000","last_became_peered":"2024-03-26T02:58:56.930871+0000","last_unstale":"2024-03-26T03:05:43.978178+0000","last_undegraded":"2024-03-26T03:05:43.978178+0000","last_fullsized":"2024-03-26T03:05:43.978178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:03:59.710985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":276,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.986828+0000","last_change":"2024-03-26T03:04:36.837815+0000","last_active":"2024-03-26T03:05:43.986828+0000","last_peered":"2024-03-26T03:05:43.986828+0000","last_clean":"2024-03-26T03:05:43.986828+0000","last_became_active":"2024-03-26T02:59:00.941194+0000","last_became_peered":"2024-03-26T02:59:00.941194+0000","last_unstale":"2024-03-26T03:05:43.986828+0000","last_undegraded":"2024-03-26T03:05:43.986828+0000","last_fullsized":"2024-03-26T03:05:43.986828+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:02.467182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988962+0000","last_change":"2024-03-26T03:04:47.630302+0000","last_active":"2024-03-26T03:05:43.988962+0000","last_peered":"2024-03-26T03:05:43.988962+0000","last_clean":"2024-03-26T03:05:43.988962+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:05:43.988962+0000","last_undegraded":"2024-03-26T03:05:43.988962+0000","last_fullsized":"2024-03-26T03:05:43.988962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:47.630187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:04:47.630187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:59.447692+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"191'4273","reported_seq":4320,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:45.120477+0000","last_change":"2024-03-26T03:04:52.165009+0000","last_active":"2024-03-26T03:05:45.120477+0000","last_peered":"2024-03-26T03:05:45.120477+0000","last_clean":"2024-03-26T03:05:45.120477+0000","last_became_active":"2024-03-26T03:04:52.164468+0000","last_became_peered":"2024-03-26T03:04:52.164468+0000","last_unstale":"2024-03-26T03:05:45.120477+0000","last_undegraded":"2024-03-26T03:05:45.120477+0000","last_fullsized":"2024-03-26T03:05:45.120477+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:04.943306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":12819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":273472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":359,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981783+0000","last_change":"2024-03-26T03:05:36.627545+0000","last_active":"2024-03-26T03:05:43.981783+0000","last_peered":"2024-03-26T03:05:43.981783+0000","last_clean":"2024-03-26T03:05:43.981783+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:05:43.981783+0000","last_undegraded":"2024-03-26T03:05:43.981783+0000","last_fullsized":"2024-03-26T03:05:43.981783+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:05:36.627397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:36.627397+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:47.590465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.980643+0000","last_change":"2024-03-26T03:04:47.600713+0000","last_active":"2024-03-26T03:05:43.980643+0000","last_peered":"2024-03-26T03:05:43.980643+0000","last_clean":"2024-03-26T03:05:43.980643+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:05:43.980643+0000","last_undegraded":"2024-03-26T03:05:43.980643+0000","last_fullsized":"2024-03-26T03:05:43.980643+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:04:47.600600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.828347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.980022+0000","last_change":"2024-03-26T03:04:45.653970+0000","last_active":"2024-03-26T03:05:43.980022+0000","last_peered":"2024-03-26T03:05:43.980022+0000","last_clean":"2024-03-26T03:05:43.980022+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:05:43.980022+0000","last_undegraded":"2024-03-26T03:05:43.980022+0000","last_fullsized":"2024-03-26T03:05:43.980022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:45.653866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:04:45.653866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:55.060254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.992873+0000","last_change":"2024-03-26T03:04:41.640595+0000","last_active":"2024-03-26T03:05:43.992873+0000","last_peered":"2024-03-26T03:05:43.992873+0000","last_clean":"2024-03-26T03:05:43.992873+0000","last_became_active":"2024-03-26T02:59:02.945424+0000","last_became_peered":"2024-03-26T02:59:02.945424+0000","last_unstale":"2024-03-26T03:05:43.992873+0000","last_undegraded":"2024-03-26T03:05:43.992873+0000","last_fullsized":"2024-03-26T03:05:43.992873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:47.003467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"191'4150","reported_seq":4196,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.229734+0000","last_change":"2024-03-26T03:04:52.197465+0000","last_active":"2024-03-26T03:05:44.229734+0000","last_peered":"2024-03-26T03:05:44.229734+0000","last_clean":"2024-03-26T03:05:44.229734+0000","last_became_active":"2024-03-26T03:04:52.196952+0000","last_became_peered":"2024-03-26T03:04:52.196952+0000","last_unstale":"2024-03-26T03:05:44.229734+0000","last_undegraded":"2024-03-26T03:05:44.229734+0000","last_fullsized":"2024-03-26T03:05:44.229734+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.864582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271974400,"num_objects":4150,"num_object_clones":0,"num_object_copies":12450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4150,"num_write_kb":265600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"153'25","reported_seq":323,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971948+0000","last_change":"2024-03-26T03:05:42.663585+0000","last_active":"2024-03-26T03:05:43.971948+0000","last_peered":"2024-03-26T03:05:43.971948+0000","last_clean":"2024-03-26T03:05:43.971948+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:05:43.971948+0000","last_undegraded":"2024-03-26T03:05:43.971948+0000","last_fullsized":"2024-03-26T03:05:43.971948+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'25","last_scrub_stamp":"2024-03-26T03:05:42.663443+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:05:42.663443+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:55.207882+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993816+0000","last_change":"2024-03-26T03:05:14.733614+0000","last_active":"2024-03-26T03:05:43.993816+0000","last_peered":"2024-03-26T03:05:43.993816+0000","last_clean":"2024-03-26T03:05:43.993816+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:05:43.993816+0000","last_undegraded":"2024-03-26T03:05:43.993816+0000","last_fullsized":"2024-03-26T03:05:43.993816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:14.733500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:14.733500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:14.977872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":275,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977389+0000","last_change":"2024-03-26T03:05:15.629029+0000","last_active":"2024-03-26T03:05:43.977389+0000","last_peered":"2024-03-26T03:05:43.977389+0000","last_clean":"2024-03-26T03:05:43.977389+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:05:43.977389+0000","last_undegraded":"2024-03-26T03:05:43.977389+0000","last_fullsized":"2024-03-26T03:05:43.977389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.628906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.628906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:20.091078+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.980072+0000","last_change":"2024-03-26T03:05:29.490680+0000","last_active":"2024-03-26T03:05:43.980072+0000","last_peered":"2024-03-26T03:05:43.980072+0000","last_clean":"2024-03-26T03:05:43.980072+0000","last_became_active":"2024-03-26T02:59:02.942465+0000","last_became_peered":"2024-03-26T02:59:02.942465+0000","last_unstale":"2024-03-26T03:05:43.980072+0000","last_undegraded":"2024-03-26T03:05:43.980072+0000","last_fullsized":"2024-03-26T03:05:43.980072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:49.390025+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":356,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.979502+0000","last_change":"2024-03-26T03:04:43.715167+0000","last_active":"2024-03-26T03:05:43.979502+0000","last_peered":"2024-03-26T03:05:43.979502+0000","last_clean":"2024-03-26T03:05:43.979502+0000","last_became_active":"2024-03-26T03:04:41.967552+0000","last_became_peered":"2024-03-26T03:04:41.967552+0000","last_unstale":"2024-03-26T03:05:43.979502+0000","last_undegraded":"2024-03-26T03:05:43.979502+0000","last_fullsized":"2024-03-26T03:05:43.979502+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:49.647595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"191'4138","reported_seq":4186,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.792289+0000","last_change":"2024-03-26T03:04:52.165355+0000","last_active":"2024-03-26T03:05:44.792289+0000","last_peered":"2024-03-26T03:05:44.792289+0000","last_clean":"2024-03-26T03:05:44.792289+0000","last_became_active":"2024-03-26T03:04:52.164397+0000","last_became_peered":"2024-03-26T03:04:52.164397+0000","last_unstale":"2024-03-26T03:05:44.792289+0000","last_undegraded":"2024-03-26T03:05:44.792289+0000","last_fullsized":"2024-03-26T03:05:44.792289+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:10.262037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271187968,"num_objects":4138,"num_object_clones":0,"num_object_copies":12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":264832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":547,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.992937+0000","last_change":"2024-03-26T03:04:42.630937+0000","last_active":"2024-03-26T03:05:43.992937+0000","last_peered":"2024-03-26T03:05:43.992937+0000","last_clean":"2024-03-26T03:05:43.992937+0000","last_became_active":"2024-03-26T03:04:41.962917+0000","last_became_peered":"2024-03-26T03:04:41.962917+0000","last_unstale":"2024-03-26T03:05:43.992937+0000","last_undegraded":"2024-03-26T03:05:43.992937+0000","last_fullsized":"2024-03-26T03:05:43.992937+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":294,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971228+0000","last_change":"2024-03-26T03:03:48.428215+0000","last_active":"2024-03-26T03:05:43.971228+0000","last_peered":"2024-03-26T03:05:43.971228+0000","last_clean":"2024-03-26T03:05:43.971228+0000","last_became_active":"2024-03-26T02:58:58.930735+0000","last_became_peered":"2024-03-26T02:58:58.930735+0000","last_unstale":"2024-03-26T03:05:43.971228+0000","last_undegraded":"2024-03-26T03:05:43.971228+0000","last_fullsized":"2024-03-26T03:05:43.971228+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:03:48.427958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:48.427958+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.984147+0000","last_change":"2024-03-26T03:05:23.659400+0000","last_active":"2024-03-26T03:05:43.984147+0000","last_peered":"2024-03-26T03:05:43.984147+0000","last_clean":"2024-03-26T03:05:43.984147+0000","last_became_active":"2024-03-26T02:58:56.930356+0000","last_became_peered":"2024-03-26T02:58:56.930356+0000","last_unstale":"2024-03-26T03:05:43.984147+0000","last_undegraded":"2024-03-26T03:05:43.984147+0000","last_fullsized":"2024-03-26T03:05:43.984147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:40.348436+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":374,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.986228+0000","last_change":"2024-03-26T03:05:26.411884+0000","last_active":"2024-03-26T03:05:43.986228+0000","last_peered":"2024-03-26T03:05:43.986228+0000","last_clean":"2024-03-26T03:05:43.986228+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:05:43.986228+0000","last_undegraded":"2024-03-26T03:05:43.986228+0000","last_fullsized":"2024-03-26T03:05:43.986228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:05:26.411739+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.411739+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:28.833076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":270,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971561+0000","last_change":"2024-03-26T03:05:41.654107+0000","last_active":"2024-03-26T03:05:43.971561+0000","last_peered":"2024-03-26T03:05:43.971561+0000","last_clean":"2024-03-26T03:05:43.971561+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:05:43.971561+0000","last_undegraded":"2024-03-26T03:05:43.971561+0000","last_fullsized":"2024-03-26T03:05:43.971561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:41.654012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:41.654012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:51.175775+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"191'4195","reported_seq":4243,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.229281+0000","last_change":"2024-03-26T03:04:52.162802+0000","last_active":"2024-03-26T03:05:44.229281+0000","last_peered":"2024-03-26T03:05:44.229281+0000","last_clean":"2024-03-26T03:05:44.229281+0000","last_became_active":"2024-03-26T03:04:52.162480+0000","last_became_peered":"2024-03-26T03:04:52.162480+0000","last_unstale":"2024-03-26T03:05:44.229281+0000","last_undegraded":"2024-03-26T03:05:44.229281+0000","last_fullsized":"2024-03-26T03:05:44.229281+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.505434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274923520,"num_objects":4195,"num_object_clones":0,"num_object_copies":12585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":268480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":357,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.986719+0000","last_change":"2024-03-26T03:04:42.754060+0000","last_active":"2024-03-26T03:05:43.986719+0000","last_peered":"2024-03-26T03:05:43.986719+0000","last_clean":"2024-03-26T03:05:43.986719+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:05:43.986719+0000","last_undegraded":"2024-03-26T03:05:43.986719+0000","last_fullsized":"2024-03-26T03:05:43.986719+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.753882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.753882+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:58.379811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":305,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978580+0000","last_change":"2024-03-26T03:04:05.712151+0000","last_active":"2024-03-26T03:05:43.978580+0000","last_peered":"2024-03-26T03:05:43.978580+0000","last_clean":"2024-03-26T03:05:43.978580+0000","last_became_active":"2024-03-26T02:58:58.931406+0000","last_became_peered":"2024-03-26T02:58:58.931406+0000","last_unstale":"2024-03-26T03:05:43.978580+0000","last_undegraded":"2024-03-26T03:05:43.978580+0000","last_fullsized":"2024-03-26T03:05:43.978580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:05.711979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:05.711979+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977632+0000","last_change":"2024-03-26T03:05:12.539163+0000","last_active":"2024-03-26T03:05:43.977632+0000","last_peered":"2024-03-26T03:05:43.977632+0000","last_clean":"2024-03-26T03:05:43.977632+0000","last_became_active":"2024-03-26T02:58:56.930282+0000","last_became_peered":"2024-03-26T02:58:56.930282+0000","last_unstale":"2024-03-26T03:05:43.977632+0000","last_undegraded":"2024-03-26T03:05:43.977632+0000","last_fullsized":"2024-03-26T03:05:43.977632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:23.892924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":364,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971415+0000","last_change":"2024-03-26T03:03:56.360899+0000","last_active":"2024-03-26T03:05:43.971415+0000","last_peered":"2024-03-26T03:05:43.971415+0000","last_clean":"2024-03-26T03:05:43.971415+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:05:43.971415+0000","last_undegraded":"2024-03-26T03:05:43.971415+0000","last_fullsized":"2024-03-26T03:05:43.971415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:03:56.360747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:56.360747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":76,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977615+0000","last_change":"2024-03-26T03:04:14.431790+0000","last_active":"2024-03-26T03:05:43.977615+0000","last_peered":"2024-03-26T03:05:43.977615+0000","last_clean":"2024-03-26T03:05:43.977615+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:05:43.977615+0000","last_undegraded":"2024-03-26T03:05:43.977615+0000","last_fullsized":"2024-03-26T03:05:43.977615+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.305791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.305791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"190'4064","reported_seq":4112,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970389+0000","last_change":"2024-03-26T03:04:52.165534+0000","last_active":"2024-03-26T03:05:43.970389+0000","last_peered":"2024-03-26T03:05:43.970389+0000","last_clean":"2024-03-26T03:05:43.970389+0000","last_became_active":"2024-03-26T03:04:52.165202+0000","last_became_peered":"2024-03-26T03:04:52.165202+0000","last_unstale":"2024-03-26T03:05:43.970389+0000","last_undegraded":"2024-03-26T03:05:43.970389+0000","last_fullsized":"2024-03-26T03:05:43.970389+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.699745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":260096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":317,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971149+0000","last_change":"2024-03-26T03:05:34.681062+0000","last_active":"2024-03-26T03:05:43.971149+0000","last_peered":"2024-03-26T03:05:43.971149+0000","last_clean":"2024-03-26T03:05:43.971149+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:05:43.971149+0000","last_undegraded":"2024-03-26T03:05:43.971149+0000","last_fullsized":"2024-03-26T03:05:43.971149+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:05:34.680763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.680763+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:45.306434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993139+0000","last_change":"2024-03-26T03:05:17.724594+0000","last_active":"2024-03-26T03:05:43.993139+0000","last_peered":"2024-03-26T03:05:43.993139+0000","last_clean":"2024-03-26T03:05:43.993139+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:05:43.993139+0000","last_undegraded":"2024-03-26T03:05:43.993139+0000","last_fullsized":"2024-03-26T03:05:43.993139+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.724462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.724462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:24.206611+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":82,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971617+0000","last_change":"2024-03-26T03:05:32.342821+0000","last_active":"2024-03-26T03:05:43.971617+0000","last_peered":"2024-03-26T03:05:43.971617+0000","last_clean":"2024-03-26T03:05:43.971617+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:05:43.971617+0000","last_undegraded":"2024-03-26T03:05:43.971617+0000","last_fullsized":"2024-03-26T03:05:43.971617+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:00.224975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:00.224975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977225+0000","last_change":"2024-03-26T03:04:35.720859+0000","last_active":"2024-03-26T03:05:43.977225+0000","last_peered":"2024-03-26T03:05:43.977225+0000","last_clean":"2024-03-26T03:05:43.977225+0000","last_became_active":"2024-03-26T02:59:03.206399+0000","last_became_peered":"2024-03-26T02:59:03.206399+0000","last_unstale":"2024-03-26T03:05:43.977225+0000","last_undegraded":"2024-03-26T03:05:43.977225+0000","last_fullsized":"2024-03-26T03:05:43.977225+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:02.340517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"190'4160","reported_seq":4208,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994127+0000","last_change":"2024-03-26T03:04:52.197065+0000","last_active":"2024-03-26T03:05:43.994127+0000","last_peered":"2024-03-26T03:05:43.994127+0000","last_clean":"2024-03-26T03:05:43.994127+0000","last_became_active":"2024-03-26T03:04:52.196605+0000","last_became_peered":"2024-03-26T03:04:52.196605+0000","last_unstale":"2024-03-26T03:05:43.994127+0000","last_undegraded":"2024-03-26T03:05:43.994127+0000","last_fullsized":"2024-03-26T03:05:43.994127+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:13.687550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":4160,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4160,"num_write_kb":266240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"91'9","reported_seq":367,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.974578+0000","last_change":"2024-03-26T03:05:15.906304+0000","last_active":"2024-03-26T03:05:43.974578+0000","last_peered":"2024-03-26T03:05:43.974578+0000","last_clean":"2024-03-26T03:05:43.974578+0000","last_became_active":"2024-03-26T02:59:00.937005+0000","last_became_peered":"2024-03-26T02:59:00.937005+0000","last_unstale":"2024-03-26T03:05:43.974578+0000","last_undegraded":"2024-03-26T03:05:43.974578+0000","last_fullsized":"2024-03-26T03:05:43.974578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:33.752036+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":320,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970436+0000","last_change":"2024-03-26T03:05:29.615450+0000","last_active":"2024-03-26T03:05:43.970436+0000","last_peered":"2024-03-26T03:05:43.970436+0000","last_clean":"2024-03-26T03:05:43.970436+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:05:43.970436+0000","last_undegraded":"2024-03-26T03:05:43.970436+0000","last_fullsized":"2024-03-26T03:05:43.970436+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:05:29.615310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.615310+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:30.344464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.982814+0000","last_change":"2024-03-26T03:05:29.683402+0000","last_active":"2024-03-26T03:05:43.982814+0000","last_peered":"2024-03-26T03:05:43.982814+0000","last_clean":"2024-03-26T03:05:43.982814+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:05:43.982814+0000","last_undegraded":"2024-03-26T03:05:43.982814+0000","last_fullsized":"2024-03-26T03:05:43.982814+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.683310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:32.535718+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994013+0000","last_change":"2024-03-26T03:04:34.494561+0000","last_active":"2024-03-26T03:05:43.994013+0000","last_peered":"2024-03-26T03:05:43.994013+0000","last_clean":"2024-03-26T03:05:43.994013+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:05:43.994013+0000","last_undegraded":"2024-03-26T03:05:43.994013+0000","last_fullsized":"2024-03-26T03:05:43.994013+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.494481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.494481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":357,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978002+0000","last_change":"2024-03-26T03:04:42.754550+0000","last_active":"2024-03-26T03:05:43.978002+0000","last_peered":"2024-03-26T03:05:43.978002+0000","last_clean":"2024-03-26T03:05:43.978002+0000","last_became_active":"2024-03-26T03:04:41.967378+0000","last_became_peered":"2024-03-26T03:04:41.967378+0000","last_unstale":"2024-03-26T03:05:43.978002+0000","last_undegraded":"2024-03-26T03:05:43.978002+0000","last_fullsized":"2024-03-26T03:05:43.978002+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.754402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.754402+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:02.232433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"191'4275","reported_seq":4323,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:45.109112+0000","last_change":"2024-03-26T03:04:52.164796+0000","last_active":"2024-03-26T03:05:45.109112+0000","last_peered":"2024-03-26T03:05:45.109112+0000","last_clean":"2024-03-26T03:05:45.109112+0000","last_became_active":"2024-03-26T03:04:52.164450+0000","last_became_peered":"2024-03-26T03:04:52.164450+0000","last_unstale":"2024-03-26T03:05:45.109112+0000","last_undegraded":"2024-03-26T03:05:45.109112+0000","last_fullsized":"2024-03-26T03:05:45.109112+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:18.393856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280166400,"num_objects":4275,"num_object_clones":0,"num_object_copies":12825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":273600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":275,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970701+0000","last_change":"2024-03-26T03:05:20.890094+0000","last_active":"2024-03-26T03:05:43.970701+0000","last_peered":"2024-03-26T03:05:43.970701+0000","last_clean":"2024-03-26T03:05:43.970701+0000","last_became_active":"2024-03-26T02:59:00.942395+0000","last_became_peered":"2024-03-26T02:59:00.942395+0000","last_unstale":"2024-03-26T03:05:43.970701+0000","last_undegraded":"2024-03-26T03:05:43.970701+0000","last_fullsized":"2024-03-26T03:05:43.970701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:29.034380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":348,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.974483+0000","last_change":"2024-03-26T03:01:36.769019+0000","last_active":"2024-03-26T03:05:43.974483+0000","last_peered":"2024-03-26T03:05:43.974483+0000","last_clean":"2024-03-26T03:05:43.974483+0000","last_became_active":"2024-03-26T02:58:58.945639+0000","last_became_peered":"2024-03-26T02:58:58.945639+0000","last_unstale":"2024-03-26T03:05:43.974483+0000","last_undegraded":"2024-03-26T03:05:43.974483+0000","last_fullsized":"2024-03-26T03:05:43.974483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":286,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978702+0000","last_change":"2024-03-26T03:04:32.627947+0000","last_active":"2024-03-26T03:05:43.978702+0000","last_peered":"2024-03-26T03:05:43.978702+0000","last_clean":"2024-03-26T03:05:43.978702+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:05:43.978702+0000","last_undegraded":"2024-03-26T03:05:43.978702+0000","last_fullsized":"2024-03-26T03:05:43.978702+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:04:32.627849+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:00:17.286796+0000","last_clean_scrub_stamp":"2024-03-26T03:04:32.627849+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:57.916085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971311+0000","last_change":"2024-03-26T03:05:25.691339+0000","last_active":"2024-03-26T03:05:43.971311+0000","last_peered":"2024-03-26T03:05:43.971311+0000","last_clean":"2024-03-26T03:05:43.971311+0000","last_became_active":"2024-03-26T02:59:02.943280+0000","last_became_peered":"2024-03-26T02:59:02.943280+0000","last_unstale":"2024-03-26T03:05:43.971311+0000","last_undegraded":"2024-03-26T03:05:43.971311+0000","last_fullsized":"2024-03-26T03:05:43.971311+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:45.269505+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":357,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971421+0000","last_change":"2024-03-26T03:04:42.002277+0000","last_active":"2024-03-26T03:05:43.971421+0000","last_peered":"2024-03-26T03:05:43.971421+0000","last_clean":"2024-03-26T03:05:43.971421+0000","last_became_active":"2024-03-26T03:04:41.967264+0000","last_became_peered":"2024-03-26T03:04:41.967264+0000","last_unstale":"2024-03-26T03:05:43.971421+0000","last_undegraded":"2024-03-26T03:05:43.971421+0000","last_fullsized":"2024-03-26T03:05:43.971421+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.002145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.002145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:07.546577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"191'4309","reported_seq":4355,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.803580+0000","last_change":"2024-03-26T03:04:52.198930+0000","last_active":"2024-03-26T03:05:44.803580+0000","last_peered":"2024-03-26T03:05:44.803580+0000","last_clean":"2024-03-26T03:05:44.803580+0000","last_became_active":"2024-03-26T03:04:52.198593+0000","last_became_peered":"2024-03-26T03:04:52.198593+0000","last_unstale":"2024-03-26T03:05:44.803580+0000","last_undegraded":"2024-03-26T03:05:44.803580+0000","last_fullsized":"2024-03-26T03:05:44.803580+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:51.341071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282394624,"num_objects":4309,"num_object_clones":0,"num_object_copies":12927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":275776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981803+0000","last_change":"2024-03-26T03:05:40.581414+0000","last_active":"2024-03-26T03:05:43.981803+0000","last_peered":"2024-03-26T03:05:43.981803+0000","last_clean":"2024-03-26T03:05:43.981803+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:05:43.981803+0000","last_undegraded":"2024-03-26T03:05:43.981803+0000","last_fullsized":"2024-03-26T03:05:43.981803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:40.581302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:40.581302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:41.546973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":336,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.989128+0000","last_change":"2024-03-26T03:05:21.641808+0000","last_active":"2024-03-26T03:05:43.989128+0000","last_peered":"2024-03-26T03:05:43.989128+0000","last_clean":"2024-03-26T03:05:43.989128+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:05:43.989128+0000","last_undegraded":"2024-03-26T03:05:43.989128+0000","last_fullsized":"2024-03-26T03:05:43.989128+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:05:21.641655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:21.641655+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:33.393939+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994389+0000","last_change":"2024-03-26T03:04:35.520145+0000","last_active":"2024-03-26T03:05:43.994389+0000","last_peered":"2024-03-26T03:05:43.994389+0000","last_clean":"2024-03-26T03:05:43.994389+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:05:43.994389+0000","last_undegraded":"2024-03-26T03:05:43.994389+0000","last_fullsized":"2024-03-26T03:05:43.994389+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.520064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.520064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:48.039960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":280,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977800+0000","last_change":"2024-03-26T03:05:37.842641+0000","last_active":"2024-03-26T03:05:43.977800+0000","last_peered":"2024-03-26T03:05:43.977800+0000","last_clean":"2024-03-26T03:05:43.977800+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:05:43.977800+0000","last_undegraded":"2024-03-26T03:05:43.977800+0000","last_fullsized":"2024-03-26T03:05:43.977800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:37.842529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.842529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:59.989205+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":363,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.982021+0000","last_change":"2024-03-26T03:04:43.812006+0000","last_active":"2024-03-26T03:05:43.982021+0000","last_peered":"2024-03-26T03:05:43.982021+0000","last_clean":"2024-03-26T03:05:43.982021+0000","last_became_active":"2024-03-26T03:04:41.970404+0000","last_became_peered":"2024-03-26T03:04:41.970404+0000","last_unstale":"2024-03-26T03:05:43.982021+0000","last_undegraded":"2024-03-26T03:05:43.982021+0000","last_fullsized":"2024-03-26T03:05:43.982021+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:48.587501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"191'4329","reported_seq":4377,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.206884+0000","last_change":"2024-03-26T03:04:52.198694+0000","last_active":"2024-03-26T03:05:44.206884+0000","last_peered":"2024-03-26T03:05:44.206884+0000","last_clean":"2024-03-26T03:05:44.206884+0000","last_became_active":"2024-03-26T03:04:52.198372+0000","last_became_peered":"2024-03-26T03:04:52.198372+0000","last_unstale":"2024-03-26T03:05:44.206884+0000","last_undegraded":"2024-03-26T03:05:44.206884+0000","last_fullsized":"2024-03-26T03:05:44.206884+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:19.398452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283705344,"num_objects":4329,"num_object_clones":0,"num_object_copies":12987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4329,"num_write_kb":277056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976714+0000","last_change":"2024-03-26T03:05:34.665897+0000","last_active":"2024-03-26T03:05:43.976714+0000","last_peered":"2024-03-26T03:05:43.976714+0000","last_clean":"2024-03-26T03:05:43.976714+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:05:43.976714+0000","last_undegraded":"2024-03-26T03:05:43.976714+0000","last_fullsized":"2024-03-26T03:05:43.976714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.665798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.665798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:56.110267+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":306,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.990137+0000","last_change":"2024-03-26T03:05:18.666697+0000","last_active":"2024-03-26T03:05:43.990137+0000","last_peered":"2024-03-26T03:05:43.990137+0000","last_clean":"2024-03-26T03:05:43.990137+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:05:43.990137+0000","last_undegraded":"2024-03-26T03:05:43.990137+0000","last_fullsized":"2024-03-26T03:05:43.990137+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:05:18.662600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:18.662600+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:25.832402+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993061+0000","last_change":"2024-03-26T03:05:28.667880+0000","last_active":"2024-03-26T03:05:43.993061+0000","last_peered":"2024-03-26T03:05:43.993061+0000","last_clean":"2024-03-26T03:05:43.993061+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:05:43.993061+0000","last_undegraded":"2024-03-26T03:05:43.993061+0000","last_fullsized":"2024-03-26T03:05:43.993061+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:28.667767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:28.667767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:34.452455+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":291,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988653+0000","last_change":"2024-03-26T03:04:36.749372+0000","last_active":"2024-03-26T03:05:43.988653+0000","last_peered":"2024-03-26T03:05:43.988653+0000","last_clean":"2024-03-26T03:05:43.988653+0000","last_became_active":"2024-03-26T02:58:56.915863+0000","last_became_peered":"2024-03-26T02:58:56.915863+0000","last_unstale":"2024-03-26T03:05:43.988653+0000","last_undegraded":"2024-03-26T03:05:43.988653+0000","last_fullsized":"2024-03-26T03:05:43.988653+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.749289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.749289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:45.385780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977486+0000","last_change":"2024-03-26T03:04:34.648671+0000","last_active":"2024-03-26T03:05:43.977486+0000","last_peered":"2024-03-26T03:05:43.977486+0000","last_clean":"2024-03-26T03:05:43.977486+0000","last_became_active":"2024-03-26T02:59:02.934726+0000","last_became_peered":"2024-03-26T02:59:02.934726+0000","last_unstale":"2024-03-26T03:05:43.977486+0000","last_undegraded":"2024-03-26T03:05:43.977486+0000","last_fullsized":"2024-03-26T03:05:43.977486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.648562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.648562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":357,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978901+0000","last_change":"2024-03-26T03:04:42.604828+0000","last_active":"2024-03-26T03:05:43.978901+0000","last_peered":"2024-03-26T03:05:43.978901+0000","last_clean":"2024-03-26T03:05:43.978901+0000","last_became_active":"2024-03-26T03:04:41.964807+0000","last_became_peered":"2024-03-26T03:04:41.964807+0000","last_unstale":"2024-03-26T03:05:43.978901+0000","last_undegraded":"2024-03-26T03:05:43.978901+0000","last_fullsized":"2024-03-26T03:05:43.978901+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.604727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.604727+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:52.917317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"191'4217","reported_seq":4264,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.987546+0000","last_change":"2024-03-26T03:04:52.166972+0000","last_active":"2024-03-26T03:05:43.987546+0000","last_peered":"2024-03-26T03:05:43.987546+0000","last_clean":"2024-03-26T03:05:43.987546+0000","last_became_active":"2024-03-26T03:04:52.166591+0000","last_became_peered":"2024-03-26T03:04:52.166591+0000","last_unstale":"2024-03-26T03:05:43.987546+0000","last_undegraded":"2024-03-26T03:05:43.987546+0000","last_fullsized":"2024-03-26T03:05:43.987546+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4217,"log_dups_size":0,"ondisk_log_size":4217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:04.715629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276365312,"num_objects":4217,"num_object_clones":0,"num_object_copies":12651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":269888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994292+0000","last_change":"2024-03-26T03:05:20.645163+0000","last_active":"2024-03-26T03:05:43.994292+0000","last_peered":"2024-03-26T03:05:43.994292+0000","last_clean":"2024-03-26T03:05:43.994292+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:05:43.994292+0000","last_undegraded":"2024-03-26T03:05:43.994292+0000","last_fullsized":"2024-03-26T03:05:43.994292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.645063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.645063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:43.749493+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":392,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981895+0000","last_change":"2024-03-26T03:05:21.760217+0000","last_active":"2024-03-26T03:05:43.981895+0000","last_peered":"2024-03-26T03:05:43.981895+0000","last_clean":"2024-03-26T03:05:43.981895+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:05:43.981895+0000","last_undegraded":"2024-03-26T03:05:43.981895+0000","last_fullsized":"2024-03-26T03:05:43.981895+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:21.760091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:21.760091+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:48.041356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":540,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993314+0000","last_change":"2024-03-26T03:04:37.514101+0000","last_active":"2024-03-26T03:05:43.993314+0000","last_peered":"2024-03-26T03:05:43.993314+0000","last_clean":"2024-03-26T03:05:43.993314+0000","last_became_active":"2024-03-26T03:04:37.115068+0000","last_became_peered":"2024-03-26T03:04:37.115068+0000","last_unstale":"2024-03-26T03:05:43.993314+0000","last_undegraded":"2024-03-26T03:05:43.993314+0000","last_fullsized":"2024-03-26T03:05:43.993314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:37.514019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:37.514019+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:06.164221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":305,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976268+0000","last_change":"2024-03-26T03:05:40.720703+0000","last_active":"2024-03-26T03:05:43.976268+0000","last_peered":"2024-03-26T03:05:43.976268+0000","last_clean":"2024-03-26T03:05:43.976268+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:05:43.976268+0000","last_undegraded":"2024-03-26T03:05:43.976268+0000","last_fullsized":"2024-03-26T03:05:43.976268+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:05:40.720544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:47.098744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":293,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976026+0000","last_change":"2024-03-26T03:05:26.989337+0000","last_active":"2024-03-26T03:05:43.976026+0000","last_peered":"2024-03-26T03:05:43.976026+0000","last_clean":"2024-03-26T03:05:43.976026+0000","last_became_active":"2024-03-26T02:58:56.924006+0000","last_became_peered":"2024-03-26T02:58:56.924006+0000","last_unstale":"2024-03-26T03:05:43.976026+0000","last_undegraded":"2024-03-26T03:05:43.976026+0000","last_fullsized":"2024-03-26T03:05:43.976026+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:52.166634+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976803+0000","last_change":"2024-03-26T03:04:36.275126+0000","last_active":"2024-03-26T03:05:43.976803+0000","last_peered":"2024-03-26T03:05:43.976803+0000","last_clean":"2024-03-26T03:05:43.976803+0000","last_became_active":"2024-03-26T02:59:02.942155+0000","last_became_peered":"2024-03-26T02:59:02.942155+0000","last_unstale":"2024-03-26T03:05:43.976803+0000","last_undegraded":"2024-03-26T03:05:43.976803+0000","last_fullsized":"2024-03-26T03:05:43.976803+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.985829+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"191'4201","reported_seq":4249,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.770732+0000","last_change":"2024-03-26T03:04:52.157100+0000","last_active":"2024-03-26T03:05:44.770732+0000","last_peered":"2024-03-26T03:05:44.770732+0000","last_clean":"2024-03-26T03:05:44.770732+0000","last_became_active":"2024-03-26T03:04:52.156769+0000","last_became_peered":"2024-03-26T03:04:52.156769+0000","last_unstale":"2024-03-26T03:05:44.770732+0000","last_undegraded":"2024-03-26T03:05:44.770732+0000","last_fullsized":"2024-03-26T03:05:44.770732+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:06.242188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275316736,"num_objects":4201,"num_object_clones":0,"num_object_copies":12603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":268864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.987617+0000","last_change":"2024-03-26T03:05:19.409953+0000","last_active":"2024-03-26T03:05:43.987617+0000","last_peered":"2024-03-26T03:05:43.987617+0000","last_clean":"2024-03-26T03:05:43.987617+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:05:43.987617+0000","last_undegraded":"2024-03-26T03:05:43.987617+0000","last_fullsized":"2024-03-26T03:05:43.987617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:19.409819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:19.409819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:44.165248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":319,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978593+0000","last_change":"2024-03-26T03:04:36.643519+0000","last_active":"2024-03-26T03:05:43.978593+0000","last_peered":"2024-03-26T03:05:43.978593+0000","last_clean":"2024-03-26T03:05:43.978593+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:05:43.978593+0000","last_undegraded":"2024-03-26T03:05:43.978593+0000","last_fullsized":"2024-03-26T03:05:43.978593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:04:36.643371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.643371+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:51.493597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.979859+0000","last_change":"2024-03-26T03:05:16.457235+0000","last_active":"2024-03-26T03:05:43.979859+0000","last_peered":"2024-03-26T03:05:43.979859+0000","last_clean":"2024-03-26T03:05:43.979859+0000","last_became_active":"2024-03-26T02:58:56.927123+0000","last_became_peered":"2024-03-26T02:58:56.927123+0000","last_unstale":"2024-03-26T03:05:43.979859+0000","last_undegraded":"2024-03-26T03:05:43.979859+0000","last_fullsized":"2024-03-26T03:05:43.979859+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:38.048745+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":700,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975419+0000","last_change":"2024-03-26T03:05:29.766106+0000","last_active":"2024-03-26T03:05:43.975419+0000","last_peered":"2024-03-26T03:05:43.975419+0000","last_clean":"2024-03-26T03:05:43.975419+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:05:43.975419+0000","last_undegraded":"2024-03-26T03:05:43.975419+0000","last_fullsized":"2024-03-26T03:05:43.975419+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:05:29.765966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.765966+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:31.283368+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":398,"num_read_kb":393,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":280,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994304+0000","last_change":"2024-03-26T03:05:43.562183+0000","last_active":"2024-03-26T03:05:43.994304+0000","last_peered":"2024-03-26T03:05:43.994304+0000","last_clean":"2024-03-26T03:05:43.994304+0000","last_became_active":"2024-03-26T02:59:03.207155+0000","last_became_peered":"2024-03-26T02:59:03.207155+0000","last_unstale":"2024-03-26T03:05:43.994304+0000","last_undegraded":"2024-03-26T03:05:43.994304+0000","last_fullsized":"2024-03-26T03:05:43.994304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:47.974666+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":547,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993675+0000","last_change":"2024-03-26T03:04:43.651718+0000","last_active":"2024-03-26T03:05:43.993675+0000","last_peered":"2024-03-26T03:05:43.993675+0000","last_clean":"2024-03-26T03:05:43.993675+0000","last_became_active":"2024-03-26T03:04:41.965091+0000","last_became_peered":"2024-03-26T03:04:41.965091+0000","last_unstale":"2024-03-26T03:05:43.993675+0000","last_undegraded":"2024-03-26T03:05:43.993675+0000","last_fullsized":"2024-03-26T03:05:43.993675+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.651633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.651633+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:09.790183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"190'4237","reported_seq":4284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975740+0000","last_change":"2024-03-26T03:04:52.164646+0000","last_active":"2024-03-26T03:05:43.975740+0000","last_peered":"2024-03-26T03:05:43.975740+0000","last_clean":"2024-03-26T03:05:43.975740+0000","last_became_active":"2024-03-26T03:04:52.164220+0000","last_became_peered":"2024-03-26T03:04:52.164220+0000","last_unstale":"2024-03-26T03:05:43.975740+0000","last_undegraded":"2024-03-26T03:05:43.975740+0000","last_fullsized":"2024-03-26T03:05:43.975740+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:18.073246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277676032,"num_objects":4237,"num_object_clones":0,"num_object_copies":12711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4237,"num_write_kb":271168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988504+0000","last_change":"2024-03-26T03:05:41.687687+0000","last_active":"2024-03-26T03:05:43.988504+0000","last_peered":"2024-03-26T03:05:43.988504+0000","last_clean":"2024-03-26T03:05:43.988504+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:05:43.988504+0000","last_undegraded":"2024-03-26T03:05:43.988504+0000","last_fullsized":"2024-03-26T03:05:43.988504+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:41.687546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:41.687546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:57.544446+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"86'45","reported_seq":441,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977036+0000","last_change":"2024-03-26T03:05:25.716607+0000","last_active":"2024-03-26T03:05:43.977036+0000","last_peered":"2024-03-26T03:05:43.977036+0000","last_clean":"2024-03-26T03:05:43.977036+0000","last_became_active":"2024-03-26T02:58:58.931487+0000","last_became_peered":"2024-03-26T02:58:58.931487+0000","last_unstale":"2024-03-26T03:05:43.977036+0000","last_undegraded":"2024-03-26T03:05:43.977036+0000","last_fullsized":"2024-03-26T03:05:43.977036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:34.492885+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":84,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.983996+0000","last_change":"2024-03-26T03:05:43.795666+0000","last_active":"2024-03-26T03:05:43.983996+0000","last_peered":"2024-03-26T03:05:43.983996+0000","last_clean":"2024-03-26T03:05:43.983996+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:05:43.983996+0000","last_undegraded":"2024-03-26T03:05:43.983996+0000","last_fullsized":"2024-03-26T03:05:43.983996+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.795558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.795558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:04.877924+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":264,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977184+0000","last_change":"2024-03-26T03:01:33.720224+0000","last_active":"2024-03-26T03:05:43.977184+0000","last_peered":"2024-03-26T03:05:43.977184+0000","last_clean":"2024-03-26T03:05:43.977184+0000","last_became_active":"2024-03-26T02:59:02.941365+0000","last_became_peered":"2024-03-26T02:59:02.941365+0000","last_unstale":"2024-03-26T03:05:43.977184+0000","last_undegraded":"2024-03-26T03:05:43.977184+0000","last_fullsized":"2024-03-26T03:05:43.977184+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:33.719795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:01:33.719795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":363,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988553+0000","last_change":"2024-03-26T03:04:42.737754+0000","last_active":"2024-03-26T03:05:43.988553+0000","last_peered":"2024-03-26T03:05:43.988553+0000","last_clean":"2024-03-26T03:05:43.988553+0000","last_became_active":"2024-03-26T03:04:41.966771+0000","last_became_peered":"2024-03-26T03:04:41.966771+0000","last_unstale":"2024-03-26T03:05:43.988553+0000","last_undegraded":"2024-03-26T03:05:43.988553+0000","last_fullsized":"2024-03-26T03:05:43.988553+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.737606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.737606+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:06.784665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"191'4322","reported_seq":4369,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:45.100259+0000","last_change":"2024-03-26T03:04:52.197778+0000","last_active":"2024-03-26T03:05:45.100259+0000","last_peered":"2024-03-26T03:05:45.100259+0000","last_clean":"2024-03-26T03:05:45.100259+0000","last_became_active":"2024-03-26T03:04:52.197016+0000","last_became_peered":"2024-03-26T03:04:52.197016+0000","last_unstale":"2024-03-26T03:05:45.100259+0000","last_undegraded":"2024-03-26T03:05:45.100259+0000","last_fullsized":"2024-03-26T03:05:45.100259+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:07.883964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283246592,"num_objects":4322,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4322,"num_write_kb":276608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":377,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.989509+0000","last_change":"2024-03-26T03:05:19.664549+0000","last_active":"2024-03-26T03:05:43.989509+0000","last_peered":"2024-03-26T03:05:43.989509+0000","last_clean":"2024-03-26T03:05:43.989509+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:05:43.989509+0000","last_undegraded":"2024-03-26T03:05:43.989509+0000","last_fullsized":"2024-03-26T03:05:43.989509+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:05:19.664419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:19.664419+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:42.056161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"92'22","reported_seq":313,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.984334+0000","last_change":"2024-03-26T03:05:14.583778+0000","last_active":"2024-03-26T03:05:43.984334+0000","last_peered":"2024-03-26T03:05:43.984334+0000","last_clean":"2024-03-26T03:05:43.984334+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:05:43.984334+0000","last_undegraded":"2024-03-26T03:05:43.984334+0000","last_fullsized":"2024-03-26T03:05:43.984334+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:05:14.583487+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:05:14.583487+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:36.726075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977153+0000","last_change":"2024-03-26T03:04:33.655187+0000","last_active":"2024-03-26T03:05:43.977153+0000","last_peered":"2024-03-26T03:05:43.977153+0000","last_clean":"2024-03-26T03:05:43.977153+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:05:43.977153+0000","last_undegraded":"2024-03-26T03:05:43.977153+0000","last_fullsized":"2024-03-26T03:05:43.977153+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:33.655103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:33.655103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":280,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988193+0000","last_change":"2024-03-26T03:05:24.614342+0000","last_active":"2024-03-26T03:05:43.988193+0000","last_peered":"2024-03-26T03:05:43.988193+0000","last_clean":"2024-03-26T03:05:43.988193+0000","last_became_active":"2024-03-26T02:59:02.946425+0000","last_became_peered":"2024-03-26T02:59:02.946425+0000","last_unstale":"2024-03-26T03:05:43.988193+0000","last_undegraded":"2024-03-26T03:05:43.988193+0000","last_fullsized":"2024-03-26T03:05:43.988193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:46.594475+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":363,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.982706+0000","last_change":"2024-03-26T03:04:42.846233+0000","last_active":"2024-03-26T03:05:43.982706+0000","last_peered":"2024-03-26T03:05:43.982706+0000","last_clean":"2024-03-26T03:05:43.982706+0000","last_became_active":"2024-03-26T03:04:41.969936+0000","last_became_peered":"2024-03-26T03:04:41.969936+0000","last_unstale":"2024-03-26T03:05:43.982706+0000","last_undegraded":"2024-03-26T03:05:43.982706+0000","last_fullsized":"2024-03-26T03:05:43.982706+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:55.734011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"191'4293","reported_seq":4341,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:44.003099+0000","last_change":"2024-03-26T03:04:52.164436+0000","last_active":"2024-03-26T03:05:44.003099+0000","last_peered":"2024-03-26T03:05:44.003099+0000","last_clean":"2024-03-26T03:05:44.003099+0000","last_became_active":"2024-03-26T03:04:52.164107+0000","last_became_peered":"2024-03-26T03:04:52.164107+0000","last_unstale":"2024-03-26T03:05:44.003099+0000","last_undegraded":"2024-03-26T03:05:44.003099+0000","last_fullsized":"2024-03-26T03:05:44.003099+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:20.354862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4293,"num_write_kb":274752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.979319+0000","last_change":"2024-03-26T03:04:38.837560+0000","last_active":"2024-03-26T03:05:43.979319+0000","last_peered":"2024-03-26T03:05:43.979319+0000","last_clean":"2024-03-26T03:05:43.979319+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:05:43.979319+0000","last_undegraded":"2024-03-26T03:05:43.979319+0000","last_fullsized":"2024-03-26T03:05:43.979319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:38.837480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:38.837480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:47.988811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":348,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970568+0000","last_change":"2024-03-26T03:03:50.380060+0000","last_active":"2024-03-26T03:05:43.970568+0000","last_peered":"2024-03-26T03:05:43.970568+0000","last_clean":"2024-03-26T03:05:43.970568+0000","last_became_active":"2024-03-26T02:59:30.865686+0000","last_became_peered":"2024-03-26T02:59:30.865686+0000","last_unstale":"2024-03-26T03:05:43.970568+0000","last_undegraded":"2024-03-26T03:05:43.970568+0000","last_fullsized":"2024-03-26T03:05:43.970568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":300,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971456+0000","last_change":"2024-03-26T03:04:47.886299+0000","last_active":"2024-03-26T03:05:43.971456+0000","last_peered":"2024-03-26T03:05:43.971456+0000","last_clean":"2024-03-26T03:05:43.971456+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:05:43.971456+0000","last_undegraded":"2024-03-26T03:05:43.971456+0000","last_fullsized":"2024-03-26T03:05:43.971456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:04:47.886200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:47.886200+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:49.716601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977119+0000","last_change":"2024-03-26T03:05:20.934344+0000","last_active":"2024-03-26T03:05:43.977119+0000","last_peered":"2024-03-26T03:05:43.977119+0000","last_clean":"2024-03-26T03:05:43.977119+0000","last_became_active":"2024-03-26T02:59:02.942170+0000","last_became_peered":"2024-03-26T02:59:02.942170+0000","last_unstale":"2024-03-26T03:05:43.977119+0000","last_undegraded":"2024-03-26T03:05:43.977119+0000","last_fullsized":"2024-03-26T03:05:43.977119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:35.861932+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":327,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.990238+0000","last_change":"2024-03-26T03:05:35.652733+0000","last_active":"2024-03-26T03:05:43.990238+0000","last_peered":"2024-03-26T03:05:43.990238+0000","last_clean":"2024-03-26T03:05:43.990238+0000","last_became_active":"2024-03-26T02:58:58.945618+0000","last_became_peered":"2024-03-26T02:58:58.945618+0000","last_unstale":"2024-03-26T03:05:43.990238+0000","last_undegraded":"2024-03-26T03:05:43.990238+0000","last_fullsized":"2024-03-26T03:05:43.990238+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:59.529056+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":297,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.979222+0000","last_change":"2024-03-26T03:05:17.402372+0000","last_active":"2024-03-26T03:05:43.979222+0000","last_peered":"2024-03-26T03:05:43.979222+0000","last_clean":"2024-03-26T03:05:43.979222+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:05:43.979222+0000","last_undegraded":"2024-03-26T03:05:43.979222+0000","last_fullsized":"2024-03-26T03:05:43.979222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:05:17.402243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.402243+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:21.997319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978636+0000","last_change":"2024-03-26T03:05:34.055131+0000","last_active":"2024-03-26T03:05:43.978636+0000","last_peered":"2024-03-26T03:05:43.978636+0000","last_clean":"2024-03-26T03:05:43.978636+0000","last_became_active":"2024-03-26T02:59:00.930380+0000","last_became_peered":"2024-03-26T02:59:00.930380+0000","last_unstale":"2024-03-26T03:05:43.978636+0000","last_undegraded":"2024-03-26T03:05:43.978636+0000","last_fullsized":"2024-03-26T03:05:43.978636+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:37.297507+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.980368+0000","last_change":"2024-03-26T03:05:27.373948+0000","last_active":"2024-03-26T03:05:43.980368+0000","last_peered":"2024-03-26T03:05:43.980368+0000","last_clean":"2024-03-26T03:05:43.980368+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:05:43.980368+0000","last_undegraded":"2024-03-26T03:05:43.980368+0000","last_fullsized":"2024-03-26T03:05:43.980368+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:27.373709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:27.373709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:41.569854+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":316,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978325+0000","last_change":"2024-03-26T03:01:50.876168+0000","last_active":"2024-03-26T03:05:43.978325+0000","last_peered":"2024-03-26T03:05:43.978325+0000","last_clean":"2024-03-26T03:05:43.978325+0000","last_became_active":"2024-03-26T02:59:30.868828+0000","last_became_peered":"2024-03-26T02:59:30.868828+0000","last_unstale":"2024-03-26T03:05:43.978325+0000","last_undegraded":"2024-03-26T03:05:43.978325+0000","last_fullsized":"2024-03-26T03:05:43.978325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":291,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981504+0000","last_change":"2024-03-26T03:05:31.682849+0000","last_active":"2024-03-26T03:05:43.981504+0000","last_peered":"2024-03-26T03:05:43.981504+0000","last_clean":"2024-03-26T03:05:43.981504+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:05:43.981504+0000","last_undegraded":"2024-03-26T03:05:43.981504+0000","last_fullsized":"2024-03-26T03:05:43.981504+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:31.682729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:31.682729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:58.945135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":282,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976046+0000","last_change":"2024-03-26T03:05:17.507359+0000","last_active":"2024-03-26T03:05:43.976046+0000","last_peered":"2024-03-26T03:05:43.976046+0000","last_clean":"2024-03-26T03:05:43.976046+0000","last_became_active":"2024-03-26T02:59:00.930262+0000","last_became_peered":"2024-03-26T02:59:00.930262+0000","last_unstale":"2024-03-26T03:05:43.976046+0000","last_undegraded":"2024-03-26T03:05:43.976046+0000","last_fullsized":"2024-03-26T03:05:43.976046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:28.631522+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994158+0000","last_change":"2024-03-26T03:05:16.754129+0000","last_active":"2024-03-26T03:05:43.994158+0000","last_peered":"2024-03-26T03:05:43.994158+0000","last_clean":"2024-03-26T03:05:43.994158+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:05:43.994158+0000","last_undegraded":"2024-03-26T03:05:43.994158+0000","last_fullsized":"2024-03-26T03:05:43.994158+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.754027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.754027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:17.185707+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"151'29","reported_seq":325,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978321+0000","last_change":"2024-03-26T03:03:52.392314+0000","last_active":"2024-03-26T03:05:43.978321+0000","last_peered":"2024-03-26T03:05:43.978321+0000","last_clean":"2024-03-26T03:05:43.978321+0000","last_became_active":"2024-03-26T02:58:58.945647+0000","last_became_peered":"2024-03-26T02:58:58.945647+0000","last_unstale":"2024-03-26T03:05:43.978321+0000","last_undegraded":"2024-03-26T03:05:43.978321+0000","last_fullsized":"2024-03-26T03:05:43.978321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":290,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976136+0000","last_change":"2024-03-26T03:04:40.320823+0000","last_active":"2024-03-26T03:05:43.976136+0000","last_peered":"2024-03-26T03:05:43.976136+0000","last_clean":"2024-03-26T03:05:43.976136+0000","last_became_active":"2024-03-26T02:58:56.927762+0000","last_became_peered":"2024-03-26T02:58:56.927762+0000","last_unstale":"2024-03-26T03:05:43.976136+0000","last_undegraded":"2024-03-26T03:05:43.976136+0000","last_fullsized":"2024-03-26T03:05:43.976136+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.320691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"71'8","reported_seq":372,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977010+0000","last_change":"2024-03-26T03:05:31.665728+0000","last_active":"2024-03-26T03:05:43.977010+0000","last_peered":"2024-03-26T03:05:43.977010+0000","last_clean":"2024-03-26T03:05:43.977010+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:05:43.977010+0000","last_undegraded":"2024-03-26T03:05:43.977010+0000","last_fullsized":"2024-03-26T03:05:43.977010+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:05:31.665613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:31.665613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:35.060627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.989996+0000","last_change":"2024-03-26T03:05:16.678875+0000","last_active":"2024-03-26T03:05:43.989996+0000","last_peered":"2024-03-26T03:05:43.989996+0000","last_clean":"2024-03-26T03:05:43.989996+0000","last_became_active":"2024-03-26T02:59:02.948257+0000","last_became_peered":"2024-03-26T02:59:02.948257+0000","last_unstale":"2024-03-26T03:05:43.989996+0000","last_undegraded":"2024-03-26T03:05:43.989996+0000","last_fullsized":"2024-03-26T03:05:43.989996+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:46.392539+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":123,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994253+0000","last_change":"2024-03-26T03:04:14.416461+0000","last_active":"2024-03-26T03:05:43.994253+0000","last_peered":"2024-03-26T03:05:43.994253+0000","last_clean":"2024-03-26T03:05:43.994253+0000","last_became_active":"2024-03-26T03:04:14.415940+0000","last_became_peered":"2024-03-26T03:04:14.415940+0000","last_unstale":"2024-03-26T03:05:43.994253+0000","last_undegraded":"2024-03-26T03:05:43.994253+0000","last_fullsized":"2024-03-26T03:05:43.994253+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.989248+0000","last_change":"2024-03-26T03:04:46.666940+0000","last_active":"2024-03-26T03:05:43.989248+0000","last_peered":"2024-03-26T03:05:43.989248+0000","last_clean":"2024-03-26T03:05:43.989248+0000","last_became_active":"2024-03-26T02:58:56.914541+0000","last_became_peered":"2024-03-26T02:58:56.914541+0000","last_unstale":"2024-03-26T03:05:43.989248+0000","last_undegraded":"2024-03-26T03:05:43.989248+0000","last_fullsized":"2024-03-26T03:05:43.989248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:05.225883+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"71'8","reported_seq":365,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978161+0000","last_change":"2024-03-26T03:04:44.796750+0000","last_active":"2024-03-26T03:05:43.978161+0000","last_peered":"2024-03-26T03:05:43.978161+0000","last_clean":"2024-03-26T03:05:43.978161+0000","last_became_active":"2024-03-26T02:59:00.933390+0000","last_became_peered":"2024-03-26T02:59:00.933390+0000","last_unstale":"2024-03-26T03:05:43.978161+0000","last_undegraded":"2024-03-26T03:05:43.978161+0000","last_fullsized":"2024-03-26T03:05:43.978161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.839204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971713+0000","last_change":"2024-03-26T03:05:24.775144+0000","last_active":"2024-03-26T03:05:43.971713+0000","last_peered":"2024-03-26T03:05:43.971713+0000","last_clean":"2024-03-26T03:05:43.971713+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:05:43.971713+0000","last_undegraded":"2024-03-26T03:05:43.971713+0000","last_fullsized":"2024-03-26T03:05:43.971713+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:05:24.775044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.775044+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:40.743174+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":295,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.984238+0000","last_change":"2024-03-26T03:05:16.621691+0000","last_active":"2024-03-26T03:05:43.984238+0000","last_peered":"2024-03-26T03:05:43.984238+0000","last_clean":"2024-03-26T03:05:43.984238+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:05:43.984238+0000","last_undegraded":"2024-03-26T03:05:43.984238+0000","last_fullsized":"2024-03-26T03:05:43.984238+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-26T03:05:16.621526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.621526+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:39.497908+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":291,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981564+0000","last_change":"2024-03-26T03:05:15.802290+0000","last_active":"2024-03-26T03:05:43.981564+0000","last_peered":"2024-03-26T03:05:43.981564+0000","last_clean":"2024-03-26T03:05:43.981564+0000","last_became_active":"2024-03-26T02:58:56.927288+0000","last_became_peered":"2024-03-26T02:58:56.927288+0000","last_unstale":"2024-03-26T03:05:43.981564+0000","last_undegraded":"2024-03-26T03:05:43.981564+0000","last_fullsized":"2024-03-26T03:05:43.981564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:16.394666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":275,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977034+0000","last_change":"2024-03-26T03:03:44.286188+0000","last_active":"2024-03-26T03:05:43.977034+0000","last_peered":"2024-03-26T03:05:43.977034+0000","last_clean":"2024-03-26T03:05:43.977034+0000","last_became_active":"2024-03-26T02:59:00.932931+0000","last_became_peered":"2024-03-26T02:59:00.932931+0000","last_unstale":"2024-03-26T03:05:43.977034+0000","last_undegraded":"2024-03-26T03:05:43.977034+0000","last_fullsized":"2024-03-26T03:05:43.977034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":274,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988933+0000","last_change":"2024-03-26T03:05:15.643960+0000","last_active":"2024-03-26T03:05:43.988933+0000","last_peered":"2024-03-26T03:05:43.988933+0000","last_clean":"2024-03-26T03:05:43.988933+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:05:43.988933+0000","last_undegraded":"2024-03-26T03:05:43.988933+0000","last_fullsized":"2024-03-26T03:05:43.988933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.643830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.643830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:27.062308+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":297,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971452+0000","last_change":"2024-03-26T03:04:44.941104+0000","last_active":"2024-03-26T03:05:43.971452+0000","last_peered":"2024-03-26T03:05:43.971452+0000","last_clean":"2024-03-26T03:05:43.971452+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:05:43.971452+0000","last_undegraded":"2024-03-26T03:05:43.971452+0000","last_fullsized":"2024-03-26T03:05:43.971452+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:44.940984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.940984+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:55.600513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":291,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977965+0000","last_change":"2024-03-26T03:05:33.718258+0000","last_active":"2024-03-26T03:05:43.977965+0000","last_peered":"2024-03-26T03:05:43.977965+0000","last_clean":"2024-03-26T03:05:43.977965+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:05:43.977965+0000","last_undegraded":"2024-03-26T03:05:43.977965+0000","last_fullsized":"2024-03-26T03:05:43.977965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:33.718124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:05:33.718124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:43.895613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":275,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.984432+0000","last_change":"2024-03-26T03:05:30.742351+0000","last_active":"2024-03-26T03:05:43.984432+0000","last_peered":"2024-03-26T03:05:43.984432+0000","last_clean":"2024-03-26T03:05:43.984432+0000","last_became_active":"2024-03-26T02:59:00.936502+0000","last_became_peered":"2024-03-26T02:59:00.936502+0000","last_unstale":"2024-03-26T03:05:43.984432+0000","last_undegraded":"2024-03-26T03:05:43.984432+0000","last_fullsized":"2024-03-26T03:05:43.984432+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:50.817167+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975468+0000","last_change":"2024-03-26T03:03:54.412437+0000","last_active":"2024-03-26T03:05:43.975468+0000","last_peered":"2024-03-26T03:05:43.975468+0000","last_clean":"2024-03-26T03:05:43.975468+0000","last_became_active":"2024-03-26T02:59:02.944779+0000","last_became_peered":"2024-03-26T02:59:02.944779+0000","last_unstale":"2024-03-26T03:05:43.975468+0000","last_undegraded":"2024-03-26T03:05:43.975468+0000","last_fullsized":"2024-03-26T03:05:43.975468+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":285,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.974327+0000","last_change":"2024-03-26T03:01:37.776671+0000","last_active":"2024-03-26T03:05:43.974327+0000","last_peered":"2024-03-26T03:05:43.974327+0000","last_clean":"2024-03-26T03:05:43.974327+0000","last_became_active":"2024-03-26T02:58:58.935176+0000","last_became_peered":"2024-03-26T02:58:58.935176+0000","last_unstale":"2024-03-26T03:05:43.974327+0000","last_undegraded":"2024-03-26T03:05:43.974327+0000","last_fullsized":"2024-03-26T03:05:43.974327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":290,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976239+0000","last_change":"2024-03-26T03:04:35.287576+0000","last_active":"2024-03-26T03:05:43.976239+0000","last_peered":"2024-03-26T03:05:43.976239+0000","last_clean":"2024-03-26T03:05:43.976239+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:05:43.976239+0000","last_undegraded":"2024-03-26T03:05:43.976239+0000","last_fullsized":"2024-03-26T03:05:43.976239+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.287500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.287500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:50.321216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981403+0000","last_change":"2024-03-26T03:05:43.606386+0000","last_active":"2024-03-26T03:05:43.981403+0000","last_peered":"2024-03-26T03:05:43.981403+0000","last_clean":"2024-03-26T03:05:43.981403+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:05:43.981403+0000","last_undegraded":"2024-03-26T03:05:43.981403+0000","last_fullsized":"2024-03-26T03:05:43.981403+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.606275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.606275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:43.822179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993710+0000","last_change":"2024-03-26T03:05:23.665275+0000","last_active":"2024-03-26T03:05:43.993710+0000","last_peered":"2024-03-26T03:05:43.993710+0000","last_clean":"2024-03-26T03:05:43.993710+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:05:43.993710+0000","last_undegraded":"2024-03-26T03:05:43.993710+0000","last_fullsized":"2024-03-26T03:05:43.993710+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.664911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.664911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:28.951625+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"150'23","reported_seq":316,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976454+0000","last_change":"2024-03-26T03:03:50.334581+0000","last_active":"2024-03-26T03:05:43.976454+0000","last_peered":"2024-03-26T03:05:43.976454+0000","last_clean":"2024-03-26T03:05:43.976454+0000","last_became_active":"2024-03-26T02:58:58.944804+0000","last_became_peered":"2024-03-26T02:58:58.944804+0000","last_unstale":"2024-03-26T03:05:43.976454+0000","last_undegraded":"2024-03-26T03:05:43.976454+0000","last_fullsized":"2024-03-26T03:05:43.976454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.982754+0000","last_change":"2024-03-26T03:04:34.772569+0000","last_active":"2024-03-26T03:05:43.982754+0000","last_peered":"2024-03-26T03:05:43.982754+0000","last_clean":"2024-03-26T03:05:43.982754+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:05:43.982754+0000","last_undegraded":"2024-03-26T03:05:43.982754+0000","last_fullsized":"2024-03-26T03:05:43.982754+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:34.772488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:34.772488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:52.450698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993300+0000","last_change":"2024-03-26T03:04:40.594869+0000","last_active":"2024-03-26T03:05:43.993300+0000","last_peered":"2024-03-26T03:05:43.993300+0000","last_clean":"2024-03-26T03:05:43.993300+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:05:43.993300+0000","last_undegraded":"2024-03-26T03:05:43.993300+0000","last_fullsized":"2024-03-26T03:05:43.993300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.594788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:44.412549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977346+0000","last_change":"2024-03-26T03:05:38.188714+0000","last_active":"2024-03-26T03:05:43.977346+0000","last_peered":"2024-03-26T03:05:43.977346+0000","last_clean":"2024-03-26T03:05:43.977346+0000","last_became_active":"2024-03-26T02:59:03.206755+0000","last_became_peered":"2024-03-26T02:59:03.206755+0000","last_unstale":"2024-03-26T03:05:43.977346+0000","last_undegraded":"2024-03-26T03:05:43.977346+0000","last_fullsized":"2024-03-26T03:05:43.977346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:49.589002+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":315,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.990426+0000","last_change":"2024-03-26T03:05:43.709313+0000","last_active":"2024-03-26T03:05:43.990426+0000","last_peered":"2024-03-26T03:05:43.990426+0000","last_clean":"2024-03-26T03:05:43.990426+0000","last_became_active":"2024-03-26T02:58:58.945180+0000","last_became_peered":"2024-03-26T02:58:58.945180+0000","last_unstale":"2024-03-26T03:05:43.990426+0000","last_undegraded":"2024-03-26T03:05:43.990426+0000","last_fullsized":"2024-03-26T03:05:43.990426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:47.241634+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":290,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975815+0000","last_change":"2024-03-26T03:04:42.310863+0000","last_active":"2024-03-26T03:05:43.975815+0000","last_peered":"2024-03-26T03:05:43.975815+0000","last_clean":"2024-03-26T03:05:43.975815+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:05:43.975815+0000","last_undegraded":"2024-03-26T03:05:43.975815+0000","last_fullsized":"2024-03-26T03:05:43.975815+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:42.310782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.310782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:10.649449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":369,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.982849+0000","last_change":"2024-03-26T03:04:02.892852+0000","last_active":"2024-03-26T03:05:43.982849+0000","last_peered":"2024-03-26T03:05:43.982849+0000","last_clean":"2024-03-26T03:05:43.982849+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:05:43.982849+0000","last_undegraded":"2024-03-26T03:05:43.982849+0000","last_fullsized":"2024-03-26T03:05:43.982849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:02.892717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.892717+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":280,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994705+0000","last_change":"2024-03-26T03:04:45.715721+0000","last_active":"2024-03-26T03:05:43.994705+0000","last_peered":"2024-03-26T03:05:43.994705+0000","last_clean":"2024-03-26T03:05:43.994705+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:05:43.994705+0000","last_undegraded":"2024-03-26T03:05:43.994705+0000","last_fullsized":"2024-03-26T03:05:43.994705+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:04:45.715635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:45.715635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:58.465028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":319,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.982907+0000","last_change":"2024-03-26T03:05:42.603323+0000","last_active":"2024-03-26T03:05:43.982907+0000","last_peered":"2024-03-26T03:05:43.982907+0000","last_clean":"2024-03-26T03:05:43.982907+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:05:43.982907+0000","last_undegraded":"2024-03-26T03:05:43.982907+0000","last_fullsized":"2024-03-26T03:05:43.982907+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:05:42.603184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:42.603184+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:48.654150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975691+0000","last_change":"2024-03-26T03:04:40.740150+0000","last_active":"2024-03-26T03:05:43.975691+0000","last_peered":"2024-03-26T03:05:43.975691+0000","last_clean":"2024-03-26T03:05:43.975691+0000","last_became_active":"2024-03-26T02:58:56.922467+0000","last_became_peered":"2024-03-26T02:58:56.922467+0000","last_unstale":"2024-03-26T03:05:43.975691+0000","last_undegraded":"2024-03-26T03:05:43.975691+0000","last_fullsized":"2024-03-26T03:05:43.975691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:07.702274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.990509+0000","last_change":"2024-03-26T03:05:22.647188+0000","last_active":"2024-03-26T03:05:43.990509+0000","last_peered":"2024-03-26T03:05:43.990509+0000","last_clean":"2024-03-26T03:05:43.990509+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:05:43.990509+0000","last_undegraded":"2024-03-26T03:05:43.990509+0000","last_fullsized":"2024-03-26T03:05:43.990509+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.647026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.647026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:30.821342+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977723+0000","last_change":"2024-03-26T03:05:24.719757+0000","last_active":"2024-03-26T03:05:43.977723+0000","last_peered":"2024-03-26T03:05:43.977723+0000","last_clean":"2024-03-26T03:05:43.977723+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:05:43.977723+0000","last_undegraded":"2024-03-26T03:05:43.977723+0000","last_fullsized":"2024-03-26T03:05:43.977723+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.719643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:43.600902+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":312,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.980819+0000","last_change":"2024-03-26T03:04:44.684514+0000","last_active":"2024-03-26T03:05:43.980819+0000","last_peered":"2024-03-26T03:05:43.980819+0000","last_clean":"2024-03-26T03:05:43.980819+0000","last_became_active":"2024-03-26T02:58:58.929608+0000","last_became_peered":"2024-03-26T02:58:58.929608+0000","last_unstale":"2024-03-26T03:05:43.980819+0000","last_undegraded":"2024-03-26T03:05:43.980819+0000","last_fullsized":"2024-03-26T03:05:43.980819+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:08.955521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":291,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.981708+0000","last_change":"2024-03-26T03:04:44.859471+0000","last_active":"2024-03-26T03:05:43.981708+0000","last_peered":"2024-03-26T03:05:43.981708+0000","last_clean":"2024-03-26T03:05:43.981708+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:05:43.981708+0000","last_undegraded":"2024-03-26T03:05:43.981708+0000","last_fullsized":"2024-03-26T03:05:43.981708+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:44.859394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.859394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:49.587400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978301+0000","last_change":"2024-03-26T03:05:27.725312+0000","last_active":"2024-03-26T03:05:43.978301+0000","last_peered":"2024-03-26T03:05:43.978301+0000","last_clean":"2024-03-26T03:05:43.978301+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:05:43.978301+0000","last_undegraded":"2024-03-26T03:05:43.978301+0000","last_fullsized":"2024-03-26T03:05:43.978301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:27.725192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:27.725192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:40.341761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971255+0000","last_change":"2024-03-26T03:05:35.639348+0000","last_active":"2024-03-26T03:05:43.971255+0000","last_peered":"2024-03-26T03:05:43.971255+0000","last_clean":"2024-03-26T03:05:43.971255+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:05:43.971255+0000","last_undegraded":"2024-03-26T03:05:43.971255+0000","last_fullsized":"2024-03-26T03:05:43.971255+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:35.639230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.639230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:42.780308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":314,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976867+0000","last_change":"2024-03-26T03:03:47.301632+0000","last_active":"2024-03-26T03:05:43.976867+0000","last_peered":"2024-03-26T03:05:43.976867+0000","last_clean":"2024-03-26T03:05:43.976867+0000","last_became_active":"2024-03-26T02:58:58.934833+0000","last_became_peered":"2024-03-26T02:58:58.934833+0000","last_unstale":"2024-03-26T03:05:43.976867+0000","last_undegraded":"2024-03-26T03:05:43.976867+0000","last_fullsized":"2024-03-26T03:05:43.976867+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976822+0000","last_change":"2024-03-26T03:01:49.896560+0000","last_active":"2024-03-26T03:05:43.976822+0000","last_peered":"2024-03-26T03:05:43.976822+0000","last_clean":"2024-03-26T03:05:43.976822+0000","last_became_active":"2024-03-26T02:58:56.923272+0000","last_became_peered":"2024-03-26T02:58:56.923272+0000","last_unstale":"2024-03-26T03:05:43.976822+0000","last_undegraded":"2024-03-26T03:05:43.976822+0000","last_fullsized":"2024-03-26T03:05:43.976822+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":274,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971273+0000","last_change":"2024-03-26T03:03:53.357848+0000","last_active":"2024-03-26T03:05:43.971273+0000","last_peered":"2024-03-26T03:05:43.971273+0000","last_clean":"2024-03-26T03:05:43.971273+0000","last_became_active":"2024-03-26T02:59:00.942498+0000","last_became_peered":"2024-03-26T02:59:00.942498+0000","last_unstale":"2024-03-26T03:05:43.971273+0000","last_undegraded":"2024-03-26T03:05:43.971273+0000","last_fullsized":"2024-03-26T03:05:43.971273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.978485+0000","last_change":"2024-03-26T03:05:22.998322+0000","last_active":"2024-03-26T03:05:43.978485+0000","last_peered":"2024-03-26T03:05:43.978485+0000","last_clean":"2024-03-26T03:05:43.978485+0000","last_became_active":"2024-03-26T02:59:03.206417+0000","last_became_peered":"2024-03-26T02:59:03.206417+0000","last_unstale":"2024-03-26T03:05:43.978485+0000","last_undegraded":"2024-03-26T03:05:43.978485+0000","last_fullsized":"2024-03-26T03:05:43.978485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:51.464322+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"188'27","reported_seq":332,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970993+0000","last_change":"2024-03-26T03:05:37.637940+0000","last_active":"2024-03-26T03:05:43.970993+0000","last_peered":"2024-03-26T03:05:43.970993+0000","last_clean":"2024-03-26T03:05:43.970993+0000","last_became_active":"2024-03-26T02:58:58.930847+0000","last_became_peered":"2024-03-26T02:58:58.930847+0000","last_unstale":"2024-03-26T03:05:43.970993+0000","last_undegraded":"2024-03-26T03:05:43.970993+0000","last_fullsized":"2024-03-26T03:05:43.970993+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:40.088856+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":291,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994706+0000","last_change":"2024-03-26T03:05:18.717654+0000","last_active":"2024-03-26T03:05:43.994706+0000","last_peered":"2024-03-26T03:05:43.994706+0000","last_clean":"2024-03-26T03:05:43.994706+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:05:43.994706+0000","last_undegraded":"2024-03-26T03:05:43.994706+0000","last_fullsized":"2024-03-26T03:05:43.994706+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:18.717525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:18.717525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:34.887927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":282,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976185+0000","last_change":"2024-03-26T03:04:37.286514+0000","last_active":"2024-03-26T03:05:43.976185+0000","last_peered":"2024-03-26T03:05:43.976185+0000","last_clean":"2024-03-26T03:05:43.976185+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:05:43.976185+0000","last_undegraded":"2024-03-26T03:05:43.976185+0000","last_fullsized":"2024-03-26T03:05:43.976185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:37.286411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:37.286411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.988761+0000","last_change":"2024-03-26T03:05:26.597073+0000","last_active":"2024-03-26T03:05:43.988761+0000","last_peered":"2024-03-26T03:05:43.988761+0000","last_clean":"2024-03-26T03:05:43.988761+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:05:43.988761+0000","last_undegraded":"2024-03-26T03:05:43.988761+0000","last_fullsized":"2024-03-26T03:05:43.988761+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:26.596958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.596958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:27.259439+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":300,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.971624+0000","last_change":"2024-03-26T03:04:41.075472+0000","last_active":"2024-03-26T03:05:43.971624+0000","last_peered":"2024-03-26T03:05:43.971624+0000","last_clean":"2024-03-26T03:05:43.971624+0000","last_became_active":"2024-03-26T02:58:58.930753+0000","last_became_peered":"2024-03-26T02:58:58.930753+0000","last_unstale":"2024-03-26T03:05:43.971624+0000","last_undegraded":"2024-03-26T03:05:43.971624+0000","last_fullsized":"2024-03-26T03:05:43.971624+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:05:58.734700+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.994224+0000","last_change":"2024-03-26T03:04:04.517568+0000","last_active":"2024-03-26T03:05:43.994224+0000","last_peered":"2024-03-26T03:05:43.994224+0000","last_clean":"2024-03-26T03:05:43.994224+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:05:43.994224+0000","last_undegraded":"2024-03-26T03:05:43.994224+0000","last_fullsized":"2024-03-26T03:05:43.994224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:04.517463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:04.517463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":283,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.993583+0000","last_change":"2024-03-26T03:05:21.649497+0000","last_active":"2024-03-26T03:05:43.993583+0000","last_peered":"2024-03-26T03:05:43.993583+0000","last_clean":"2024-03-26T03:05:43.993583+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:05:43.993583+0000","last_undegraded":"2024-03-26T03:05:43.993583+0000","last_fullsized":"2024-03-26T03:05:43.993583+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:21.649373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:21.649373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:39.326280+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":278,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976723+0000","last_change":"2024-03-26T03:05:20.598926+0000","last_active":"2024-03-26T03:05:43.976723+0000","last_peered":"2024-03-26T03:05:43.976723+0000","last_clean":"2024-03-26T03:05:43.976723+0000","last_became_active":"2024-03-26T02:59:02.942792+0000","last_became_peered":"2024-03-26T02:59:02.942792+0000","last_unstale":"2024-03-26T03:05:43.976723+0000","last_undegraded":"2024-03-26T03:05:43.976723+0000","last_fullsized":"2024-03-26T03:05:43.976723+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:27.773479+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":306,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970196+0000","last_change":"2024-03-26T03:04:40.028444+0000","last_active":"2024-03-26T03:05:43.970196+0000","last_peered":"2024-03-26T03:05:43.970196+0000","last_clean":"2024-03-26T03:05:43.970196+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:05:43.970196+0000","last_undegraded":"2024-03-26T03:05:43.970196+0000","last_fullsized":"2024-03-26T03:05:43.970196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:40.028340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.028340+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977938+0000","last_change":"2024-03-26T03:05:25.976763+0000","last_active":"2024-03-26T03:05:43.977938+0000","last_peered":"2024-03-26T03:05:43.977938+0000","last_clean":"2024-03-26T03:05:43.977938+0000","last_became_active":"2024-03-26T02:58:56.920206+0000","last_became_peered":"2024-03-26T02:58:56.920206+0000","last_unstale":"2024-03-26T03:05:43.977938+0000","last_undegraded":"2024-03-26T03:05:43.977938+0000","last_fullsized":"2024-03-26T03:05:43.977938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:32.188708+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":275,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.977730+0000","last_change":"2024-03-26T03:05:40.845183+0000","last_active":"2024-03-26T03:05:43.977730+0000","last_peered":"2024-03-26T03:05:43.977730+0000","last_clean":"2024-03-26T03:05:43.977730+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:05:43.977730+0000","last_undegraded":"2024-03-26T03:05:43.977730+0000","last_fullsized":"2024-03-26T03:05:43.977730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:40.845075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:40.845075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:53.008824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":270,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970952+0000","last_change":"2024-03-26T03:04:01.340828+0000","last_active":"2024-03-26T03:05:43.970952+0000","last_peered":"2024-03-26T03:05:43.970952+0000","last_clean":"2024-03-26T03:05:43.970952+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:05:43.970952+0000","last_undegraded":"2024-03-26T03:05:43.970952+0000","last_fullsized":"2024-03-26T03:05:43.970952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:01.340706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:01.340706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":353,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.975557+0000","last_change":"2024-03-26T03:05:32.082939+0000","last_active":"2024-03-26T03:05:43.975557+0000","last_peered":"2024-03-26T03:05:43.975557+0000","last_clean":"2024-03-26T03:05:43.975557+0000","last_became_active":"2024-03-26T02:58:58.929474+0000","last_became_peered":"2024-03-26T02:58:58.929474+0000","last_unstale":"2024-03-26T03:05:43.975557+0000","last_undegraded":"2024-03-26T03:05:43.975557+0000","last_fullsized":"2024-03-26T03:05:43.975557+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":11,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:56.526687+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":292,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.989764+0000","last_change":"2024-03-26T03:05:23.629721+0000","last_active":"2024-03-26T03:05:43.989764+0000","last_peered":"2024-03-26T03:05:43.989764+0000","last_clean":"2024-03-26T03:05:43.989764+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:05:43.989764+0000","last_undegraded":"2024-03-26T03:05:43.989764+0000","last_fullsized":"2024-03-26T03:05:43.989764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.629601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.629601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:38.741075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":282,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.970358+0000","last_change":"2024-03-26T03:05:36.609903+0000","last_active":"2024-03-26T03:05:43.970358+0000","last_peered":"2024-03-26T03:05:43.970358+0000","last_clean":"2024-03-26T03:05:43.970358+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:05:43.970358+0000","last_undegraded":"2024-03-26T03:05:43.970358+0000","last_fullsized":"2024-03-26T03:05:43.970358+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:36.609793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:36.609793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:41.709894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:05:43.976738+0000","last_change":"2024-03-26T03:05:17.983303+0000","last_active":"2024-03-26T03:05:43.976738+0000","last_peered":"2024-03-26T03:05:43.976738+0000","last_clean":"2024-03-26T03:05:43.976738+0000","last_became_active":"2024-03-26T02:59:03.208488+0000","last_became_peered":"2024-03-26T02:59:03.208488+0000","last_unstale":"2024-03-26T03:05:43.976738+0000","last_undegraded":"2024-03-26T03:05:43.976738+0000","last_fullsized":"2024-03-26T03:05:43.976738+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:06:33.241120+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4433575936,"num_objects":67651,"num_object_clones":0,"num_object_copies":202953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67651,"num_write_kb":4329664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13273792512,"data_stored":13273792512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67651,"ondisk_log_size":67651,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":398,"num_read_kb":393,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":647,"ondisk_log_size":647,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103254,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1563104,"kb_used_data":1337556,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92182048,"statfs":{"total":95995035648,"available":94394417152,"internally_reserved":0,"allocated":1369657344,"data_stored":1369159846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":230947274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430298,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1535128,"kb_used_data":1341516,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92210024,"statfs":{"total":95995035648,"available":94423064576,"internally_reserved":0,"allocated":1373712384,"data_stored":1373222460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043580,"kb_used_data":1868016,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91701572,"statfs":{"total":95995035648,"available":93902409728,"internally_reserved":0,"allocated":1912848384,"data_stored":1912323416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1910452,"kb_used_data":1622440,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91834700,"statfs":{"total":95995035648,"available":94038732800,"internally_reserved":0,"allocated":1661378560,"data_stored":1660868153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1816156,"kb_used_data":1604496,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":91928996,"statfs":{"total":95995035648,"available":94135291904,"internally_reserved":0,"allocated":1643003904,"data_stored":1642511485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1822404,"kb_used_data":1614904,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":91922748,"statfs":{"total":95995035648,"available":94128893952,"internally_reserved":0,"allocated":1653661696,"data_stored":1653134607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986276,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454120,"kb_used_data":1332704,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92291032,"statfs":{"total":95995035648,"available":94506016768,"internally_reserved":0,"allocated":1364688896,"data_stored":1364179019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691936,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2281568,"kb_used_data":2128212,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91463584,"statfs":{"total":95995035648,"available":93658710016,"internally_reserved":0,"allocated":2179289088,"data_stored":2178791594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1676148736,"data_stored":1676148736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1929773056,"data_stored":1929773056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1669464064,"data_stored":1669464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1672740864,"data_stored":1672740864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1371799552,"data_stored":1371799552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2191982592,"data_stored":2191982592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1382547456,"data_stored":1382547456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1379336192,"data_stored":1379336192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:05:51.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-26T03:05:51.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:51.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:51.161 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:51.193 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:51.198 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:51.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:51.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:50 smithi192 ceph-mon[33110]: pgmap v302: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-26T03:05:51.290 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:51.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:50 smithi066 ceph-mon[29666]: pgmap v302: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-26T03:05:51.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:50 smithi066 ceph-mon[37920]: pgmap v302: 158 pgs: 158 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-26T03:05:51.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:51.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:51.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 45 16 76502 76486 106.205 68.5625 0.0110256 0.00941073 2024-03-26T03:05:51.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:51.578 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:51.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.729 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:51.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:51.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:51.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:51.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:51.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:51.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:52.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:52.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:52.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:52.241 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:52.286 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:52.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:52.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:52.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:52.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:52.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 46 16 77711 77695 105.539 75.5625 0.015529 0.00946949 2024-03-26T03:05:52.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:52.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:52.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:52.664 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:52.711 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:52.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:52.813 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:52.864 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:52.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:52.966 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:53.018 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:53.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:53.171 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:53.233 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:53 smithi066 ceph-mon[37920]: pgmap v303: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-26T03:05:53.233 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:53 smithi066 ceph-mon[29666]: pgmap v303: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-26T03:05:53.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:53.302 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:53.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:53.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:53 smithi192 ceph-mon[33110]: pgmap v303: 158 pgs: 158 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-26T03:05:53.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 47 16 78947 78931 104.937 77.25 0.00848044 0.0095246 2024-03-26T03:05:53.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:53.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:53.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:53.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:53.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:53.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:53.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:53.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:53.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:54.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:54.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:05:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:05:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:05:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:54.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:54.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:54.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:54.347 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:54.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:54.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:54.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 48 15 80199 80184 104.382 78.3125 0.0122317 0.00957504 2024-03-26T03:05:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:54.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:54.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:54.699 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:54.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:54.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:54.936 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:54.976 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:54.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:55.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:55.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:55.138 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:55.138 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:55.138 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:54 smithi066 ceph-mon[29666]: pgmap v304: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-26T03:05:55.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:55.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:55.138 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:54 smithi066 ceph-mon[37920]: pgmap v304: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-26T03:05:55.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:55.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:05:55.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:05:55.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:54 smithi192 ceph-mon[33110]: pgmap v304: 158 pgs: 158 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-26T03:05:55.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:55.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:55.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:55.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:55.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 49 16 81369 81353 103.743 73.0625 0.00956526 0.00963396 2024-03-26T03:05:55.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:55.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:55.539 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:05:55.539 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:05:55.583 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:05:55.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:55.625 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:05:55.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:55.663 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:05:55.703 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:05:55.745 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:05:55.755 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:55.788 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:05:55.794 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:05:55.794 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:05:55.794 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:05:55.794 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:05:55.794 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:05:55.794 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:05:55.795 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:05:55.795 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:05:55.801 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:55.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:55.948 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:55.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:56.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.100 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:56.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:56.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2473452893' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:56.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2473452893' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:56.201 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:56.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2473452893' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:05:56.257 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:56.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:56.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:56.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:56.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 50 16 82562 82546 103.159 74.5625 0.00679478 0.00968784 2024-03-26T03:05:56.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:56.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:56.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:56.664 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:05:56.753 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:56.804 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:56.906 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:56.954 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:56.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:57.002 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:05:57.056 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:57.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:57.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:56 smithi066 ceph-mon[29666]: pgmap v305: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-26T03:05:57.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/380491127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:57.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:56 smithi066 ceph-mon[37920]: pgmap v305: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-26T03:05:57.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/380491127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:57.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:57.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:57.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:57.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:57.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:56 smithi192 ceph-mon[33110]: pgmap v305: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-26T03:05:57.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/380491127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:05:57.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:57.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 51 16 83644 83628 102.463 67.625 0.00961925 0.00975463 2024-03-26T03:05:57.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:57.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:05:57.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:57.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:57.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:57.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:57.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:58.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:58.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:58.139 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:58.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:58.286 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:05:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:58.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:05:58.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:57 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/380491127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:58.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:57 smithi066 ceph-mon[29666]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T03:05:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:57 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/380491127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:58.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:57 smithi066 ceph-mon[37920]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T03:05:58.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:58.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:57 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:05:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/380491127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:05:58.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:57 smithi192 ceph-mon[33110]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T03:05:58.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 52 16 84879 84863 101.977 77.1875 0.0125644 0.00980107 2024-03-26T03:05:58.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:05:58.600 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:05:58.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:58.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:58.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:05:58.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:58.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:05:58.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:05:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.032 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:59.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:05:59.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:59.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:59.216 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:05:59 smithi066 ceph-mon[29666]: pgmap v307: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-26T03:05:59.216 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:05:59 smithi066 ceph-mon[37920]: pgmap v307: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-26T03:05:59.268 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:05:59.300 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:05:59 smithi192 ceph-mon[33110]: pgmap v307: 158 pgs: 158 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-26T03:05:59.361 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:05:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:05:59.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:05:59.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:05:59.510 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 53 16 86052 86036 101.436 73.3125 0.00804941 0.00985328 2024-03-26T03:05:59.543 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:05:59.543 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:05:59.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:59.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:59.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:05:59.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:05:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:05:59.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:05:59.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:05:59.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:05:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:05:59.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:00.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:00.049 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:00.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:00.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:00.203 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:00.203 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:00.287 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:00.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:00.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:00.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:00.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 54 16 87308 87292 101.011 78.5 0.00859402 0.00989423 2024-03-26T03:06:00.577 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.623 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:00.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:00.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:00.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:00.859 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:00.905 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:00.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:01.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:01.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:01.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:01.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:01.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:06:01.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:01.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:01.319 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:01 smithi066 ceph-mon[29666]: pgmap v308: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T03:06:01.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:01 smithi066 ceph-mon[37920]: pgmap v308: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T03:06:01.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:01.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:01.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:01 smithi192 ceph-mon[33110]: pgmap v308: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T03:06:01.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 55 16 88405 88389 100.421 68.5625 0.0140708 0.00995312 2024-03-26T03:06:01.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:01.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:01.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.715 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:01.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:01.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:01.862 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:01.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:01.994 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:01.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:02.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:02.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:02.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.259 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:02.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:02.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:06:02.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.408 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:02.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:02.509 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:02.512 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 56 16 89537 89521 99.8909 70.75 0.00303768 0.0100038 2024-03-26T03:06:02.537 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:02.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:02.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:02.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:02.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:02.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:02.979 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:02.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:03.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:03.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:03.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:03.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:03.346 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:03.387 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:03.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:03 smithi192 ceph-mon[33110]: pgmap v309: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T03:06:03.489 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:03.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 57 16 90603 90587 99.3073 66.625 0.00327285 0.0100579 2024-03-26T03:06:03.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:03 smithi066 ceph-mon[29666]: pgmap v309: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T03:06:03.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:03 smithi066 ceph-mon[37920]: pgmap v309: 158 pgs: 158 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-26T03:06:03.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:03.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:03.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:03.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:03.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:03.944 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:03.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:04.046 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:04.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:06:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:06:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:06:04.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:04.237 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:04.335 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:06:04.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:04.370 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:04.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:04.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:04.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 58 16 91798 91782 98.8828 74.6875 0.0239939 0.010108 2024-03-26T03:06:04.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:04.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:04.735 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:04.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:04.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:04.984 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:04.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:05.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:05.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:04 smithi066 ceph-mon[29666]: pgmap v310: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:06:05.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:04 smithi066 ceph-mon[37920]: pgmap v310: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:06:05.113 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:05.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:05.167 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:05.167 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:05.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:05.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:05.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:05.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:05.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:04 smithi192 ceph-mon[33110]: pgmap v310: 158 pgs: 158 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:06:05.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 59 16 92908 92892 98.3826 69.375 0.0110455 0.0101594 2024-03-26T03:06:05.516 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:05.618 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.671 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:05.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:05.772 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.826 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:05.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:05.927 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:05.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:05.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:06.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:06.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:06.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:06.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:06.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:06.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:06.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:06.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:06:06.511753+0000 min lat: 0.00133267 max lat: 0.217836 avg lat: 0.0102268 2024-03-26T03:06:06.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:06:06.511 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 60 15 93865 93850 97.7408 59.875 0.00582997 0.0102268 2024-03-26T03:06:06.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:06.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:06.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:06.747 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.793 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:06.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:06.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:06.944 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:06.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:07.046 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:07.095 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:07.147 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 1 pgp_num to 11 2024-03-26T03:06:07.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:07.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:07 smithi066 ceph-mon[29666]: pgmap v311: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T03:06:07.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/125855014' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:06:07.208 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:07 smithi066 ceph-mon[37920]: pgmap v311: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T03:06:07.208 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/125855014' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:06:07.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:07.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:07.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:07.467 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:07 smithi192 ceph-mon[33110]: pgmap v311: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T03:06:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/125855014' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:06:07.513 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 61 16 94982 94966 97.2786 69.75 0.00890172 0.0102748 2024-03-26T03:06:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.521 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:07.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:07.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.666 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:07.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:07.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:07.802 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:07.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:07.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:07.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:08.037 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:08.081 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:08.182 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:08.235 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:08.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:08.381 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:08.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:08.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:08 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/125855014' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T03:06:08.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:08 smithi192 ceph-mon[33110]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T03:06:08.513 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 62 15 96254 96239 96.9927 79.5625 0.0169082 0.0103055 2024-03-26T03:06:08.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:08.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:08 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/125855014' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T03:06:08.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:08 smithi066 ceph-mon[29666]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T03:06:08.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:08 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/125855014' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T03:06:08.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:08 smithi066 ceph-mon[37920]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T03:06:08.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:08.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:08.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:08.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:08.887 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:08.943 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:08.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:09.045 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:09.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:09.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:09.242 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:09.326 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:09 smithi066 ceph-mon[29666]: pgmap v313: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T03:06:09.326 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:09.326 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:09 smithi066 ceph-mon[37920]: pgmap v313: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T03:06:09.327 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:09.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:09.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:09.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:09 smithi192 ceph-mon[33110]: pgmap v313: 158 pgs: 158 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-26T03:06:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:09.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:06:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:06:09.514 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 63 16 97483 97467 96.6704 76.75 0.0219552 0.01034 2024-03-26T03:06:09.526 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:09.628 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:09.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:09.815 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:09.857 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:09.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:09.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:09.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:10.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:10.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:10.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:10.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:10.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:10.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:10.387 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:10.388 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:10.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:10.441 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:10.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:10.495 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:06:10.495 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:06:10.514 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 64 16 98648 98632 96.2976 72.8125 0.009232 0.0103796 2024-03-26T03:06:10.544 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:10.594 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:10.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:10.696 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:10.737 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:10.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:10.839 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:10.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:10.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:10.988 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:11.033 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:11.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:11.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:11.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:11.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:11.356 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:06:11.356 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:11 smithi066 ceph-mon[29666]: pgmap v314: 158 pgs: 158 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:06:11.357 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:11 smithi066 ceph-mon[37920]: pgmap v314: 158 pgs: 158 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:06:11.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:11.405 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:06:11.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:11.452 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:06:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:11 smithi192 ceph-mon[33110]: pgmap v314: 158 pgs: 158 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:06:11.501 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:06:11.514 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 65 16 99706 99690 95.8334 66.125 0.0304467 0.0104295 2024-03-26T03:06:11.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:11.547 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:06:11.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:11.588 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:06:11.628 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:06:11.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:11.671 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:06:11.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:06:11.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:06:11.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:06:11.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:06:11.679 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:06:11.679 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:06:11.679 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:06:11.679 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:06:11.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:11.798 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:11.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:11.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:11.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:11.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:12.088 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:12.165 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:12.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:12.189 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:06:12.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:12.323 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:12.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:12.425 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:12.477 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:12.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1185374815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3230332780' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:06:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:12 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:06:12.514 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 66 16 100780 100764 95.3984 67.125 0.0207929 0.0104774 2024-03-26T03:06:12.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1185374815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:12.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3230332780' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:06:12.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:12 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:06:12.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1185374815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:12.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3230332780' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:06:12.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:12 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:06:12.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:12.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:12.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:12.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:12.753 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:12.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:12.854 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:12.898 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:12.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:13.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:13.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:13.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:13.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:13.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.311 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:13.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:13.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:13.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:13.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:13 smithi192 ceph-mon[33110]: pgmap v315: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:06:13.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:13 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:13.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:13 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3230332780' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:06:13.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:13 smithi192 ceph-mon[33110]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T03:06:13.514 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 67 15 101914 101899 95.0332 70.9375 0.00777498 0.010518 2024-03-26T03:06:13.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:13.593 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[29666]: pgmap v315: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:06:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3230332780' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:06:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[29666]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T03:06:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[37920]: pgmap v315: 158 pgs: 158 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-26T03:06:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3230332780' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:06:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-mon[37920]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T03:06:13.696 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:13.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:13.838 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:13.880 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:13.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:13.982 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:14.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:06:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:06:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:06:14.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.172 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:14.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:14.227 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:06:14.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.331 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:14.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:14.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:14.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:14 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:06:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:14 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:06:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:14 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:06:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:14 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:06:14.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 68 16 103055 103039 94.6834 71.25 0.0114524 0.0105566 2024-03-26T03:06:14.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:14.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:06:14.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:06:14.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:06:14.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:06:14.567 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:06:14.567 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:06:14.567 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:06:14.567 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:14 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:06:14.591 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:06:14.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:14.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.763 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:14.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:14.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:14.931 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:14.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:15.033 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:15.118 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:15.138 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:15.172 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:15.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:15.267 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:15.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: 4.c scrub starts 2024-03-26T03:06:15.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: 4.c scrub ok 2024-03-26T03:06:15.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: pgmap v317: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-26T03:06:15.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: 2.d scrub starts 2024-03-26T03:06:15.290 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: 2.d scrub ok 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: 4.c scrub starts 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: 4.c scrub ok 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: pgmap v317: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: 2.d scrub starts 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: 2.d scrub ok 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:06:15.291 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:15 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:15.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:15.413 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: 4.c scrub starts 2024-03-26T03:06:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: 4.c scrub ok 2024-03-26T03:06:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: pgmap v317: 158 pgs: 158 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-26T03:06:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: 2.d scrub starts 2024-03-26T03:06:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: 2.d scrub ok 2024-03-26T03:06:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:06:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:06:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:15 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:15.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:15.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 69 16 104121 104105 94.2766 66.625 0.0068659 0.0106023 2024-03-26T03:06:15.559 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:15.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:15.703 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:15.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:15.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:15.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:15.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:06:15.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:15.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:06:16.089 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:16.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:16.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:16.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:16.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:16.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:16.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:16.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:16.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:16 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:06:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:16 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:06:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:16 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:06:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:16 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:06:16.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:16.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 70 16 105365 105349 94.0401 77.75 0.0120978 0.0106289 2024-03-26T03:06:16.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:06:16.553 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:16 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:06:16.553 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:16.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:16.655 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:16.720 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:16.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:16.823 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:16.831 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:06:16.831 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:06:16.832 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:06:16.832 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T03:06:16.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3 2024-03-26T03:06:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:16.872 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:16.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:16.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:17.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:17.031 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:06:17.079 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:16 smithi066 systemd[1]: Stopping Ceph osd.3 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:06:17.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:17.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:17.236 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.289 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:17.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: pgmap v318: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 1.4 scrub starts 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 1.4 scrub ok 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:06:17.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: pgmap v318: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 1.4 scrub starts 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 1.4 scrub ok 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:06:17.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:06:17.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:17 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:06:17.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:17.444 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: pgmap v318: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 1.4 scrub starts 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 1.4 scrub ok 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:06:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:17 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:06:17.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 71 16 106488 106472 93.7041 70.1875 0.0132279 0.0106673 2024-03-26T03:06:17.547 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.611 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:17.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:17.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:17.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:17.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:17.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:17.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:17.987 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.038 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:18.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:18.139 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.190 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:18.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:18.283 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:06:18.283 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:06:18.283 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:06:18.283 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:06:18.284 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:06:18.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:06:18.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T03:06:18.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:06:18.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:06:18.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:06:18.285 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:06:18.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:18.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:18.394 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:06:18.409 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:06:18.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:18.446 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:06:18.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:18.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:06:18.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 72 16 107580 107564 93.3505 68.25 0.0125011 0.0107073 2024-03-26T03:06:18.537 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:06:18.553 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:18.591 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:06:18.606 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:18.639 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:06:18.677 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:18.680 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:06:18.693 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:06:18.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:18.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:06:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:06:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:06:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:18 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:06:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:18.739 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:06:18.739 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:06:18.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:18.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:18.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:18.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:19.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:19.108 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[59389]: 2024-03-26T03:06:18.806+0000 7f5bef94b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:06:19.108 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[59389]: 2024-03-26T03:06:18.806+0000 7f5bef94b700 -1 osd.3 197 *** Got signal Terminated *** 2024-03-26T03:06:19.109 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:18 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[59389]: 2024-03-26T03:06:18.806+0000 7f5bef94b700 -1 osd.3 197 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:06:19.117 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:19.165 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:19.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:19.311 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:19.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:19.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:19.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 73 16 108444 108428 92.8114 54 0.00363018 0.0107352 2024-03-26T03:06:19.538 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:19.565 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: pgmap v320: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2693279449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: 3.1 scrub starts 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: 3.1 scrub ok 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[29666]: osd.3 marked itself down and dead 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: pgmap v320: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-26T03:06:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2693279449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: 3.1 scrub starts 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: 3.1 scrub ok 2024-03-26T03:06:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:19 smithi066 ceph-mon[37920]: osd.3 marked itself down and dead 2024-03-26T03:06:19.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:19.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:19.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:06:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: pgmap v320: 158 pgs: 158 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2693279449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: 3.1 scrub starts 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: 3.1 scrub ok 2024-03-26T03:06:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:19 smithi192 ceph-mon[33110]: osd.3 marked itself down and dead 2024-03-26T03:06:19.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:19.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:19.922 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:19.949 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:19.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:20.051 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:20.079 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:20.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:20.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:20.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:20.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:20.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:20.478 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: 6.6 scrub starts 2024-03-26T03:06:20.478 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: 6.6 scrub ok 2024-03-26T03:06:20.478 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: 6.4 scrub starts 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: 6.4 scrub ok 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: osdmap e198: 8 total, 7 up, 8 in 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[29666]: osdmap e199: 8 total, 7 up, 8 in 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: 6.6 scrub starts 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: 6.6 scrub ok 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: 6.4 scrub starts 2024-03-26T03:06:20.479 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: 6.4 scrub ok 2024-03-26T03:06:20.480 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T03:06:20.480 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: osdmap e198: 8 total, 7 up, 8 in 2024-03-26T03:06:20.480 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:20.480 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:20 smithi066 ceph-mon[37920]: osdmap e199: 8 total, 7 up, 8 in 2024-03-26T03:06:20.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:20.515 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 74 16 108733 108717 91.8014 18.0625 0.0118209 0.0108884 2024-03-26T03:06:20.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:20.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:20.728 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: 6.6 scrub starts 2024-03-26T03:06:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: 6.6 scrub ok 2024-03-26T03:06:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: 6.4 scrub starts 2024-03-26T03:06:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: 6.4 scrub ok 2024-03-26T03:06:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T03:06:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: osdmap e198: 8 total, 7 up, 8 in 2024-03-26T03:06:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:20 smithi192 ceph-mon[33110]: osdmap e199: 8 total, 7 up, 8 in 2024-03-26T03:06:20.766 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:20.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:20.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:20.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:21.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:21.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:21.145 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:21.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:21.286 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:21.325 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:21.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:21.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:21.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 75 16 110151 110135 91.759 88.625 0.00728804 0.0108937 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[29666]: pgmap v322: 158 pgs: 21 peering, 19 stale+active+clean, 118 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 908 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 1 pg inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[37920]: pgmap v322: 158 pgs: 21 peering, 19 stale+active+clean, 118 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 908 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 1 pg inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:06:21.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:21 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:06:21.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:21.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:06:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:21.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:21.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:21 smithi192 ceph-mon[33110]: pgmap v322: 158 pgs: 21 peering, 19 stale+active+clean, 118 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 908 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:06:21.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:21 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 1 pg inactive, 6 pgs peering (PG_AVAILABILITY) 2024-03-26T03:06:21.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:21 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:06:21.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:21 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:06:21.707 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.748 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:21.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:21.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:06:21.812 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:21 smithi066 podman[108069]: 2024-03-26 03:06:21.611054236 +0000 UTC m=+3.679964637 container died 29770e69880e7c4989c3167ea35eff53c99bf58d8a8907bdbb04765314553e72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T03:06:21.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:21.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:21.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:22.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:22.051 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:22.059 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:22.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:22.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:22.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:22.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:22.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:22.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:22 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:06:22.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:22 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:06:22.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:22 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:06:22.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:22 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:06:22.482 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:22 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:06:22.483 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:22.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 76 16 111477 111461 91.642 82.875 0.015666 0.0109071 2024-03-26T03:06:22.585 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:22.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:06:22.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:06:22.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:06:22.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:06:22.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:06:22.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:06:22.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:06:22.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:06:22.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:06:22.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:22 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:06:22.611 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:22 smithi066 podman[108069]: 2024-03-26 03:06:22.12453016 +0000 UTC m=+4.193440543 container cleanup 29770e69880e7c4989c3167ea35eff53c99bf58d8a8907bdbb04765314553e72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:06:22.611 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:22 smithi066 bash[108069]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3 2024-03-26T03:06:22.629 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:22.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:22.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:22.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:22.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:22.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:23.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:23.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:23.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:23.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:23.280 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.328 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:23.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:23.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:23.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:23.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 77 16 112843 112827 91.5606 85.375 0.0054335 0.010917 2024-03-26T03:06:23.536 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:23 smithi066 podman[108297]: 2024-03-26 03:06:23.241711591 +0000 UTC m=+3.369792389 container remove 29770e69880e7c4989c3167ea35eff53c99bf58d8a8907bdbb04765314553e72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[29666]: pgmap v324: 158 pgs: 32 peering, 15 stale+active+clean, 111 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 341 B/s rd, 49 MiB/s wr, 777 op/s 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:06:23.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[37920]: pgmap v324: 158 pgs: 32 peering, 15 stale+active+clean, 111 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 341 B/s rd, 49 MiB/s wr, 777 op/s 2024-03-26T03:06:23.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:06:23.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:06:23.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:23.600 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:23 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:06:23.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:23 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:06:23.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:23 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:06:23.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:23 smithi192 ceph-mon[33110]: pgmap v324: 158 pgs: 32 peering, 15 stale+active+clean, 111 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 341 B/s rd, 49 MiB/s wr, 777 op/s 2024-03-26T03:06:23.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:23 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:06:23.601 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:23 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:06:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.603 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:23.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:23.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:23.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:23.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:23.887 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:23.895 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:23.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:23.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:24.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:24.074 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:06:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:06:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:06:24.113 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:06:24.113 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:06:24.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:24.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:24.285 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:24.327 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:06:24.327 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:06:24.327 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:24.327 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:24.327 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:06:24.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:06:24.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:24.328 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:24.330 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:24.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:24.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:24.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 78 16 114022 114006 91.3314 73.6875 0.0144986 0.0109445 2024-03-26T03:06:24.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:24.605 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:24.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:24 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:06:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:24 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:06:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:24.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:24.848 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:24.876 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:24.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:24.978 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:25.022 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:25.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:25.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:25.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:25.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:25.331 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:06:25.358 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:25 smithi066 podman[108727]: 2024-03-26 03:06:25.013957069 +0000 UTC m=+0.665061857 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[29666]: pgmap v325: 158 pgs: 14 active+undersized+degraded, 32 peering, 20 active+undersized, 92 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 21209/333261 objects degraded (6.364%); 32 KiB/s, 0 objects/s recovering 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3985230180' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[29666]: from='client.25036 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[29666]: Health check failed: Degraded data redundancy: 21209/333261 objects degraded (6.364%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:06:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[37920]: pgmap v325: 158 pgs: 14 active+undersized+degraded, 32 peering, 20 active+undersized, 92 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 21209/333261 objects degraded (6.364%); 32 KiB/s, 0 objects/s recovering 2024-03-26T03:06:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:06:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3985230180' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[37920]: from='client.25036 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:25 smithi066 ceph-mon[37920]: Health check failed: Degraded data redundancy: 21209/333261 objects degraded (6.364%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:25.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:25.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:25.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 79 16 115026 115010 90.9696 62.75 0.0104393 0.0109881 2024-03-26T03:06:25.525 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:25.568 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:25.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:25.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:25 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:06:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:25 smithi192 ceph-mon[33110]: pgmap v325: 158 pgs: 14 active+undersized+degraded, 32 peering, 20 active+undersized, 92 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 21209/333261 objects degraded (6.364%); 32 KiB/s, 0 objects/s recovering 2024-03-26T03:06:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:25 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:06:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3985230180' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:25 smithi192 ceph-mon[33110]: from='client.25036 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:25 smithi192 ceph-mon[33110]: Health check failed: Degraded data redundancy: 21209/333261 objects degraded (6.364%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:25.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:25.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:25.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:25.958 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:26.008 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:26.108 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:25 smithi066 podman[108727]: 2024-03-26 03:06:25.969874614 +0000 UTC m=+1.620979436 container create 551450103b0e18047300bcfda843b6e6a4c3c8bf21233c2dbafd211366ce7251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T03:06:26.110 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:26.162 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:26.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:26.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:06:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:26.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:26.376 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:26 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:06:26.376 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:26 smithi066 ceph-mon[29666]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:06:26.376 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:26 smithi066 ceph-mon[29666]: osdmap e200: 8 total, 7 up, 8 in 2024-03-26T03:06:26.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:26 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:06:26.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:26 smithi066 ceph-mon[37920]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:06:26.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:26 smithi066 ceph-mon[37920]: osdmap e200: 8 total, 7 up, 8 in 2024-03-26T03:06:26.377 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:26 smithi066 podman[108727]: 2024-03-26 03:06:26.283428752 +0000 UTC m=+1.934533544 container init 551450103b0e18047300bcfda843b6e6a4c3c8bf21233c2dbafd211366ce7251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T03:06:26.377 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:26 smithi066 podman[108727]: 2024-03-26 03:06:26.289069383 +0000 UTC m=+1.940174165 container start 551450103b0e18047300bcfda843b6e6a4c3c8bf21233c2dbafd211366ce7251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T03:06:26.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:26 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:06:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:26 smithi192 ceph-mon[33110]: from='client.25036 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:06:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:26 smithi192 ceph-mon[33110]: osdmap e200: 8 total, 7 up, 8 in 2024-03-26T03:06:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.498 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:26.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:26.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:06:26.516598+0000 min lat: 0.00133267 max lat: 1.09198 avg lat: 0.0110088 2024-03-26T03:06:26.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:06:26.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 80 16 116262 116246 90.798 77.25 0.0126003 0.0110088 2024-03-26T03:06:26.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:26.620 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:06:26.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:26.683 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:06:26.686 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:26.729 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:06:26.754 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:26.781 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:06:26.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.808 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:26.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:26.827 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:06:26.863 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:06:26.902 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:06:26.903 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:06:26.909 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:26.954 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:26.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:27.044 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:26 smithi066 conmon[108869]: conmon 551450103b0e18047300 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-551450103b0e18047300bcfda843b6e6a4c3c8bf21233c2dbafd211366ce7251.scope/container/memory.events 2024-03-26T03:06:27.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:27.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:27.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:27.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:27.306 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:27 smithi066 podman[108727]: 2024-03-26 03:06:27.043932907 +0000 UTC m=+2.695037688 container attach 551450103b0e18047300bcfda843b6e6a4c3c8bf21233c2dbafd211366ce7251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:06:27.306 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:06:27 smithi066 podman[108727]: 2024-03-26 03:06:27.044356904 +0000 UTC m=+2.695461700 container died 551450103b0e18047300bcfda843b6e6a4c3c8bf21233c2dbafd211366ce7251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T03:06:27.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:27.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:27.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: pgmap v327: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.0 KiB/s rd, 79 MiB/s wr, 1.28k op/s; 42723/341991 objects degraded (12.492%); 32 KiB/s, 0 objects/s recovering 2024-03-26T03:06:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:06:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:06:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-03-26T03:06:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1497123698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: 2.1d deep-scrub starts 2024-03-26T03:06:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: 2.1d deep-scrub ok 2024-03-26T03:06:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:06:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:27 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:06:27.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 81 16 117481 117465 90.6176 76.1875 0.00469471 0.0110307 2024-03-26T03:06:27.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: pgmap v327: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.0 KiB/s rd, 79 MiB/s wr, 1.28k op/s; 42723/341991 objects degraded (12.492%); 32 KiB/s, 0 objects/s recovering 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1497123698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: 2.1d deep-scrub starts 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: 2.1d deep-scrub ok 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: pgmap v327: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.0 KiB/s rd, 79 MiB/s wr, 1.28k op/s; 42723/341991 objects degraded (12.492%); 32 KiB/s, 0 objects/s recovering 2024-03-26T03:06:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1497123698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: 2.1d deep-scrub starts 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: 2.1d deep-scrub ok 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:06:27.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:27 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:06:27.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:27.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:27.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:27.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:27.807 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T03:06:27.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:27.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:06:27.894 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T03:06:27.894 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-03-26T03:06:27.894 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi066.front.sepia.ceph.com 2024-03-26T03:06:27.895 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.3 --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-03-26T03:06:27.932 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:27.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:28.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:28.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:28.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.229 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:28.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:28.330 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.370 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:28.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:28.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:28 smithi192 ceph-mon[33110]: 8.6 scrub starts 2024-03-26T03:06:28.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:28 smithi192 ceph-mon[33110]: osdmap e201: 8 total, 7 up, 8 in 2024-03-26T03:06:28.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:28.484 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:28 smithi066 ceph-mon[29666]: 8.6 scrub starts 2024-03-26T03:06:28.484 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:28 smithi066 ceph-mon[29666]: osdmap e201: 8 total, 7 up, 8 in 2024-03-26T03:06:28.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:28 smithi066 ceph-mon[37920]: 8.6 scrub starts 2024-03-26T03:06:28.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:28 smithi066 ceph-mon[37920]: osdmap e201: 8 total, 7 up, 8 in 2024-03-26T03:06:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:28.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 82 16 118754 118738 90.4827 79.5625 0.0142028 0.0110468 2024-03-26T03:06:28.612 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:28.638 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:06:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:06:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:06:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:06:28.638 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:06:28.659 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:28.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:28.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:28.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:28.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:28.981 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/osd.3/config 2024-03-26T03:06:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:06:29.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:06:29.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:29.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:29.163 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:29.202 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:29.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:29.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:29.443 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:29.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:29.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 83 16 120017 120001 90.3435 78.9375 0.00646049 0.0110642 2024-03-26T03:06:29.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:06:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:06:29.585 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:29 smithi192 ceph-mon[33110]: pgmap v329: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.1 KiB/s rd, 49 MiB/s wr, 786 op/s; 42723/341991 objects degraded (12.492%); 28 KiB/s, 0 objects/s recovering 2024-03-26T03:06:29.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:29 smithi192 ceph-mon[33110]: 1.3 deep-scrub starts 2024-03-26T03:06:29.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:29 smithi192 ceph-mon[33110]: 1.3 deep-scrub ok 2024-03-26T03:06:29.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:29 smithi192 ceph-mon[33110]: 2.16 deep-scrub starts 2024-03-26T03:06:29.586 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:29 smithi192 ceph-mon[33110]: 2.16 deep-scrub ok 2024-03-26T03:06:29.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[29666]: pgmap v329: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.1 KiB/s rd, 49 MiB/s wr, 786 op/s; 42723/341991 objects degraded (12.492%); 28 KiB/s, 0 objects/s recovering 2024-03-26T03:06:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[29666]: 1.3 deep-scrub starts 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[29666]: 1.3 deep-scrub ok 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[29666]: 2.16 deep-scrub starts 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[29666]: 2.16 deep-scrub ok 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[37920]: pgmap v329: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.1 KiB/s rd, 49 MiB/s wr, 786 op/s; 42723/341991 objects degraded (12.492%); 28 KiB/s, 0 objects/s recovering 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[37920]: 1.3 deep-scrub starts 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[37920]: 1.3 deep-scrub ok 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[37920]: 2.16 deep-scrub starts 2024-03-26T03:06:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:29 smithi066 ceph-mon[37920]: 2.16 deep-scrub ok 2024-03-26T03:06:29.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:06:29.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:29.743 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:29.793 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:29.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:29.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:29.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:30.050 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:30.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:30.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:30.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:30.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:30.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:30.380 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:30.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:30.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:30 smithi192 ceph-mon[33110]: 8.6 scrub ok 2024-03-26T03:06:30.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:30 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:06:30.481 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:30 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:06:30.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:30.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 84 16 121338 121322 90.2509 82.5625 0.0109128 0.0110753 2024-03-26T03:06:30.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:30.533 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:06:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[29666]: 8.6 scrub ok 2024-03-26T03:06:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:06:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:06:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[37920]: 8.6 scrub ok 2024-03-26T03:06:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:06:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:30 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:06:30.626 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.675 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:30.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:30.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:30.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:30.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:30.951 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:30.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:30.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:31.026 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:06:31.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:31.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:31.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.242 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:31.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:31.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.372 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:31.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:31.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: pgmap v330: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.6 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 44645/356511 objects degraded (12.523%); 28 KiB/s, 0 objects/s recovering 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 44645/356511 objects degraded (12.523%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:06:31.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:31 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:06:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:31.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:31.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 85 16 122531 122515 90.0663 74.5625 0.0242761 0.0110978 2024-03-26T03:06:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: pgmap v330: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.6 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 44645/356511 objects degraded (12.523%); 28 KiB/s, 0 objects/s recovering 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 44645/356511 objects degraded (12.523%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: pgmap v330: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.6 KiB/s rd, 85 MiB/s wr, 1.37k op/s; 44645/356511 objects degraded (12.523%); 28 KiB/s, 0 objects/s recovering 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:06:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:06:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 44645/356511 objects degraded (12.523%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:06:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:06:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:06:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:31 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:06:31.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:31.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:31.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:31.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:31.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:31.918 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28229 7.1 7:81134404:::benchmark_data_smithi066_76917_object28228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.292280+0000", 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 316.61127577100001, 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.110167232, 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.919 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28229 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292280+0000", 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292280+0000", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292282+0000", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292321+0000", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8554999999999999e-05 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292344+0000", 2024-03-26T03:06:31.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2906999999999999e-05 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292350+0000", 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1510000000000001e-06 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292408+0000", 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8455999999999997e-05 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292540+0000", 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131292 2024-03-26T03:06:31.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292689+0000", 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014918900000000001 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295453+0000", 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027638609999999998 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295465+0000", 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2139e-05 2024-03-26T03:06:31.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402380+0000", 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.106915063 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402418+0000", 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7787999999999998e-05 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402447+0000", 2024-03-26T03:06:31.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9796999999999999e-05 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28233 7.1 7:88f223fb:::benchmark_data_smithi066_76917_object28232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 316.61118445400001, 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11016767, 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28233 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292372+0000", 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6199999999999996e-07 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292414+0000", 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-05 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292422+0000", 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8599999999999993e-06 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292428+0000", 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2960000000000004e-06 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.293764+0000", 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001336215 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.293895+0000", 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013044700000000001 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.294096+0000", 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020120000000000001 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295610+0000", 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015145429999999999 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.296215+0000", 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060502799999999997 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402479+0000", 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10626340099999999 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402514+0000", 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4782e-05 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.402539+0000", 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5636e-05 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28781 7.1 7:87d55f3c:::benchmark_data_smithi066_76917_object28780:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.644242+0000", 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 316.25931395100002, 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.142996018, 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28781 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644242+0000", 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644242+0000", 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644244+0000", 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.232e-06 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644294+0000", 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9721e-05 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644314+0000", 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0225999999999999e-05 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644324+0000", 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4550000000000001e-06 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644401+0000", 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7171999999999999e-05 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644519+0000", 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118093 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644644+0000", 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012525399999999999 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645900+0000", 2024-03-26T03:06:31.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012556819999999999 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.646099+0000", 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019884899999999999 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787137+0000", 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14103827899999999 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787191+0000", 2024-03-26T03:06:31.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4150999999999997e-05 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787238+0000", 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6903999999999997e-05 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:29443 7.1 7:8992cdbb:::benchmark_data_smithi066_76917_object29442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.091655+0000", 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 315.81190132099999, 2024-03-26T03:06:31.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17979831600000001, 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 29443 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091655+0000", 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091655+0000", 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091657+0000", 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9319999999999999e-06 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091723+0000", 2024-03-26T03:06:31.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6415999999999996e-05 2024-03-26T03:06:31.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091736+0000", 2024-03-26T03:06:31.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2965e-05 2024-03-26T03:06:31.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091744+0000", 2024-03-26T03:06:31.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8890000000000002e-06 2024-03-26T03:06:31.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091809+0000", 2024-03-26T03:06:31.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5334999999999997e-05 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.091930+0000", 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120327 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.092053+0000", 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012322900000000001 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.093050+0000", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00099677399999999997 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.093374+0000", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032402400000000001 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.271342+0000", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17796842700000001 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.271409+0000", 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7268999999999996e-05 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.271453+0000", 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3729000000000001e-05 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:29453 7.1 7:8ca24ebb:::benchmark_data_smithi066_76917_object29452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.101594+0000", 2024-03-26T03:06:31.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 315.80196246000003, 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17664884, 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 29453 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101594+0000", 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101594+0000", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101595+0000", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.796e-06 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101653+0000", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7182999999999998e-05 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101663+0000", 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.062e-05 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101669+0000", 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6049999999999997e-06 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101722+0000", 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2827000000000001e-05 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101823+0000", 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010179 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.101919+0000", 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5328000000000006e-05 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.102836+0000", 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00091741399999999997 2024-03-26T03:06:31.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.102981+0000", 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014494200000000001 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.278161+0000", 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.175180164 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.278212+0000", 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0963e-05 2024-03-26T03:06:31.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.278242+0000", 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0207999999999999e-05 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30122 7.1 7:8b71f513:::benchmark_data_smithi066_76917_object30121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.589752+0000", 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 315.31380400699999, 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.163766524, 2024-03-26T03:06:31.952 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30122 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589752+0000", 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589752+0000", 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589754+0000", 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9290000000000001e-06 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589808+0000", 2024-03-26T03:06:31.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.359e-05 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589823+0000", 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5143000000000001e-05 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589831+0000", 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4370000000000007e-06 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.589891+0000", 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9944000000000001e-05 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.590029+0000", 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013794100000000001 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.590172+0000", 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014341199999999999 2024-03-26T03:06:31.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591392+0000", 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001219176 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.591642+0000", 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025028600000000001 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.753429+0000", 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16178671 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.753472+0000", 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3648e-05 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.753519+0000", 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6307999999999998e-05 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.958 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30127 7.1 7:808933d7:::benchmark_data_smithi066_76917_object30126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.592242+0000", 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 315.31131359699998, 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16545906399999999, 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30127 2024-03-26T03:06:31.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592242+0000", 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592242+0000", 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592244+0000", 2024-03-26T03:06:31.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7880000000000001e-06 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592286+0000", 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1542999999999998e-05 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592299+0000", 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2835e-05 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592307+0000", 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6899999999999998e-06 2024-03-26T03:06:31.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592365+0000", 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7985000000000001e-05 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592486+0000", 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120586 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.592668+0000", 2024-03-26T03:06:31.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018237200000000001 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.595129+0000", 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024603810000000002 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.595628+0000", 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049977500000000005 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757605+0000", 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.161976328 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757662+0000", 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7012999999999998e-05 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.757701+0000", 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9768000000000003e-05 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31273 7.1 7:8ded8593:::benchmark_data_smithi066_76917_object31272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 314.169975497, 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14420369, 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31273 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733581+0000", 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9999999999999996e-07 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733615+0000", 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3147e-05 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733623+0000", 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7309999999999999e-06 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.733631+0000", 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9680000000000006e-06 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.734918+0000", 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012864090000000001 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.735011+0000", 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3194000000000003e-05 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.735122+0000", 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000110952 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.736425+0000", 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001302896 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.736477+0000", 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2564e-05 2024-03-26T03:06:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.877677+0000", 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14120012800000001 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.877738+0000", 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0130000000000002e-05 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.877784+0000", 2024-03-26T03:06:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6671000000000001e-05 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31280 7.1 7:8c9719fd:::benchmark_data_smithi066_76917_object31279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.738079+0000", 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 314.165477409, 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14374624499999999, 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31280 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738079+0000", 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738079+0000", 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738080+0000", 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3820000000000001e-06 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738152+0000", 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1687999999999995e-05 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738165+0000", 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3115000000000001e-05 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738173+0000", 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5790000000000004e-06 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738216+0000", 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2929000000000002e-05 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738340+0000", 2024-03-26T03:06:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:31.976 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:31.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012344499999999999 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738519+0000", 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017883499999999999 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.739797+0000", 2024-03-26T03:06:31.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012786220000000001 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.740145+0000", 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034736400000000002 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881757+0000", 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14161243200000001 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881798+0000", 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0618999999999999e-05 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881825+0000", 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7234999999999999e-05 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31288 7.1 7:899370f3:::benchmark_data_smithi066_76917_object31287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.741047+0000", 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 314.16250924399998, 2024-03-26T03:06:31.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.140931367, 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31288 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741047+0000", 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741047+0000", 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741049+0000", 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9080000000000002e-06 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741107+0000", 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8177999999999998e-05 2024-03-26T03:06:31.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741123+0000", 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6472000000000001e-05 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741132+0000", 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4079999999999998e-06 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741193+0000", 2024-03-26T03:06:31.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1387e-05 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741314+0000", 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120697 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.741445+0000", 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131128 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.742377+0000", 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00093240100000000002 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.742544+0000", 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016628199999999999 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881873+0000", 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.139329279 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881928+0000", 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5538000000000003e-05 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.881978+0000", 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9688999999999998e-05 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:70253 7.1 7:879a87ed:::benchmark_data_smithi066_76917_object70252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T03:06:31.985 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.078364+0000", 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.82519227300003, 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.127659565, 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 70253 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078364+0000", 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078364+0000", 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078366+0000", 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3089999999999998e-06 2024-03-26T03:06:31.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078416+0000", 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0214999999999999e-05 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078427+0000", 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0276e-05 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078432+0000", 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7209999999999999e-06 2024-03-26T03:06:31.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078500+0000", 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7341999999999993e-05 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078629+0000", 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012928200000000001 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.078766+0000", 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137179 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.082939+0000", 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041725970000000001 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.085752+0000", 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028134219999999999 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205898+0000", 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.120146137 2024-03-26T03:06:31.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205958+0000", 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9562999999999999e-05 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206023+0000", 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5522e-05 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:70256 7.1 7:80dfd34f:::benchmark_data_smithi066_76917_object70255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T03:06:31.991 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.079991+0000", 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.823564858, 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.129241039, 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 70256 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.992 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.079991+0000", 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.079991+0000", 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:31.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.079993+0000", 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.604e-06 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.080022+0000", 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9312e-05 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.080034+0000", 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1834000000000001e-05 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.080042+0000", 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9130000000000003e-06 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.081063+0000", 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010207129999999999 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.081186+0000", 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000123743 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.081305+0000", 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011904999999999999 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.083049+0000", 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001744128 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205854+0000", 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.122804116 2024-03-26T03:06:31.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.209083+0000", 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003229043 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.209186+0000", 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103151 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.209232+0000", 2024-03-26T03:06:31.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6431999999999999e-05 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:70259 7.9 e107/96 7:9c43e8b1:::benchmark_data_smithi066_76917_object70258:head v 107'4278, mlcod=107'4276)", 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.088001+0000", 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.81555465399998, 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13058771799999999, 2024-03-26T03:06:31.998 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088001+0000", 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088001+0000", 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:31.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088003+0000", 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.556e-06 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088044+0000", 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1539999999999999e-05 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088054+0000", 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0030999999999999e-05 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088062+0000", 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3019999999999997e-06 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206537+0000", 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.118474827 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206559+0000", 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2141e-05 2024-03-26T03:06:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.218546+0000", 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011987506 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.218589+0000", 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2815000000000003e-05 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:70266 7.3 e107/96 7:cc21e419:::benchmark_data_smithi066_76917_object70265:head v 107'4406, mlcod=107'4404)", 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:45.088266+0000", 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.815289866, 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.122352424, 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088266+0000", 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088266+0000", 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088267+0000", 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.048e-06 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.088328+0000", 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0368000000000002e-05 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205571+0000", 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11724303799999999 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.205587+0000", 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5962999999999999e-05 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206072+0000", 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00048576000000000001 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.206093+0000", 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0253000000000001e-05 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.210542+0000", 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044494640000000002 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:45.210619+0000", 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6530000000000001e-05 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:75520 7.8 e110/96)", 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.269554+0000", 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 281.63400197599998, 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.120205865, 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269554+0000", 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269554+0000", 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269564+0000", 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0151e-05 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269623+0000", 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8856e-05 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269637+0000", 2024-03-26T03:06:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3495e-05 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.269647+0000", 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0515e-05 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.389740+0000", 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12009255100000001 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.389760+0000", 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0296999999999999e-05 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.14841.0:75525 7.4 e110/96)", 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.276113+0000", 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 281.62744318599999, 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11648288499999999, 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276113+0000", 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276113+0000", 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276114+0000", 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6599999999999994e-07 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276151+0000", 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7635000000000001e-05 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276159+0000", 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3300000000000001e-06 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.276164+0000", 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0459999999999996e-06 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.392574+0000", 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.116410069 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.392596+0000", 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1838999999999999e-05 2024-03-26T03:06:32.014 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75533 7.11 7:884ac876:::benchmark_data_smithi066_76917_object75532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.320059+0000", 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 281.58349677699999, 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.028185074, 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:32.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75533 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320059+0000", 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320059+0000", 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320063+0000", 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8959999999999996e-06 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320130+0000", 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6713999999999999e-05 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320161+0000", 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0799000000000001e-05 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.320176+0000", 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5477000000000001e-05 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.393808+0000", 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.073632049000000005 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.393820+0000", 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2065999999999999e-05 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.340179+0000", 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94635899000000001 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.340309+0000", 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013020100000000001 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.340475+0000", 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016509200000000001 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.341934+0000", 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014593360000000001 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.342178+0000", 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024444700000000001 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.348143+0000", 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059647160000000001 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.348201+0000", 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7819000000000003e-05 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.348244+0000", 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3472e-05 2024-03-26T03:06:32.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:111993 7.1 7:83b0452a:::benchmark_data_smithi066_76917_object11810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:50.996705+0000", 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 220.90685095000001, 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13555013299999999, 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:32.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111993 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996705+0000", 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996705+0000", 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996708+0000", 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4880000000000001e-06 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996714+0000", 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7240000000000003e-06 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996730+0000", 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5498e-05 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996739+0000", 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4779999999999997e-06 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996813+0000", 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3743000000000002e-05 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.996948+0000", 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000135227 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.997298+0000", 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034946199999999998 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.999128+0000", 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001829883 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.999771+0000", 2024-03-26T03:06:32.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064344100000000002 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.132141+0000", 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13236953600000001 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.132203+0000", 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2395000000000001e-05 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.132255+0000", 2024-03-26T03:06:32.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2258e-05 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112710 7.16 7:6af1cf5d:::benchmark_data_smithi066_76917_object12527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:52.511950+0000", 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 219.39160601699999, 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.161410046, 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:32.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112710 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511950+0000", 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511950+0000", 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511952+0000", 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7659999999999999e-06 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511957+0000", 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3689999999999998e-06 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511980+0000", 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3217000000000001e-05 2024-03-26T03:06:32.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.511989+0000", 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7260000000000008e-06 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.512069+0000", 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9432999999999997e-05 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.512188+0000", 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119064 2024-03-26T03:06:32.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.512531+0000", 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034361400000000001 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.514375+0000", 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001843337 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.514542+0000", 2024-03-26T03:06:32.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016794499999999999 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.673196+0000", 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.15865315499999999 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.673288+0000", 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2441000000000005e-05 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.673360+0000", 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1978999999999996e-05 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:33128 8.2 8:4016a61d:::benchmark_data_smithi066_100612_object33127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:05:21.036581+0000", 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 70.866974572000004, 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13672023699999999, 2024-03-26T03:06:32.034 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 33128 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036581+0000", 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036581+0000", 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036584+0000", 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2790000000000001e-06 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036630+0000", 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6088999999999999e-05 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036643+0000", 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3645999999999999e-05 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036651+0000", 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5039999999999996e-06 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036711+0000", 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0071999999999997e-05 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:32.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036830+0000", 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011924199999999999 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.036958+0000", 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012736900000000001 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.038120+0000", 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001162618 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.038281+0000", 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016061100000000001 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.173198+0000", 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13491710800000001 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.173257+0000", 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9123999999999998e-05 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:32.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:21.173302+0000", 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4574999999999999e-05 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:32.041 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:06:32.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:32.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:32.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:32.276 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:32.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:32.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:32.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 86 16 123636 123620 89.822 69.0625 0.0102119 0.0111265 2024-03-26T03:06:32.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:32.558 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:32 smithi066 ceph-mon[29666]: 8.e scrub starts 2024-03-26T03:06:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:32 smithi066 ceph-mon[29666]: 2.6 deep-scrub starts 2024-03-26T03:06:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:32 smithi066 ceph-mon[29666]: 2.6 deep-scrub ok 2024-03-26T03:06:32.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:32 smithi066 ceph-mon[37920]: 8.e scrub starts 2024-03-26T03:06:32.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:32 smithi066 ceph-mon[37920]: 2.6 deep-scrub starts 2024-03-26T03:06:32.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:32 smithi066 ceph-mon[37920]: 2.6 deep-scrub ok 2024-03-26T03:06:32.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:32.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:32 smithi192 ceph-mon[33110]: 8.e scrub starts 2024-03-26T03:06:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:32 smithi192 ceph-mon[33110]: 2.6 deep-scrub starts 2024-03-26T03:06:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:32 smithi192 ceph-mon[33110]: 2.6 deep-scrub ok 2024-03-26T03:06:32.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:32.797 INFO:teuthology.orchestra.run.smithi066.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:06:32.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:32.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:06:32.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:32.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:32.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:33.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:33.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:33.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:33.236 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:33.264 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:33.271 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:33.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:33.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:33.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:33.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:33.440 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:33.497 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:33.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 87 16 124600 124584 89.4821 60.25 0.00422478 0.0111674 2024-03-26T03:06:33.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:33.578 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:33.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:33.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:06:33.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:06:33.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:06:33.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:06:33.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[37920]: pgmap v331: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 45012/360033 objects degraded (12.502%) 2024-03-26T03:06:33.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[37920]: 8.3 deep-scrub starts 2024-03-26T03:06:33.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:06:33.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:06:33.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:06:33.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:06:33.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[29666]: pgmap v331: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 45012/360033 objects degraded (12.502%) 2024-03-26T03:06:33.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-mon[29666]: 8.3 deep-scrub starts 2024-03-26T03:06:33.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:33 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:06:33.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:33 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:06:33.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:33 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:06:33.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:33 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:06:33.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:33 smithi192 ceph-mon[33110]: pgmap v331: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 45012/360033 objects degraded (12.502%) 2024-03-26T03:06:33.663 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:33 smithi192 ceph-mon[33110]: 8.3 deep-scrub starts 2024-03-26T03:06:33.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:33.722 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:33.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:33.823 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:33.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:06:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:06:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:06:33.867 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:33.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:33.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:34.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:34.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:34.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:34.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:34.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:34.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:34.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:34.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 88 16 125658 125642 89.2166 66.125 0.075549 0.0112026 2024-03-26T03:06:34.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:34.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:06:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:06:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 8.e scrub ok 2024-03-26T03:06:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 8.e scrub ok 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:06:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:34 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:06:34.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.711 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:34.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:06:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:06:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 8.e scrub ok 2024-03-26T03:06:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:06:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:06:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:06:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:34 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:06:34.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:34.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:34.947 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:34.990 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:34.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:35.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:35.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:35.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:35.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:35.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:35.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:35 smithi192 ceph-mon[33110]: 8.9 scrub starts 2024-03-26T03:06:35.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:35 smithi192 ceph-mon[33110]: pgmap v332: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 584 B/s rd, 64 MiB/s wr, 1.02k op/s; 46095/368712 objects degraded (12.502%) 2024-03-26T03:06:35.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:35 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:06:35.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45866 8.3 e201/172 8:c160dbdf:::benchmark_data_smithi066_100612_object6521:head v 201'8369, mlcod=201'8358)", 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.465283+0000", 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017833359999999999, 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017865369999999998, 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.488 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465283+0000", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465283+0000", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465284+0000", 2024-03-26T03:06:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1680000000000001e-06 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465291+0000", 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2150000000000004e-06 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465300+0000", 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5259999999999993e-06 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465306+0000", 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.742e-06 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.466797+0000", 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014900250000000001 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.466815+0000", 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8839e-05 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.491 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45867 8.3 e201/172 8:c162188e:::benchmark_data_smithi066_100612_object44734:head v 201'8370, mlcod=201'8358)", 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.465392+0000", 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017724272999999999, 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017831973000000001, 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465392+0000", 2024-03-26T03:06:35.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465392+0000", 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465393+0000", 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7000000000000004e-07 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465398+0000", 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465405+0000", 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1620000000000002e-06 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.465411+0000", 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5430000000000001e-06 2024-03-26T03:06:35.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.469162+0000", 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003751599 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.469180+0000", 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7992e-05 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45868 8.3 e201/172 8:c162f986:::benchmark_data_smithi066_100612_object26922:head v 201'8371, mlcod=201'8358)", 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.467189+0000", 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015926872000000002, 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016103517000000001, 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.467189+0000", 2024-03-26T03:06:35.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.467189+0000", 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.467190+0000", 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0610000000000001e-06 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.467197+0000", 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0500000000000003e-06 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.467207+0000", 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3530000000000001e-06 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.467214+0000", 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9530000000000002e-06 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.471403+0000", 2024-03-26T03:06:35.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041891150000000002 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.471421+0000", 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8264999999999999e-05 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45870 8.3 e201/172 8:c1630004:::benchmark_data_smithi066_100612_object91160:head v 201'8372, mlcod=201'8358)", 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.470004+0000", 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013112093, 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013359459000000001, 2024-03-26T03:06:35.499 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.470004+0000", 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.470004+0000", 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.470005+0000", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.268e-06 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.470013+0000", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3000000000000004e-06 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.470022+0000", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8820000000000007e-06 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.470029+0000", 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1389999999999998e-06 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474166+0000", 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041371480000000002 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474218+0000", 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2071999999999999e-05 2024-03-26T03:06:35.502 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45871 8.3 e201/172 8:c164f2ec:::benchmark_data_smithi066_100612_object58311:head v 201'8373, mlcod=201'8358)", 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.472072+0000", 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011044457000000001, 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011361227999999999, 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.472072+0000", 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.472072+0000", 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.472073+0000", 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8199999999999998e-07 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.472080+0000", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8290000000000008e-06 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.472088+0000", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2320000000000004e-06 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.472094+0000", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9120000000000003e-06 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476345+0000", 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042514939999999998 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476427+0000", 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1670999999999998e-05 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45872 8.3 e201/172 8:c1653068:::benchmark_data_smithi066_100612_object75872:head v 201'8374, mlcod=201'8358)", 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.474928+0000", 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0081886709999999998, 2024-03-26T03:06:35.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0085673429999999998, 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474928+0000", 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474928+0000", 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474929+0000", 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.612e-06 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474937+0000", 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3119999999999994e-06 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474948+0000", 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0959e-05 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474957+0000", 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1589999999999993e-06 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.478780+0000", 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038229370000000002 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.478798+0000", 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8199999999999999e-05 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45873 8.3 e201/172 8:c16548b5:::benchmark_data_smithi066_100612_object11783:head v 201'8375, mlcod=201'8358)", 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.474977+0000", 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0081388299999999997, 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0085574210000000008, 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474977+0000", 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474977+0000", 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474978+0000", 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6199999999999998e-07 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474983+0000", 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5820000000000003e-06 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474987+0000", 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8770000000000001e-06 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.474992+0000", 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7099999999999998e-06 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.480803+0000", 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058108810000000004 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.480823+0000", 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0474000000000001e-05 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45875 8.3 e201/172 8:c165e14b:::benchmark_data_smithi066_100612_object22379:head v 201'8376, mlcod=201'8358)", 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.476360+0000", 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0067566140000000002, 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072201030000000003, 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.513 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476360+0000", 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476360+0000", 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476361+0000", 2024-03-26T03:06:35.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.33e-06 2024-03-26T03:06:35.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476367+0000", 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2219999999999999e-06 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476374+0000", 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9469999999999997e-06 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.476380+0000", 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9789999999999998e-06 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.483279+0000", 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0068985940000000001 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.483297+0000", 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8760999999999999e-05 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.517 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45876 8.3 e201/172)", 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.477988+0000", 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.005128194, 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056294839999999997, 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.477988+0000", 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.477988+0000", 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.477989+0000", 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1379999999999999e-06 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.477996+0000", 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1420000000000004e-06 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.478011+0000", 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.43e-05 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.478017+0000", 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8009999999999998e-06 2024-03-26T03:06:35.520 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45878 8.3 e201/172)", 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.479803+0000", 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0033128889999999999, 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038428669999999998, 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.479803+0000", 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.479803+0000", 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.479804+0000", 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.172e-06 2024-03-26T03:06:35.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.479811+0000", 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3180000000000003e-06 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.479820+0000", 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9889999999999998e-06 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.479825+0000", 2024-03-26T03:06:35.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2739999999999998e-06 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24961.0:126543 8.9 e201/172 8:9e3274d0:::benchmark_data_smithi066_100612_object126542:head v 201'7981, mlcod=201'7979)", 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.481101+0000", 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0020154109999999999, 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025788540000000002, 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.524 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481101+0000", 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481101+0000", 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481102+0000", 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2950000000000001e-06 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481128+0000", 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5927000000000001e-05 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481140+0000", 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2206e-05 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481147+0000", 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1400000000000002e-06 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481240+0000", 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2606000000000001e-05 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481258+0000", 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7914000000000002e-05 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.527 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45879 8.3 e201/172)", 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.481562+0000", 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0015538629999999999, 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021524729999999998, 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481562+0000", 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481562+0000", 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481564+0000", 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3349999999999999e-06 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481571+0000", 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0829999999999998e-06 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481579+0000", 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4249999999999998e-06 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.481586+0000", 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3679999999999998e-06 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.1.0:45880 8.3 e201/172)", 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:06:35.482764+0000", 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00035240299999999999, 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00097824999999999995, 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.482764+0000", 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.482764+0000", 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.482765+0000", 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2869999999999999e-06 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.482770+0000", 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0699999999999997e-06 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.482781+0000", 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.113e-05 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:06:35.482787+0000", 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.643e-06 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 13 2024-03-26T03:06:35.533 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:06:35.534 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:35.534 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 89 16 126572 126556 88.856 57.125 0.0143625 0.0112483 2024-03-26T03:06:35.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:35.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:35.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:35.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:35 smithi066 ceph-mon[29666]: 8.9 scrub starts 2024-03-26T03:06:35.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:35 smithi066 ceph-mon[29666]: pgmap v332: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 584 B/s rd, 64 MiB/s wr, 1.02k op/s; 46095/368712 objects degraded (12.502%) 2024-03-26T03:06:35.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:35 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:06:35.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:35 smithi066 ceph-mon[37920]: 8.9 scrub starts 2024-03-26T03:06:35.765 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:35 smithi066 ceph-mon[37920]: pgmap v332: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 584 B/s rd, 64 MiB/s wr, 1.02k op/s; 46095/368712 objects degraded (12.502%) 2024-03-26T03:06:35.765 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:35 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:06:35.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:35.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:35.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:35.971 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:35.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:36.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:36.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:36.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:36.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:36.347 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:36.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:36.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:36.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 90 5 127388 127383 88.443 51.6875 0.0169589 0.0112998 2024-03-26T03:06:36.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:36.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:36.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:36 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:06:36.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:36 smithi192 ceph-mon[33110]: 1.5 deep-scrub starts 2024-03-26T03:06:36.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:36 smithi192 ceph-mon[33110]: 1.5 deep-scrub ok 2024-03-26T03:06:36.647 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:36 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:06:36.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:06:36.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:06:36.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[29666]: 1.5 deep-scrub starts 2024-03-26T03:06:36.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[29666]: 1.5 deep-scrub ok 2024-03-26T03:06:36.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:06:36.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:06:36.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[37920]: 1.5 deep-scrub starts 2024-03-26T03:06:36.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[37920]: 1.5 deep-scrub ok 2024-03-26T03:06:36.654 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:36 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:06:36.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:36.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:36.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:36.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:36.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:36.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:8.f 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:8.c 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:8.b 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:8.8 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:8.7 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:6.6 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:6.c 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:6.4 2024-03-26T03:06:36.986 INFO:teuthology.orchestra.run.smithi066.stdout:6.f 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.2 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.1f 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.16 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.13 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.7 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.11 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:5.a 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:5.1c 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:5.14 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:6.0 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:5.16 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:5.13 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:5.12 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:2.a 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:2.9 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:2.1b 2024-03-26T03:06:36.987 INFO:teuthology.orchestra.run.smithi066.stdout:2.1 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:2.14 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:1.6 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:2.d 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:6.b 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:3.2 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:4.4 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:2.e 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:5.1 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:2.1e 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:5.17 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:4.1b 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:6.5 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:4.16 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:1.7 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:4.1e 2024-03-26T03:06:36.988 INFO:teuthology.orchestra.run.smithi066.stdout:6.8 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:1.9 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:8.0 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:2.13 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:2.12 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:6.19 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.0 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.10 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.11 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.9 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.17 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.14 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:3.1 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.19 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:6.1b 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:2.19 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:1.8 2024-03-26T03:06:36.989 INFO:teuthology.orchestra.run.smithi066.stdout:4.1a 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:4.15 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:4.1c 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:1.4 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:4.1d 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:4.c 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:4.f 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi066.stdout:5.0 2024-03-26T03:06:36.990 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:37.038 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:37.068 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:37.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:37.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:37.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:37.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:37.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:37.467 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:37.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total time run: 90.0409 2024-03-26T03:06:37.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total writes made: 127388 2024-03-26T03:06:37.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Write size: 65536 2024-03-26T03:06:37.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object size: 65536 2024-03-26T03:06:37.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Bandwidth (MB/sec): 88.4237 2024-03-26T03:06:37.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Bandwidth: 30.8082 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max bandwidth (MB/sec): 198.938 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min bandwidth (MB/sec): 18.0625 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average IOPS: 1414 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev IOPS: 492.932 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max IOPS: 3183 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min IOPS: 289 2024-03-26T03:06:37.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average Latency(s): 0.0113021 2024-03-26T03:06:37.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Latency(s): 0.0149531 2024-03-26T03:06:37.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max latency(s): 1.09198 2024-03-26T03:06:37.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min latency(s): 0.00133267 2024-03-26T03:06:37.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: 8.5 deep-scrub starts 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: pgmap v333: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 47116/377082 objects degraded (12.495%) 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: 8.9 scrub ok 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 47116/377082 objects degraded (12.495%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: 5.15 deep-scrub starts 2024-03-26T03:06:37.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[29666]: 5.15 deep-scrub ok 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: 8.5 deep-scrub starts 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: pgmap v333: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 47116/377082 objects degraded (12.495%) 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: 8.9 scrub ok 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 47116/377082 objects degraded (12.495%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: 5.15 deep-scrub starts 2024-03-26T03:06:37.531 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:37 smithi066 ceph-mon[37920]: 5.15 deep-scrub ok 2024-03-26T03:06:37.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:37.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:37.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: 8.5 deep-scrub starts 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: pgmap v333: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 47116/377082 objects degraded (12.495%) 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: 8.9 scrub ok 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 47116/377082 objects degraded (12.495%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: 5.15 deep-scrub starts 2024-03-26T03:06:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:37 smithi192 ceph-mon[33110]: 5.15 deep-scrub ok 2024-03-26T03:06:37.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:37.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:37.889 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:37.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:37.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:37.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:38.026 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:38.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:38.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:38.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:38.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:38.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:38.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:38.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:38.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:38.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:38.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:38 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:06:38.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:38 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:06:38.652 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:38 smithi192 ceph-mon[33110]: 8.4 scrub starts 2024-03-26T03:06:38.725 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:38.758 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:38.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:38 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:06:38.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:38 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:06:38.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:38 smithi066 ceph-mon[29666]: 8.4 scrub starts 2024-03-26T03:06:38.791 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:38 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:06:38.791 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:38 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:06:38.791 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:38 smithi066 ceph-mon[37920]: 8.4 scrub starts 2024-03-26T03:06:38.860 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:38.861 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:06:38.861 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:06:38.861 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:06:38.861 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:06:38.861 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:06:38.909 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:38.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:39.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:39.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:39.159 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:39.194 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:39.296 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.334 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:39.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:39.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:39.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:39 smithi192 ceph-mon[33110]: pgmap v334: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 952 B/s rd, 68 MiB/s wr, 1.09k op/s; 47116/377082 objects degraded (12.495%) 2024-03-26T03:06:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:39 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:06:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:39.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:06:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:06:39.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:39.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:06:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:39.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:39.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:39.719 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[29666]: pgmap v334: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 952 B/s rd, 68 MiB/s wr, 1.09k op/s; 47116/377082 objects degraded (12.495%) 2024-03-26T03:06:39.719 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:06:39.720 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:39.720 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:39.720 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[37920]: pgmap v334: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 952 B/s rd, 68 MiB/s wr, 1.09k op/s; 47116/377082 objects degraded (12.495%) 2024-03-26T03:06:39.720 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:06:39.720 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:39.720 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:39.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:39.762 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.3 --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 6.c --file /var/log/ceph/exp.6.c.3 2024-03-26T03:06:39.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:39.907 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:39.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:40.000 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:40.008 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:40.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:40.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:40.232 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:40.306 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/osd.3/config 2024-03-26T03:06:40.334 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:40.380 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:40.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:40.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:40.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:06:40.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[29666]: 8.4 scrub ok 2024-03-26T03:06:40.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[29666]: 8.1 deep-scrub starts 2024-03-26T03:06:40.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:40.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:06:40.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[37920]: 8.4 scrub ok 2024-03-26T03:06:40.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[37920]: 8.1 deep-scrub starts 2024-03-26T03:06:40.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:40.621 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:40.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:40.693 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:06:40.730 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:06:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:40 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:06:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:40 smithi192 ceph-mon[33110]: 8.4 scrub ok 2024-03-26T03:06:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:40 smithi192 ceph-mon[33110]: 8.1 deep-scrub starts 2024-03-26T03:06:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:40.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:40.769 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:06:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:40.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:40.826 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:06:40.868 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:06:40.913 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:06:40.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:40.950 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:06:40.951 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:06:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:40.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:40.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:41.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:41.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:41.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:41.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:41.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:41.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:41.445 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:06:41.479 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:41.517 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:41.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: pgmap v335: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.21k op/s; 47390/379158 objects degraded (12.499%) 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: 8.2 scrub starts 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: 8.a scrub starts 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3078850973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/466162146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: pgmap v335: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.21k op/s; 47390/379158 objects degraded (12.499%) 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: 8.2 scrub starts 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: 8.a scrub starts 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3078850973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:06:41.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/466162146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:41.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.662 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:41.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: pgmap v335: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 70 MiB/s wr, 1.21k op/s; 47390/379158 objects degraded (12.499%) 2024-03-26T03:06:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:06:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: 8.2 scrub starts 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: 8.a scrub starts 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3078850973' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: from='client.25048 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:06:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/466162146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:41.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:41.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:41.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:41.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:41.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:42.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:42.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:42.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:42.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:42.313 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.347 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:42.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:42.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:42.492 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:42.594 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[37920]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:06:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[37920]: osdmap e202: 8 total, 7 up, 8 in 2024-03-26T03:06:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:06:42.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:42.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[29666]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:06:42.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[29666]: osdmap e202: 8 total, 7 up, 8 in 2024-03-26T03:06:42.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:42 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:06:42.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:42 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:42 smithi192 ceph-mon[33110]: from='client.25048 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:06:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:42 smithi192 ceph-mon[33110]: osdmap e202: 8 total, 7 up, 8 in 2024-03-26T03:06:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:42 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:06:42.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:42.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:42.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:42.886 INFO:teuthology.orchestra.run.smithi066.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:06:42.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:42.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:43.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:43.034 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:43.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:43.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:43.271 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:43.308 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:06:43.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:06:43.410 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:43.453 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:43 smithi192 ceph-mon[33110]: pgmap v337: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 48 MiB/s wr, 905 op/s; 47315/378525 objects degraded (12.500%) 2024-03-26T03:06:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:43 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 47315/378525 objects degraded (12.500%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:43.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:43.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:43.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:43.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:43.752 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:43 smithi066 ceph-mon[29666]: pgmap v337: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 48 MiB/s wr, 905 op/s; 47315/378525 objects degraded (12.500%) 2024-03-26T03:06:43.752 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:43 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 47315/378525 objects degraded (12.500%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:43.753 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:43 smithi066 ceph-mon[37920]: pgmap v337: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 48 MiB/s wr, 905 op/s; 47315/378525 objects degraded (12.500%) 2024-03-26T03:06:43.753 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:43 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 47315/378525 objects degraded (12.500%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:43.824 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:43.867 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:43.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:43.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:44.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:44.098 INFO:teuthology.orchestra.run.smithi066.stderr:Exporting 6.c info 6.c( empty local-lis/les=69/70 n=0 ec=69/69 lis/c=69/69 les/c/f=70/70/0 sis=69) 2024-03-26T03:06:44.098 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:06:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:06:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:06:44.103 INFO:teuthology.orchestra.run.smithi066.stderr:Export successful 2024-03-26T03:06:44.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:44.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:44.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:44.292 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:06:44.292 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:06:44.306 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:06:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:44.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:44.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:44.456 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:44.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:44.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:44.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:44.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:44 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:06:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:44 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:06:44.766 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:44 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:06:44.766 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:44 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:06:44.766 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:44 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:06:44.766 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:44 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:06:44.834 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:44.870 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:44.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:44.972 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:45.013 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:45.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:06:45.115 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:45.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:45.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:45.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.3 --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 6.c 2024-03-26T03:06:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:45.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:45.346 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:45.377 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:06:45.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:45.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:45.490 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:45 smithi066 ceph-mon[29666]: pgmap v338: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 29 MiB/s wr, 747 op/s; 46646/374601 objects degraded (12.452%) 2024-03-26T03:06:45.490 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:45 smithi066 ceph-mon[29666]: osdmap e203: 8 total, 7 up, 8 in 2024-03-26T03:06:45.491 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:45 smithi066 ceph-mon[37920]: pgmap v338: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 29 MiB/s wr, 747 op/s; 46646/374601 objects degraded (12.452%) 2024-03-26T03:06:45.491 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:45 smithi066 ceph-mon[37920]: osdmap e203: 8 total, 7 up, 8 in 2024-03-26T03:06:45.533 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:45.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:45.598 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:45.677 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/osd.3/config 2024-03-26T03:06:45.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:45 smithi192 ceph-mon[33110]: pgmap v338: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 29 MiB/s wr, 747 op/s; 46646/374601 objects degraded (12.452%) 2024-03-26T03:06:45.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:45 smithi192 ceph-mon[33110]: osdmap e203: 8 total, 7 up, 8 in 2024-03-26T03:06:45.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:45.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:45.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:45.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:45.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:45.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:46.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:46.125 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.173 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:46.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:46.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:46.314 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:46.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:46.461 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:46.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:46.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:46.704 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:46.741 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:46.843 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:46.882 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:46.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:46.983 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:47.027 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:47.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:47.166 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:47.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:47.303 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:47.404 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:47.443 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:47.544 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:47.574 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:47.676 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:47.715 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:47 smithi192 ceph-mon[33110]: pgmap v340: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 858 op/s; 46025/368037 objects degraded (12.506%) 2024-03-26T03:06:47.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:47.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:47.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:47 smithi066 ceph-mon[29666]: pgmap v340: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 858 op/s; 46025/368037 objects degraded (12.506%) 2024-03-26T03:06:47.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:47 smithi066 ceph-mon[37920]: pgmap v340: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 858 op/s; 46025/368037 objects degraded (12.506%) 2024-03-26T03:06:47.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:47.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:47.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:48.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:48.124 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:48.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:48.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:48.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:48.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:48.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:48.525 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:48.627 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:48.664 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:48 smithi192 ceph-mon[33110]: pgmap v341: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 1.4 MiB/s wr, 508 op/s; 46025/368037 objects degraded (12.506%) 2024-03-26T03:06:48.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:48.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:48.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:48 smithi066 ceph-mon[29666]: pgmap v341: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 1.4 MiB/s wr, 508 op/s; 46025/368037 objects degraded (12.506%) 2024-03-26T03:06:48.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:48 smithi066 ceph-mon[37920]: pgmap v341: 158 pgs: 3 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 88 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 1.4 MiB/s wr, 508 op/s; 46025/368037 objects degraded (12.506%) 2024-03-26T03:06:48.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:48.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:48.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:49.026 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:49.053 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:49.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:49.189 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:49.291 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:49.330 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:49.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:49.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:49.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:49.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:49.623 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 46025/368037 objects degraded (12.506%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:49.623 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:49 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:49.624 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:06:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:06:49.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:49.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:49.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 46025/368037 objects degraded (12.506%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:49.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:49 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:49.842 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 46025/368037 objects degraded (12.506%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:49.842 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:49 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:06:49.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:49.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:49.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:49.980 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.021 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:50.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:50.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:50.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:50.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:50.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:50.338 INFO:teuthology.orchestra.run.smithi066.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:06:50.342 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:50.397 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:50.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:50.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:50.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:50 smithi066 ceph-mon[29666]: pgmap v342: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 752 op/s; 44853/359028 objects degraded (12.493%) 2024-03-26T03:06:50.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:50 smithi066 ceph-mon[37920]: pgmap v342: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 752 op/s; 44853/359028 objects degraded (12.493%) 2024-03-26T03:06:50.664 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:50.706 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:50.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:50.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:50 smithi192 ceph-mon[33110]: pgmap v342: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 752 op/s; 44853/359028 objects degraded (12.493%) 2024-03-26T03:06:50.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:50.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:50.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:50.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:51.023 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:06:51.023 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:06:51.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:51.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:51.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:51.271 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:51.373 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:51.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:51.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:51.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:51.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:51.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:51.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:06:51.792 INFO:teuthology.orchestra.run.smithi066.stdout: marking collection for removal 2024-03-26T03:06:51.792 INFO:teuthology.orchestra.run.smithi066.stdout:setting '_remove' omap key 2024-03-26T03:06:51.792 INFO:teuthology.orchestra.run.smithi066.stdout:finish_remove_pgs 6.c_head removing 6.c 2024-03-26T03:06:51.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:51.798 INFO:teuthology.orchestra.run.smithi066.stdout:Remove successful 2024-03-26T03:06:51.806 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:06:51.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:51.940 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:51.983 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:51.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:52.054 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:51 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1231930922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:52.055 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:51 smithi066 ceph-mon[29666]: from='client.25057 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:52.055 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:51 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1231930922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:52.055 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:51 smithi066 ceph-mon[37920]: from='client.25057 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:52.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:52.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:52.170 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:52.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:52.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:51 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1231930922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:52.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:51 smithi192 ceph-mon[33110]: from='client.25057 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:06:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:52.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:52.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:52.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:52.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:52.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:52.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:52.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:52.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:52.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:52.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:52.958 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:52 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:52.958 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:52 smithi192 ceph-mon[33110]: from='client.25057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:06:52.958 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:52 smithi192 ceph-mon[33110]: osdmap e204: 8 total, 7 up, 8 in 2024-03-26T03:06:52.958 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:52 smithi192 ceph-mon[33110]: pgmap v344: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 737 op/s; 44633/356931 objects degraded (12.505%) 2024-03-26T03:06:52.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:52.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:52.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:53.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:53.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:53.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:53.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[29666]: from='client.25057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:06:53.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[29666]: osdmap e204: 8 total, 7 up, 8 in 2024-03-26T03:06:53.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[29666]: pgmap v344: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 737 op/s; 44633/356931 objects degraded (12.505%) 2024-03-26T03:06:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:06:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[37920]: from='client.25057 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:06:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[37920]: osdmap e204: 8 total, 7 up, 8 in 2024-03-26T03:06:53.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:52 smithi066 ceph-mon[37920]: pgmap v344: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 737 op/s; 44633/356931 objects degraded (12.505%) 2024-03-26T03:06:53.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:53.232 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:53.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:53.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:53.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:53.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:53.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:53.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:53.763 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:53.802 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:53.904 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:53.946 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:53.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:53.972 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:06:53.972 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:06:54.037 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:06:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:06:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:06:54.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:54.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:54.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:54.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:54.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 44633/356931 objects degraded (12.505%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:54.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:54.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:54.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 44633/356931 objects degraded (12.505%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:54.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:54.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:54.296 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 44633/356931 objects degraded (12.505%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:54.296 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:06:54.296 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:06:54.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:54.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:54.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:54.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:54.585 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:54.624 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:54.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:54.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.3 --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.c.3 2024-03-26T03:06:54.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:54.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:06:54.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:54.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:54.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:55.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:55.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:55.051 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:55 smithi066 ceph-mon[29666]: pgmap v345: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 731 op/s; 43680/351450 objects degraded (12.429%) 2024-03-26T03:06:55.089 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/osd.3/config 2024-03-26T03:06:55.102 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:06:55.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:55.139 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:06:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:55.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:55.174 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:06:55.211 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:06:55.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:55 smithi192 ceph-mon[33110]: pgmap v345: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 731 op/s; 43680/351450 objects degraded (12.429%) 2024-03-26T03:06:55.259 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:06:55.268 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:55.302 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:55 smithi066 ceph-mon[37920]: pgmap v345: 158 pgs: 3 active+clean+scrubbing+deep, 28 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 731 op/s; 43680/351450 objects degraded (12.429%) 2024-03-26T03:06:55.302 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:06:55.308 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:55.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:06:55.344 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:06:55.344 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:06:55.344 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:06:55.344 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:06:55.344 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:06:55.345 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:06:55.345 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:06:55.345 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:06:55.345 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:06:55.410 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:55.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:55.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:06:55.593 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:06:55.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:55.736 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:55.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:55.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:55.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:55.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:56.013 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:56.053 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:56.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:56.189 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:06:56.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:06:56.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1696798878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:56.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:56.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:56.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1696798878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:56.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1696798878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:06:56.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:56.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:56.647 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:56.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:56.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:56.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:56.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:56.970 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:56.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:57.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.109 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:57.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:57.188 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:06:57.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:57.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:57.263 INFO:teuthology.orchestra.run.smithi066.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:06:57.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:57 smithi066 ceph-mon[37920]: pgmap v346: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 893 op/s; 42651/341262 objects degraded (12.498%) 2024-03-26T03:06:57.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:57 smithi066 ceph-mon[29666]: pgmap v346: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 893 op/s; 42651/341262 objects degraded (12.498%) 2024-03-26T03:06:57.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:57.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:57.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:57 smithi192 ceph-mon[33110]: pgmap v346: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 893 op/s; 42651/341262 objects degraded (12.498%) 2024-03-26T03:06:57.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:57.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:57.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:57.671 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:57.773 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.835 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:57.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:57.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:57.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:57.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:58.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:58.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:58.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:58.181 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:58 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:06:58.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:58 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:06:58.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:58 smithi192 ceph-mon[33110]: osdmap e205: 8 total, 7 up, 8 in 2024-03-26T03:06:58.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:58 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:06:58.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:58 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:06:58.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:06:58.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:58.235 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:06:58.235 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:06:58.235 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[29666]: osdmap e205: 8 total, 7 up, 8 in 2024-03-26T03:06:58.235 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:06:58.236 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:06:58.236 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:06:58.236 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:06:58.236 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[37920]: osdmap e205: 8 total, 7 up, 8 in 2024-03-26T03:06:58.236 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:06:58.236 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:58 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:06:58.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:58.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:58.346 INFO:teuthology.orchestra.run.smithi066.stdout:get_pg_num_history pg_num_history pg_num_history(e197 pg_nums {1={20=1,164=11},2={63=32},3={63=3},4={65=32},5={67=32},6={69=32},7={111=26},8={172=16}} deleted_pools 169,7) 2024-03-26T03:06:58.346 INFO:teuthology.orchestra.run.smithi066.stdout:Importing pgid 6.c 2024-03-26T03:06:58.347 INFO:teuthology.orchestra.run.smithi066.stdout:write_pg epoch 163 info 6.c( empty local-lis/les=69/70 n=0 ec=69/69 lis/c=69/69 les/c/f=70/70/0 sis=69) 2024-03-26T03:06:58.349 INFO:teuthology.orchestra.run.smithi066.stdout:Import successful 2024-03-26T03:06:58.367 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:58.415 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:58.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:58.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:58.523 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75523 7.7 7:e4bf8bb6:::benchmark_data_smithi066_76917_object75522:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.308141+0000", 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 308.202298235, 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0242819809999999, 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75523 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308141+0000", 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308141+0000", 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308144+0000", 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2849999999999999e-06 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308199+0000", 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4549000000000002e-05 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308223+0000", 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4646000000000001e-05 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308240+0000", 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6317e-05 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370555+0000", 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.062315191999999998 2024-03-26T03:06:58.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370566+0000", 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0849000000000001e-05 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.306034+0000", 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93546880300000002 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-03-26T03:06:58.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.306851+0000", 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00081647499999999999 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332390+0000", 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025538693000000001 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332423+0000", 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3172e-05 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.530 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75529 7.7 7:e0f45fed:::benchmark_data_smithi066_76917_object75528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:06:58.530 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.308269+0000", 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 308.20217025900001, 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.024087451, 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75529 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.531 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308269+0000", 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308269+0000", 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308270+0000", 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8700000000000004e-07 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308307+0000", 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7537000000000003e-05 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308315+0000", 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0870000000000007e-06 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.308322+0000", 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6390000000000003e-06 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370590+0000", 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.062267956999999999 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.370598+0000", 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5460000000000003e-06 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.307310+0000", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93671239100000003 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for degraded object", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.307993+0000", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00068283600000000003 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332308+0000", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024315666 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332356+0000", 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7805000000000001e-05 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75541 7.7 7:e067e745:::benchmark_data_smithi066_76917_object75540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.567207+0000", 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 307.943232529, 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.765059932, 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.536 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75541 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567207+0000", 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567207+0000", 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567210+0000", 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1e-06 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567274+0000", 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4264999999999995e-05 2024-03-26T03:06:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567292+0000", 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8331000000000001e-05 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567303+0000", 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0502999999999999e-05 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567369+0000", 2024-03-26T03:06:58.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5870000000000005e-05 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.567379+0000", 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0186999999999999e-05 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.308049+0000", 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.74067035299999995 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.308219+0000", 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000169674 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3", 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.308417+0000", 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019777400000000001 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.310073+0000", 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001656554 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.330525+0000", 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020451608 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332103+0000", 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001578491 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332166+0000", 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2803000000000002e-05 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.332266+0000", 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100419 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76595 7.12 7.448c6a52 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.299275+0000", 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 306.21116406900001, 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98390260200000002, 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.543 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76595 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299275+0000", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299275+0000", 2024-03-26T03:06:58.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299277+0000", 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0020000000000001e-06 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299337+0000", 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9997999999999997e-05 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299350+0000", 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3040999999999999e-05 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299359+0000", 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8070000000000006e-06 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299430+0000", 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0667000000000006e-05 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.299438+0000", 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0360000000000006e-06 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283166+0000", 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.983728622 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283178+0000", 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1429e-05 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.547 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:76610 7.19 7.a1cdf4d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:52.322165+0000", 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 306.18827409800002, 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96114181099999996, 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 76610 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.548 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322165+0000", 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322165+0000", 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322175+0000", 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7710000000000001e-06 2024-03-26T03:06:58.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322210+0000", 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5701999999999998e-05 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322224+0000", 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3427e-05 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322236+0000", 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2529e-05 2024-03-26T03:06:58.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322294+0000", 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7765e-05 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:52.322304+0000", 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7759999999999992e-06 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283298+0000", 2024-03-26T03:06:58.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96099361900000002 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:53.283307+0000", 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2220000000000008e-06 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112085 7.12 7:493de57c:::benchmark_data_smithi066_76917_object11902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 247.23127108899999, 2024-03-26T03:06:58.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.024736326, 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112085 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279170+0000", 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.593e-06 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279174+0000", 2024-03-26T03:06:58.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8470000000000003e-06 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279184+0000", 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6080000000000002e-06 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279193+0000", 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7409999999999998e-06 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279262+0000", 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9488999999999998e-05 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279273+0000", 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.08e-05 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.291772+0000", 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0124987969999999 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.291923+0000", 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015105899999999999 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.292236+0000", 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031284800000000001 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.296791+0000", 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045547749999999996 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.297868+0000", 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010770319999999999 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303845+0000", 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059774659999999999 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303877+0000", 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1480999999999999e-05 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303904+0000", 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.779e-05 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112092 7.12 7:4c17d0e1:::benchmark_data_smithi066_76917_object11909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 247.22394230099999, 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0192903719999999, 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.559 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112092 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:06:58.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286499+0000", 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.717e-06 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286504+0000", 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286516+0000", 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1919999999999999e-05 2024-03-26T03:06:58.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286525+0000", 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0359999999999995e-06 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286577+0000", 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2420000000000003e-05 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286590+0000", 2024-03-26T03:06:58.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2469999999999999e-05 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.293804+0000", 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0072144949999999 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.293919+0000", 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115445 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.294102+0000", 2024-03-26T03:06:58.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000182339 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298885+0000", 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047836770000000001 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303949+0000", 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005063115 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305695+0000", 2024-03-26T03:06:58.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017461379999999999 2024-03-26T03:06:58.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305746+0000", 2024-03-26T03:06:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:58.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0810000000000003e-05 2024-03-26T03:06:58.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305787+0000", 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-05 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112198 7.12 7:4c79c881:::benchmark_data_smithi066_76917_object12015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 247.013020936, 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.81316842600000006, 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.567 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112198 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497420+0000", 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5859999999999999e-06 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497425+0000", 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2039999999999996e-06 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497438+0000", 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2672e-05 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497445+0000", 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8129999999999996e-06 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497501+0000", 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5503e-05 2024-03-26T03:06:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497509+0000", 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6899999999999992e-06 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.294932+0000", 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79742365100000001 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295019+0000", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7113000000000004e-05 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295174+0000", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000155014 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298890+0000", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037151950000000001 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305657+0000", 2024-03-26T03:06:58.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0067669499999999999 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310476+0000", 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048193130000000004 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310539+0000", 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3424999999999997e-05 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310587+0000", 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7296999999999998e-05 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112201 7.12 7:4b665aee:::benchmark_data_smithi066_76917_object12018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 247.00652454999999, 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80677220100000002, 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112201 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503916+0000", 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0890000000000001e-06 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503920+0000", 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.89e-06 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503929+0000", 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7229999999999999e-06 2024-03-26T03:06:58.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503937+0000", 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0080000000000002e-06 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.504005+0000", 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7787000000000007e-05 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.504018+0000", 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.252e-05 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295896+0000", 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79187867700000003 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295978+0000", 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.174e-05 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.296092+0000", 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114016 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298893+0000", 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028010299999999999 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305824+0000", 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069314609999999999 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310625+0000", 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004800861 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310658+0000", 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2891000000000001e-05 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310687+0000", 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8507999999999999e-05 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157251 7.19 7:9b7ef1c8:::benchmark_data_smithi066_76917_object57068:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:06:58.580 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.54177216100001, 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.025284095, 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157251 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968672+0000", 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6670000000000004e-06 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968678+0000", 2024-03-26T03:06:58.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5309999999999998e-06 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968695+0000", 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7135000000000001e-05 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968705+0000", 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4280000000000002e-06 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.969069+0000", 2024-03-26T03:06:58.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036450099999999999 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.969082+0000", 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2761e-05 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.987851+0000", 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0187692269999999 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.987998+0000", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000146957 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.988367+0000", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000368885 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992132+0000", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037647359999999999 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993640+0000", 2024-03-26T03:06:58.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001508604 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993833+0000", 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000192404 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993891+0000", 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8391999999999999e-05 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993951+0000", 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9867000000000001e-05 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157261 7.19 7:989eae58:::benchmark_data_smithi066_76917_object57078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.53501885700001, 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.026782426, 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.587 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157261 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975421+0000", 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7999999999999993e-07 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975426+0000", 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4510000000000002e-06 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975436+0000", 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0815999999999999e-05 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975443+0000", 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975508+0000", 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4856000000000004e-05 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975521+0000", 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2813000000000001e-05 2024-03-26T03:06:58.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990406+0000", 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.014885641 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990530+0000", 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012369900000000001 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990814+0000", 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000283503 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995533+0000", 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047195300000000004 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997610+0000", 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020769709999999999 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002088+0000", 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004477895 2024-03-26T03:06:58.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002142+0000", 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4333000000000003e-05 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002203+0000", 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0304999999999999e-05 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157262 7.19 7:9be70472:::benchmark_data_smithi066_76917_object57079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.532531307, 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244733150000001, 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157262 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.594 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977909+0000", 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977915+0000", 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3040000000000004e-06 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977925+0000", 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0761000000000001e-05 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977933+0000", 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2690000000000002e-06 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.978012+0000", 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9157000000000001e-05 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.978024+0000", 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2398999999999999e-05 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992170+0000", 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.014145802 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992257+0000", 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6532000000000004e-05 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992467+0000", 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021020199999999999 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995538+0000", 2024-03-26T03:06:58.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003071514 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997648+0000", 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021097339999999998 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002278+0000", 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046300810000000003 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002337+0000", 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9058000000000003e-05 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002381+0000", 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3902000000000001e-05 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157326 7.19 7:9db2bad9:::benchmark_data_smithi066_76917_object57143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.473977386, 2024-03-26T03:06:58.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96608487600000004, 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157326 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036464+0000", 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.486e-06 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036474+0000", 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244999999999999e-05 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036490+0000", 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5928000000000001e-05 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036499+0000", 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6670000000000002e-06 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036567+0000", 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8338999999999997e-05 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036578+0000", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0682999999999999e-05 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994011+0000", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95743293299999999 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994083+0000", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2106000000000004e-05 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994259+0000", 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017634500000000001 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997687+0000", 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034280309999999998 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.605 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:58.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002229+0000", 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045413550000000004 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002484+0000", 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025479300000000001 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002515+0000", 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1798999999999998e-05 2024-03-26T03:06:58.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002547+0000", 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1165999999999999e-05 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157394 7.19 7:9d416a04:::benchmark_data_smithi066_76917_object57211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.41591560200001, 2024-03-26T03:06:58.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90827457, 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157394 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094525+0000", 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1820000000000001e-06 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094531+0000", 2024-03-26T03:06:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6420000000000001e-06 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094543+0000", 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1473e-05 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094549+0000", 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3749999999999999e-06 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094591+0000", 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2333999999999998e-05 2024-03-26T03:06:58.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094599+0000", 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1740000000000003e-06 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995590+0000", 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.900991404 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995703+0000", 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112657 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995979+0000", 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027668000000000001 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002238+0000", 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006258581 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002425+0000", 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018703599999999999 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002629+0000", 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000204358 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002755+0000", 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012557900000000001 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002798+0000", 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3095e-05 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.614 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157412 7.19 7:99cf3667:::benchmark_data_smithi066_76917_object57229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.40145653799999, 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89402950199999998, 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157412 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108984+0000", 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.06e-06 2024-03-26T03:06:58.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108988+0000", 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108996+0000", 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999992e-06 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109003+0000", 2024-03-26T03:06:58.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0099999999999998e-06 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109077+0000", 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4035000000000005e-05 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109090+0000", 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2683e-05 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997735+0000", 2024-03-26T03:06:58.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88864550600000003 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997869+0000", 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013324800000000001 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.998178+0000", 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030980100000000001 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002441+0000", 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042626239999999996 2024-03-26T03:06:58.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002581+0000", 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014011500000000001 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002909+0000", 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032837200000000001 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002971+0000", 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1194999999999999e-05 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003012+0000", 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1405000000000001e-05 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157451 7.19 7:9ac14505:::benchmark_data_smithi066_76917_object57268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.373503992, 2024-03-26T03:06:58.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87424788600000003, 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157451 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136936+0000", 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3880000000000001e-06 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136941+0000", 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9099999999999996e-06 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136956+0000", 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4181e-05 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136971+0000", 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5316999999999999e-05 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.137210+0000", 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023869700000000001 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.137224+0000", 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4207e-05 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.999829+0000", 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86260561300000005 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.999973+0000", 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014313800000000001 2024-03-26T03:06:58.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.000235+0000", 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026196699999999999 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002445+0000", 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022107519999999999 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002855+0000", 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040992999999999997 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011105+0000", 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082499259999999994 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011144+0000", 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8408000000000002e-05 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011183+0000", 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9452000000000001e-05 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157623 7.19 7:9ae45d3c:::benchmark_data_smithi066_76917_object57440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.20977129299999, 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.71074928199999998, 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157623 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300670+0000", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300674+0000", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4469999999999999e-06 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300678+0000", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.163e-06 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300717+0000", 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300724+0000", 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0079999999999996e-06 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003059+0000", 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70233499399999999 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003162+0000", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103136 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003429+0000", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000266395 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011275+0000", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078460179999999997 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011290+0000", 2024-03-26T03:06:58.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5318000000000001e-05 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011330+0000", 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9536999999999998e-05 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011379+0000", 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9342000000000003e-05 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011417+0000", 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8158999999999998e-05 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157629 7.19 7:9d222119:::benchmark_data_smithi066_76917_object57446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.20374155600001, 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70924085000000003, 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.634 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157629 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306699+0000", 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.702e-06 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306705+0000", 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1109999999999996e-06 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306718+0000", 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2378000000000001e-05 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306725+0000", 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2250000000000003e-06 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306795+0000", 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0272999999999996e-05 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306806+0000", 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0883000000000001e-05 2024-03-26T03:06:58.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005375+0000", 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69856929300000004 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005481+0000", 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105763 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005761+0000", 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027942199999999998 2024-03-26T03:06:58.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011295+0000", 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055347560000000001 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011461+0000", 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000166088 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015840+0000", 2024-03-26T03:06:58.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043782969999999997 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015892+0000", 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2256999999999998e-05 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015938+0000", 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6402000000000001e-05 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157655 7.19 7:9b64c26d:::benchmark_data_smithi066_76917_object57472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.16288936300001, 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67836821199999997, 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157655 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347551+0000", 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.85e-07 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347554+0000", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1889999999999999e-06 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347564+0000", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8030000000000008e-06 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347568+0000", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9520000000000002e-06 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347621+0000", 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2417000000000003e-05 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347628+0000", 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5120000000000002e-06 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.008556+0000", 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66092736900000004 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.008679+0000", 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012314300000000001 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.009015+0000", 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033638899999999998 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011425+0000", 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00241004 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015997+0000", 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045720439999999999 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025826+0000", 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0098289510000000007 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025878+0000", 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1316000000000001e-05 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025918+0000", 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0302000000000001e-05 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157664 7.19 7:9fd64715:::benchmark_data_smithi066_76917_object57481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.14933221800001, 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66498246000000005, 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:06:58.647 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157664 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:06:58.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361108+0000", 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6899999999999997e-07 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361110+0000", 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9270000000000001e-06 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361117+0000", 2024-03-26T03:06:58.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9659999999999999e-06 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361122+0000", 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2459999999999997e-06 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361169+0000", 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6907999999999999e-05 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361176+0000", 2024-03-26T03:06:58.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5129999999999999e-06 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011489+0000", 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65031320299999995 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011597+0000", 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107469 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011895+0000", 2024-03-26T03:06:58.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029828400000000001 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015953+0000", 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040577169999999997 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.016041+0000", 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8707000000000003e-05 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026023+0000", 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0099818719999999993 2024-03-26T03:06:58.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026054+0000", 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0925e-05 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026089+0000", 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5054e-05 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:06:58.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:06:58.654 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:06:58.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:58.756 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:58.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:58.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:58.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:58.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:58.997 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:59.038 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:59.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:59 smithi066 ceph-mon[29666]: osdmap e206: 8 total, 7 up, 8 in 2024-03-26T03:06:59.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:59 smithi066 ceph-mon[29666]: pgmap v349: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 829 op/s; 42651/341262 objects degraded (12.498%) 2024-03-26T03:06:59.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:06:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 42651/341262 objects degraded (12.498%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:59.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:59 smithi066 ceph-mon[37920]: osdmap e206: 8 total, 7 up, 8 in 2024-03-26T03:06:59.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:59 smithi066 ceph-mon[37920]: pgmap v349: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 829 op/s; 42651/341262 objects degraded (12.498%) 2024-03-26T03:06:59.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:06:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 42651/341262 objects degraded (12.498%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:59.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:06:59.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:06:59.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:06:59.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:06:59.367 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:59 smithi192 ceph-mon[33110]: osdmap e206: 8 total, 7 up, 8 in 2024-03-26T03:06:59.367 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:59 smithi192 ceph-mon[33110]: pgmap v349: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 829 op/s; 42651/341262 objects degraded (12.498%) 2024-03-26T03:06:59.368 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:06:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 42651/341262 objects degraded (12.498%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:06:59.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:06:59.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:06:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:59.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:06:59.567 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:06:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.606 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:59.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:06:59.671 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/exp.6.c.3 2024-03-26T03:06:59.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:06:59.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:06:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:06:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:06:59.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:06:59.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:06:59.893 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:06:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:06:59.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:06:59.918 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:06:59.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:00.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:00.141 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:00.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:00.251 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:00.251 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:00.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:00.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:00.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:00.436 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:00.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:00.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:00.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:00.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:00.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:00.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:00.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:00.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:00.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:00.961 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:01.003 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:01.102 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:07:01.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.150 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:01.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:01.249 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:01 smithi066 ceph-mon[37920]: pgmap v350: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 40653/325623 objects degraded (12.485%) 2024-03-26T03:07:01.250 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:01 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3184853911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:01.250 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:01 smithi066 ceph-mon[37920]: from='client.35030 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:01.250 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:01 smithi066 ceph-mon[29666]: pgmap v350: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 40653/325623 objects degraded (12.485%) 2024-03-26T03:07:01.250 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:01 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3184853911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:01.250 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:01 smithi066 ceph-mon[29666]: from='client.35030 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:01.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:01.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:01.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:01.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:01.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:01 smithi192 ceph-mon[33110]: pgmap v350: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 40653/325623 objects degraded (12.485%) 2024-03-26T03:07:01.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:01 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3184853911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:01.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:01 smithi192 ceph-mon[33110]: from='client.35030 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:01.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:01.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:01.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:01.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:01.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.847 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:01.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:01.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:01.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:01.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:02.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:02.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:02.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:02.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:02.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:02.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:02.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[29666]: from='client.35030 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:07:02.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[29666]: osdmap e207: 8 total, 7 up, 8 in 2024-03-26T03:07:02.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:07:02.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:07:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[37920]: from='client.35030 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:07:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[37920]: osdmap e207: 8 total, 7 up, 8 in 2024-03-26T03:07:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:07:02.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:02 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:07:02.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:02.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:02 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:02 smithi192 ceph-mon[33110]: from='client.35030 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:07:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:02 smithi192 ceph-mon[33110]: osdmap e207: 8 total, 7 up, 8 in 2024-03-26T03:07:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:02 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:07:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:02 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:07:02.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:02.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:02.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:02.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:02.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:02.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:02.953 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:02.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:03.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:03.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:03.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:03.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: pgmap v352: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 40235/321768 objects degraded (12.504%) 2024-03-26T03:07:03.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: 8.2 scrub starts 2024-03-26T03:07:03.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: 8.3 scrub starts 2024-03-26T03:07:03.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:07:03.210 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:07:03.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:07:03.211 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:03 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:07:03.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:03.287 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:07:03.287 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: pgmap v352: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 40235/321768 objects degraded (12.504%) 2024-03-26T03:07:03.287 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: 8.2 scrub starts 2024-03-26T03:07:03.287 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: 8.3 scrub starts 2024-03-26T03:07:03.287 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: pgmap v352: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 40235/321768 objects degraded (12.504%) 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: 8.2 scrub starts 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: 8.3 scrub starts 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:07:03.288 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:07:03.303 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:07:03.303 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:07:03.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.303 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24961.0:152829 8.6 e207/172 8:6215dfc8:::benchmark_data_smithi066_100612_object25438:head v 207'9497, mlcod=207'9495)", 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:07:03.296894+0000", 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0055189109999999996, 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055506360000000003, 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.296894+0000", 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:07:03.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.296894+0000", 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.296894+0000", 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.296899+0000", 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1000000000000003e-06 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:07:03.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.296906+0000", 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9399999999999996e-06 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.296912+0000", 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.485e-06 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.300039+0000", 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031271519999999998 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.300061+0000", 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1895e-05 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24961.0:152836 8.6 e207/172)", 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:07:03.302093+0000", 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00031946900000000001, 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041774800000000001, 2024-03-26T03:07:03.307 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.302093+0000", 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.302093+0000", 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:07:03.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.302095+0000", 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3990000000000001e-06 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.302102+0000", 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1470000000000004e-06 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.302115+0000", 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3341e-05 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:07:03.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:03.302122+0000", 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0679999999999999e-06 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-03-26T03:07:03.310 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:07:03.323 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:07:03.342 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:03.362 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:07:03.388 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:03.403 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:07:03.450 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:07:03.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:03.493 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:07:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:03.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:03.531 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:07:03.532 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:07:03.629 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:03.670 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:03.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:03.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:03.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:03.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:03.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:04.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:04.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:07:04.053 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:07:04.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:04.098 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:07:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:07:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:07:04.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:04.222 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:07:04.308 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:07:04.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:04.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:07:04.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:07:04.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2383696560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:04.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:07:04.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:07:04.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 40235/321768 objects degraded (12.504%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2383696560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:07:04.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 40235/321768 objects degraded (12.504%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:04.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:04.463 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:07:04.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:04.487 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:07:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:07:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2383696560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:07:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:07:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:07:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:07:04.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 40235/321768 objects degraded (12.504%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:04.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:04.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:04.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:04.724 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:07:04.724 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:07:04.724 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T03:07:04.724 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-03-26T03:07:04.724 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-26T03:07:04.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:04.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:04.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:04.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:04.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:05.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:05.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:05.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:05.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:05.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:05.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:05 smithi192 ceph-mon[33110]: 8.2 scrub ok 2024-03-26T03:07:05.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:05 smithi192 ceph-mon[33110]: pgmap v353: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 1.28k op/s; 38978/314367 objects degraded (12.399%) 2024-03-26T03:07:05.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:05 smithi192 ceph-mon[33110]: 8.3 scrub ok 2024-03-26T03:07:05.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:05.318 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:05 smithi066 ceph-mon[29666]: 8.2 scrub ok 2024-03-26T03:07:05.318 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:05 smithi066 ceph-mon[29666]: pgmap v353: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 1.28k op/s; 38978/314367 objects degraded (12.399%) 2024-03-26T03:07:05.318 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:05 smithi066 ceph-mon[29666]: 8.3 scrub ok 2024-03-26T03:07:05.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:05 smithi066 ceph-mon[37920]: 8.2 scrub ok 2024-03-26T03:07:05.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:05 smithi066 ceph-mon[37920]: pgmap v353: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 1.28k op/s; 38978/314367 objects degraded (12.399%) 2024-03-26T03:07:05.318 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:05 smithi066 ceph-mon[37920]: 8.3 scrub ok 2024-03-26T03:07:05.343 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:05.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:05.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:05.583 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:05.622 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:05.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:05.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:05.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:05.896 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:05.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:05.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:06.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:06.139 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:06.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.188 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:06.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:06.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:06.308 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:07:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:06.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:06.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:06.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:06.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:06 smithi192 ceph-mon[33110]: 8.1 scrub starts 2024-03-26T03:07:06.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:06 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:07:06.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:06 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:07:06.540 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:06 smithi066 ceph-mon[29666]: 8.1 scrub starts 2024-03-26T03:07:06.540 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:06 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:07:06.540 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:06 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:07:06.540 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:06 smithi066 ceph-mon[37920]: 8.1 scrub starts 2024-03-26T03:07:06.541 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:06 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:07:06.541 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:06 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:07:06.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:06.578 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:06.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:06.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:06.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:06.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:06.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:06.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:06.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:07.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:07.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:07.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:07.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:07.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:07.377 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:07 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:07:07.377 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:07 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:07:07.377 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:07 smithi192 ceph-mon[33110]: pgmap v354: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 37780/302265 objects degraded (12.499%) 2024-03-26T03:07:07.377 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:07 smithi192 ceph-mon[33110]: 8.a scrub starts 2024-03-26T03:07:07.381 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:07.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:07:07.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:07:07.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[37920]: pgmap v354: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 37780/302265 objects degraded (12.499%) 2024-03-26T03:07:07.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[37920]: 8.a scrub starts 2024-03-26T03:07:07.429 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:07:07.430 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:07:07.430 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[29666]: pgmap v354: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 37780/302265 objects degraded (12.499%) 2024-03-26T03:07:07.430 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:07 smithi066 ceph-mon[29666]: 8.a scrub starts 2024-03-26T03:07:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:07.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:07.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:07.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:07.567 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:07:07.567 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:07:07.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:07.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:07.670 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:07.722 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:07.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:07.824 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:07.872 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:07.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:07.889 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:07.890 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:07.974 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:08.030 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:08.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:08.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:08.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: 8.1 scrub ok 2024-03-26T03:07:08.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:07:08.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:07:08.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:07:08.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:07:08.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:07:08.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:07:08.275 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:08.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:07:08.308 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: 8.1 scrub ok 2024-03-26T03:07:08.308 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:07:08.308 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:07:08.308 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:07:08.308 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:07:08.308 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:07:08.309 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: 8.1 scrub ok 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:07:08.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:07:08.318 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:08.420 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.473 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:08.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:08.574 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.616 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:08.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:08.678 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:07:08.718 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.757 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:08.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:08.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:08.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:08.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:08.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:09.046 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:09.148 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:09.154 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:07:09.198 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:09.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:09.341 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: pgmap v355: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 37780/302265 objects degraded (12.499%) 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: osdmap e208: 8 total, 7 up, 8 in 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/138159466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: from='client.35042 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 37780/302265 objects degraded (12.499%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:09.379 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:09.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:09.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:09.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:07:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: pgmap v355: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 37780/302265 objects degraded (12.499%) 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: osdmap e208: 8 total, 7 up, 8 in 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/138159466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: from='client.35042 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 37780/302265 objects degraded (12.499%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: pgmap v355: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 37780/302265 objects degraded (12.499%) 2024-03-26T03:07:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: osdmap e208: 8 total, 7 up, 8 in 2024-03-26T03:07:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/138159466' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: from='client.35042 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:07:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:07:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 37780/302265 objects degraded (12.499%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:09.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:09.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:09.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:07:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:07:09.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:09.853 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:09.893 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:09.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:09.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:10.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:10.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:10.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:10.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:10.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:10.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:10.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: 8.a scrub ok 2024-03-26T03:07:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:07:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: from='client.35042 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:07:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: osdmap e209: 8 total, 7 up, 8 in 2024-03-26T03:07:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:07:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:07:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:07:10.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:10 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:07:10.563 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:10.600 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: 8.a scrub ok 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: from='client.35042 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: osdmap e209: 8 total, 7 up, 8 in 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:07:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: 8.a scrub ok 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: from='client.35042 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: osdmap e209: 8 total, 7 up, 8 in 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:07:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:10 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:07:10.678 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:07:10.678 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:07:10.701 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:10.737 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:10.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:10.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:10.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:10.986 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:11.025 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:11.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:11.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:11.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:11.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:11.381 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:11 smithi192 ceph-mon[33110]: pgmap v358: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 35897/287793 objects degraded (12.473%) 2024-03-26T03:07:11.382 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:11 smithi192 ceph-mon[33110]: 8.5 scrub starts 2024-03-26T03:07:11.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:11.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:07:11.443 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:11.483 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:11 smithi066 ceph-mon[29666]: pgmap v358: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 35897/287793 objects degraded (12.473%) 2024-03-26T03:07:11.483 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:11 smithi066 ceph-mon[29666]: 8.5 scrub starts 2024-03-26T03:07:11.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:11 smithi066 ceph-mon[37920]: pgmap v358: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s; 35897/287793 objects degraded (12.473%) 2024-03-26T03:07:11.483 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:11 smithi066 ceph-mon[37920]: 8.5 scrub starts 2024-03-26T03:07:11.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.591 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:11.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:11.692 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.733 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:11.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:11.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:11.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:11.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:11.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:12.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:12.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:12.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:12.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:12.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:12.384 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:12.419 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:12 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:07:12.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:12 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:07:12.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:12.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:12.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:12 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:07:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:12 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:07:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:12 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:07:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:12 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:07:12.656 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:12.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:12.796 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:12.826 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:12.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:12.966 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:12.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:13.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:13.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:13.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:13.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:13.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:13.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:13 smithi192 ceph-mon[33110]: pgmap v359: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 35536/284538 objects degraded (12.489%) 2024-03-26T03:07:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:13 smithi192 ceph-mon[33110]: 8.5 scrub ok 2024-03-26T03:07:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:13 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:07:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:13 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:07:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:13 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:07:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:13 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:07:13.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[29666]: pgmap v359: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 35536/284538 objects degraded (12.489%) 2024-03-26T03:07:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[29666]: 8.5 scrub ok 2024-03-26T03:07:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:07:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[37920]: pgmap v359: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 35536/284538 objects degraded (12.489%) 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[37920]: 8.5 scrub ok 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:07:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:07:13.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:13.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:13.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:13.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:13.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:13.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:13.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:13.953 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:07:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:07:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:07:14.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:14.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:14.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[29666]: 5.1d deep-scrub starts 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[29666]: 5.1d deep-scrub ok 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[29666]: 8.d scrub starts 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 35536/284538 objects degraded (12.489%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[37920]: 5.1d deep-scrub starts 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[37920]: 5.1d deep-scrub ok 2024-03-26T03:07:14.208 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[37920]: 8.d scrub starts 2024-03-26T03:07:14.209 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 35536/284538 objects degraded (12.489%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:14.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:14.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:14.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:14.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:14 smithi192 ceph-mon[33110]: 5.1d deep-scrub starts 2024-03-26T03:07:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:14 smithi192 ceph-mon[33110]: 5.1d deep-scrub ok 2024-03-26T03:07:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:14 smithi192 ceph-mon[33110]: 8.d scrub starts 2024-03-26T03:07:14.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 35536/284538 objects degraded (12.489%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:14.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:14.592 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:14.627 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.644 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:14.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:14.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:14.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:14.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:14.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:14.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:15.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:15.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:15.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:15.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:15.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:15.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:15.450 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[29666]: pgmap v360: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 34361/277551 objects degraded (12.380%) 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[37920]: pgmap v360: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 34361/277551 objects degraded (12.380%) 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:07:15.451 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:07:15.452 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:15 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:07:15.452 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:15 smithi192 ceph-mon[33110]: pgmap v360: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 34361/277551 objects degraded (12.380%) 2024-03-26T03:07:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:15 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:07:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:15 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:07:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:15 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:07:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:15 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:07:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:15 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:07:15.506 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:15.584 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:07:15.585 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:07:15.585 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:07:15.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:15.629 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:07:15.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:15.676 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:07:15.719 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:07:15.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:07:15.762 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:07:15.763 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:07:15.763 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:07:15.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:15.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:15.922 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:15.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:15.998 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:07:16.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:16.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:16.160 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:16.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:16.309 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2400097372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: 8.d scrub ok 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:07:16.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3500871545' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2400097372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: 8.d scrub ok 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:07:16.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3500871545' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:07:16.361 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:16.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2400097372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: 8.d scrub ok 2024-03-26T03:07:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:07:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:07:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:07:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:07:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:07:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3500871545' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:07:16.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:16.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:16.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:07:16.662 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:07:16.738 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:16.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:16.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:16.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:16.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:16.927 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-26T03:07:17.021 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:17.066 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:17.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:17.210 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:17.312 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:17.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:17.457 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:17 smithi192 ceph-mon[33110]: pgmap v361: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 33175/265887 objects degraded (12.477%) 2024-03-26T03:07:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3500871545' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:07:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:17 smithi192 ceph-mon[33110]: osdmap e210: 8 total, 7 up, 8 in 2024-03-26T03:07:17.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:17 smithi066 ceph-mon[29666]: pgmap v361: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 33175/265887 objects degraded (12.477%) 2024-03-26T03:07:17.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3500871545' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:07:17.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:17 smithi066 ceph-mon[29666]: osdmap e210: 8 total, 7 up, 8 in 2024-03-26T03:07:17.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:17 smithi066 ceph-mon[37920]: pgmap v361: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 33175/265887 objects degraded (12.477%) 2024-03-26T03:07:17.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3500871545' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:07:17.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:17 smithi066 ceph-mon[37920]: osdmap e210: 8 total, 7 up, 8 in 2024-03-26T03:07:17.496 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:17.598 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.636 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:17.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:17.737 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:17.782 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:17.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:17.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:17.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:18.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:18.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:18.156 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:18.192 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:18.293 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:18.339 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:18.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:18.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:18.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:18.478 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:18 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:07:18.478 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:18 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:07:18.479 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:18 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:07:18.479 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:18 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:07:18.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:18 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:07:18.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:18 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:07:18.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:18.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:18.663 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:07:18.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:18.767 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:18.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:18.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:18.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:19.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:19.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:19.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:19.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:19.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:19.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:19.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:19.468 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:19.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:07:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-mon[33110]: pgmap v363: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 33175/265887 objects degraded (12.477%) 2024-03-26T03:07:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-mon[33110]: 5.11 deep-scrub starts 2024-03-26T03:07:19.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-mon[33110]: 5.11 deep-scrub ok 2024-03-26T03:07:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:07:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:07:19.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 33175/265887 objects degraded (12.477%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[29666]: pgmap v363: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 33175/265887 objects degraded (12.477%) 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[29666]: 5.11 deep-scrub starts 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[29666]: 5.11 deep-scrub ok 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 33175/265887 objects degraded (12.477%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[37920]: pgmap v363: 158 pgs: 1 active+clean+scrubbing, 28 active+undersized+degraded, 38 active+undersized, 91 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 33175/265887 objects degraded (12.477%) 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[37920]: 5.11 deep-scrub starts 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[37920]: 5.11 deep-scrub ok 2024-03-26T03:07:19.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:07:19.534 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:07:19.534 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 33175/265887 objects degraded (12.477%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T03:07:19.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:19.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:19.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:19.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:173565 8.6 8:642b4252:::benchmark_data_smithi066_100612_object46174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:07:19.736911+0000", 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.001510394, 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001542788, 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:19.740 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 173565 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736911+0000", 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:19.741 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736911+0000", 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736913+0000", 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8199999999999999e-06 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736918+0000", 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4870000000000002e-06 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.742 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736928+0000", 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3449999999999995e-06 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736936+0000", 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0399999999999993e-06 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.737162+0000", 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00022651799999999999 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.743 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.737275+0000", 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000112947 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.737396+0000", 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000121088 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.24961.0:173558 8.9 e210/172 8:9b184cc6:::benchmark_data_smithi066_100612_object46167:head v 210'10951, mlcod=210'10949)", 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:07:19.736273+0000", 2024-03-26T03:07:19.744 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0021484680000000002, 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002268347, 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736273+0000", 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736273+0000", 2024-03-26T03:07:19.745 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736273+0000", 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.79e-07 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736276+0000", 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3000000000000002e-06 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736279+0000", 2024-03-26T03:07:19.746 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.942e-06 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736282+0000", 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.948e-06 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736327+0000", 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5050999999999999e-05 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:07:19.736350+0000", 2024-03-26T03:07:19.747 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2427999999999999e-05 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 2 2024-03-26T03:07:19.748 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:07:19.840 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:19.882 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:19.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:19.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:20.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:20.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:20.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:20.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:20.292 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:20.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:20.444 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:07:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:07:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:07:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:07:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:07:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:20 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:07:20.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:07:20.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:20.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:07:20.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:07:20.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:07:20.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:20 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:07:20.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:20.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:20.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:20.822 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:20.863 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:20.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:20.878 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:20.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:07:20.965 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.016 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:21.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:21.117 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:21.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:21.280 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:21.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:21.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:21.476 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:21 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:07:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:21 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:07:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:21 smithi192 ceph-mon[33110]: pgmap v364: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 31070/249357 objects degraded (12.460%) 2024-03-26T03:07:21.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:21 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:07:21.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:21 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:07:21.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:21 smithi066 ceph-mon[29666]: pgmap v364: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 31070/249357 objects degraded (12.460%) 2024-03-26T03:07:21.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:21 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:07:21.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:21 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:07:21.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:21 smithi066 ceph-mon[37920]: pgmap v364: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 31070/249357 objects degraded (12.460%) 2024-03-26T03:07:21.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:21.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:21.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:21.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:21.871 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:21.912 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:21.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:22.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:22.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:22.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:22.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:22.238 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:07:22.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:22.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:22.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:22.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:22 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:07:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:22 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:07:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:22 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2665453107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:07:22.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:22.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:22.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:22 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:07:22.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:22 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:07:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:22 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2665453107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:07:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:22 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:07:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:22 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:07:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:22 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2665453107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:07:22.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:22.740 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:22.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:22.874 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:07:22.875 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:07:22.875 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T03:07:22.875 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:07:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:22.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:22.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:22.977 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.032 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:23.117 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:23.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:23.195 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:23.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:23.348 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:23.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:23.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:23 smithi192 ceph-mon[33110]: pgmap v365: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 30652/245487 objects degraded (12.486%) 2024-03-26T03:07:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:23 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:23 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2665453107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:07:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:23 smithi192 ceph-mon[33110]: osdmap e211: 8 total, 7 up, 8 in 2024-03-26T03:07:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:23.508 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[29666]: pgmap v365: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 30652/245487 objects degraded (12.486%) 2024-03-26T03:07:23.508 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:23.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2665453107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:07:23.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[29666]: osdmap e211: 8 total, 7 up, 8 in 2024-03-26T03:07:23.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[37920]: pgmap v365: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 30652/245487 objects degraded (12.486%) 2024-03-26T03:07:23.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:23.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2665453107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:07:23.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-mon[37920]: osdmap e211: 8 total, 7 up, 8 in 2024-03-26T03:07:23.590 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.633 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:23.734 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:23.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:23.871 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:23.920 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:23.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:24.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:24.060 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:07:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:07:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:07:24.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:24.201 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:24.303 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:24.343 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:24.444 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:24.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 30652/245487 objects degraded (12.486%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:24.560 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 30652/245487 objects degraded (12.486%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:24.560 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:24.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 30652/245487 objects degraded (12.486%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:24.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:24.589 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:24.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:24.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:24.782 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:24.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:24.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:24.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:25.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:25.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:25.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:07:25.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:07:25.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:07:25.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:07:25.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:07:25.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:07:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:07:25.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:25.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:25.283 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:25.295 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:25.325 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:25 smithi066 ceph-mon[29666]: pgmap v367: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s; 29389/237639 objects degraded (12.367%) 2024-03-26T03:07:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3676447811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:25 smithi066 ceph-mon[37920]: pgmap v367: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s; 29389/237639 objects degraded (12.367%) 2024-03-26T03:07:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3676447811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:25.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:25.473 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:25 smithi192 ceph-mon[33110]: pgmap v367: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s; 29389/237639 objects degraded (12.367%) 2024-03-26T03:07:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3676447811' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:25.553 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:07:25.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:25.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:25.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:25.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:25.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:25.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:25.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:25.977 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:26.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:26.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:26.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:26.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:26.301 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:26.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:26.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:26.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:26.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:26.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:26.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:26.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:26.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:26.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:26.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:26.982 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:26.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:27.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:27.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:27.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:27.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:27.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:27.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:27 smithi066 ceph-mon[29666]: pgmap v368: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s; 28081/224289 objects degraded (12.520%) 2024-03-26T03:07:27.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:27 smithi066 ceph-mon[37920]: pgmap v368: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s; 28081/224289 objects degraded (12.520%) 2024-03-26T03:07:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:27.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:07:27.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:07:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:27 smithi192 ceph-mon[33110]: pgmap v368: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.51k op/s; 28081/224289 objects degraded (12.520%) 2024-03-26T03:07:27.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:27.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:27.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:27.675 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:27.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:27.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:27.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:27.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:27.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:28.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:28.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:28.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:28.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:28.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:28.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:28.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:28.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:28.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:28.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:28.725 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:28.764 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:28.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:28.908 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:28.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:29.010 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:29.049 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:29.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:29.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:29.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:29.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:29.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:07:29.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:29.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:29.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:07:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:07:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: pgmap v369: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 28081/224289 objects degraded (12.520%) 2024-03-26T03:07:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: osdmap e212: 8 total, 7 up, 8 in 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 28081/224289 objects degraded (12.520%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: pgmap v369: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 28081/224289 objects degraded (12.520%) 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: osdmap e212: 8 total, 7 up, 8 in 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:07:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 28081/224289 objects degraded (12.520%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:29.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:29.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:07:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:07:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: pgmap v369: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 28081/224289 objects degraded (12.520%) 2024-03-26T03:07:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: osdmap e212: 8 total, 7 up, 8 in 2024-03-26T03:07:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:07:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:07:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 28081/224289 objects degraded (12.520%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:29.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:07:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:07:29.795 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:29.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:29.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:29.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:29.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:30.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:30.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:30.235 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:30.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:30.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:30.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:30.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:30.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:30 smithi066 ceph-mon[29666]: osdmap e213: 8 total, 7 up, 8 in 2024-03-26T03:07:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:30 smithi066 ceph-mon[37920]: osdmap e213: 8 total, 7 up, 8 in 2024-03-26T03:07:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:30.649 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:30.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:30.725 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:30 smithi192 ceph-mon[33110]: osdmap e213: 8 total, 7 up, 8 in 2024-03-26T03:07:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:30.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:30.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:30.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:30.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:30.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:31.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:31.116 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:31.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:31.220 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:07:31.224 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:07:31.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:31.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:31.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:31.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:31.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:31.579 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:31 smithi066 ceph-mon[29666]: pgmap v372: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s; 25871/206898 objects degraded (12.504%) 2024-03-26T03:07:31.579 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:31 smithi066 ceph-mon[37920]: pgmap v372: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s; 25871/206898 objects degraded (12.504%) 2024-03-26T03:07:31.625 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:31.659 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:31 smithi192 ceph-mon[33110]: pgmap v372: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s; 25871/206898 objects degraded (12.504%) 2024-03-26T03:07:31.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:31.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:31.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:31.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:31.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:31.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:32.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:32.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:32.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:32.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:32.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:32.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:32.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:32.493 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:32.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:32.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:32 smithi066 ceph-mon[29666]: from='client.35069 v1:172.21.15.66:0/2191151180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:07:32.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:32 smithi066 ceph-mon[37920]: from='client.35069 v1:172.21.15.66:0/2191151180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:07:32.634 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:32.736 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:32 smithi192 ceph-mon[33110]: from='client.35069 v1:172.21.15.66:0/2191151180' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:07:32.781 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:32.883 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:32.923 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:32.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:33.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:33.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:33.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:33.193 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-26T03:07:30.099866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4505004902,"num_objects":68966,"num_object_clones":0,"num_object_copies":206898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68966,"num_whiteouts":0,"num_read":1528,"num_read_kb":1257,"num_write":186685,"num_write_kb":8153098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":226876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84846,"ondisk_log_size":84846,"up":408,"acting":408,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16435660,"kb_used_data":14084276,"kb_used_omap":12,"kb_used_meta":2351219,"kb_avail":733525556,"statfs":{"total":767960285184,"available":751130169344,"internally_reserved":0,"allocated":14422298624,"data_stored":14419159165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2407648816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-842989568,"num_objects":-12863,"num_object_clones":0,"num_object_copies":-38589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12863,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12863,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.852393"},"pg_stats":[{"pgid":"8.4","version":"212'11664","reported_seq":11842,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.043053+0000","last_change":"2024-03-26T03:06:39.285387+0000","last_active":"2024-03-26T03:07:29.043053+0000","last_peered":"2024-03-26T03:07:29.043053+0000","last_clean":"2024-03-26T03:07:29.043053+0000","last_became_active":"2024-03-26T03:04:52.197080+0000","last_became_peered":"2024-03-26T03:04:52.197080+0000","last_unstale":"2024-03-26T03:07:29.043053+0000","last_undegraded":"2024-03-26T03:07:29.043053+0000","last_fullsized":"2024-03-26T03:07:29.043053+0000","mapping_epoch":172,"log_start":"197'6700","ondisk_log_start":"197'6700","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'8020","last_scrub_stamp":"2024-03-26T03:06:39.285146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:06:39.285146+0000","objects_scrubbed":7916,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:55.438174+0000","scrub_duration":2059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276430848,"num_objects":4218,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":508224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":446,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.257234+0000","last_change":"2024-03-26T03:07:12.760129+0000","last_active":"2024-03-26T03:07:28.257234+0000","last_peered":"2024-03-26T03:07:28.257234+0000","last_clean":"2024-03-26T03:07:28.257234+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:07:28.257234+0000","last_undegraded":"2024-03-26T03:07:28.257234+0000","last_fullsized":"2024-03-26T03:07:28.257234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:07:12.759985+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:07:12.759985+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:17.927368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"71'27","reported_seq":386,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.253547+0000","last_change":"2024-03-26T03:06:30.637298+0000","last_active":"2024-03-26T03:07:28.253547+0000","last_peered":"2024-03-26T03:07:28.253547+0000","last_clean":"2024-03-26T03:07:28.253547+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:07:28.253547+0000","last_undegraded":"2024-03-26T03:07:28.253547+0000","last_fullsized":"2024-03-26T03:07:28.253547+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'27","last_scrub_stamp":"2024-03-26T03:06:30.637139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:30.637139+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:34.284878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.258169+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:07:29.258169+0000","last_peered":"2024-03-26T03:07:29.258169+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:07:29.258169+0000","last_undegraded":"2024-03-26T03:07:29.258169+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":135,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.254722+0000","last_change":"2024-03-26T03:06:16.948621+0000","last_active":"2024-03-26T03:07:28.254722+0000","last_peered":"2024-03-26T03:07:28.254722+0000","last_clean":"2024-03-26T03:07:28.254722+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:07:28.254722+0000","last_undegraded":"2024-03-26T03:07:28.254722+0000","last_fullsized":"2024-03-26T03:07:28.254722+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:16.948507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.948507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"213'11839","reported_seq":11933,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.599211+0000","last_change":"2024-03-26T03:06:20.313342+0000","last_active":"2024-03-26T03:07:29.599211+0000","last_peered":"2024-03-26T03:07:29.599211+0000","last_clean":"2024-03-26T03:06:19.247674+0000","last_became_active":"2024-03-26T03:06:20.313342+0000","last_became_peered":"2024-03-26T03:06:20.313342+0000","last_unstale":"2024-03-26T03:07:29.599211+0000","last_undegraded":"2024-03-26T03:06:20.309749+0000","last_fullsized":"2024-03-26T03:06:20.309630+0000","mapping_epoch":198,"log_start":"199'7000","ondisk_log_start":"199'7000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275972096,"num_objects":4211,"num_object_clones":0,"num_object_copies":12633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11839,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":4211}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"91'20","reported_seq":369,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.239590+0000","last_change":"2024-03-26T03:07:07.792047+0000","last_active":"2024-03-26T03:07:22.239590+0000","last_peered":"2024-03-26T03:07:22.239590+0000","last_clean":"2024-03-26T03:07:22.239590+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:07:22.239590+0000","last_undegraded":"2024-03-26T03:07:22.239590+0000","last_fullsized":"2024-03-26T03:07:22.239590+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:07:07.791915+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_clean_scrub_stamp":"2024-03-26T03:07:07.791915+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:34.101942+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":339,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.267550+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:07:29.267550+0000","last_peered":"2024-03-26T03:07:29.267550+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:07:29.267550+0000","last_undegraded":"2024-03-26T03:07:29.267550+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.398593+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":323,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.263029+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:07:29.263029+0000","last_peered":"2024-03-26T03:07:29.263029+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:07:29.263029+0000","last_undegraded":"2024-03-26T03:07:29.263029+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":320,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.241984+0000","last_change":"2024-03-26T03:06:21.758333+0000","last_active":"2024-03-26T03:07:22.241984+0000","last_peered":"2024-03-26T03:07:22.241984+0000","last_clean":"2024-03-26T03:07:22.241984+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:07:22.241984+0000","last_undegraded":"2024-03-26T03:07:22.241984+0000","last_fullsized":"2024-03-26T03:07:22.241984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:21.758213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:06:21.758213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.939326+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"211'11309","reported_seq":11481,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:25.245510+0000","last_change":"2024-03-26T03:06:29.438805+0000","last_active":"2024-03-26T03:07:25.245510+0000","last_peered":"2024-03-26T03:07:25.245510+0000","last_clean":"2024-03-26T03:07:25.245510+0000","last_became_active":"2024-03-26T03:04:52.164468+0000","last_became_peered":"2024-03-26T03:04:52.164468+0000","last_unstale":"2024-03-26T03:07:25.245510+0000","last_undegraded":"2024-03-26T03:07:25.245510+0000","last_fullsized":"2024-03-26T03:07:25.245510+0000","mapping_epoch":172,"log_start":"194'5900","ondisk_log_start":"194'5900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'7499","last_scrub_stamp":"2024-03-26T03:06:29.438660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:06:29.438660+0000","objects_scrubbed":7404,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:34.386366+0000","scrub_duration":2723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301137920,"num_objects":4595,"num_object_clones":0,"num_object_copies":13785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11309,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"71'25","reported_seq":409,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.255100+0000","last_change":"2024-03-26T03:07:19.534345+0000","last_active":"2024-03-26T03:07:28.255100+0000","last_peered":"2024-03-26T03:07:28.255100+0000","last_clean":"2024-03-26T03:07:28.255100+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:07:28.255100+0000","last_undegraded":"2024-03-26T03:07:28.255100+0000","last_fullsized":"2024-03-26T03:07:28.255100+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:07:19.534200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:07:19.534200+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:43.772806+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":344,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.258913+0000","last_change":"2024-03-26T03:06:20.723735+0000","last_active":"2024-03-26T03:07:29.258913+0000","last_peered":"2024-03-26T03:07:29.258913+0000","last_clean":"2024-03-26T03:07:29.258913+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:07:29.258913+0000","last_undegraded":"2024-03-26T03:07:29.258913+0000","last_fullsized":"2024-03-26T03:07:29.258913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:20.723500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:06:20.723500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765533+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.257330+0000","last_change":"2024-03-26T03:06:17.721770+0000","last_active":"2024-03-26T03:07:29.257330+0000","last_peered":"2024-03-26T03:07:29.257330+0000","last_clean":"2024-03-26T03:07:29.257330+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:07:29.257330+0000","last_undegraded":"2024-03-26T03:07:29.257330+0000","last_fullsized":"2024-03-26T03:07:29.257330+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.721649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.721649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:46.570290+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":317,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.256677+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:07:28.256677+0000","last_peered":"2024-03-26T03:07:28.256677+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:07:28.256677+0000","last_undegraded":"2024-03-26T03:07:28.256677+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"213'14308","reported_seq":14607,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.604845+0000","last_change":"2024-03-26T03:07:07.380483+0000","last_active":"2024-03-26T03:07:29.604845+0000","last_peered":"2024-03-26T03:07:29.604845+0000","last_clean":"2024-03-26T03:07:29.604845+0000","last_became_active":"2024-03-26T03:04:52.196952+0000","last_became_peered":"2024-03-26T03:04:52.196952+0000","last_unstale":"2024-03-26T03:07:29.604845+0000","last_undegraded":"2024-03-26T03:07:29.604845+0000","last_fullsized":"2024-03-26T03:07:29.604845+0000","mapping_epoch":172,"log_start":"203'9402","ondisk_log_start":"203'9402","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12422","last_scrub_stamp":"2024-03-26T03:07:07.380185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:07:07.380185+0000","objects_scrubbed":6078,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:20.503131+0000","scrub_duration":1752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272367616,"num_objects":4156,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":505664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"153'25","reported_seq":373,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.254407+0000","last_change":"2024-03-26T03:07:21.717702+0000","last_active":"2024-03-26T03:07:28.254407+0000","last_peered":"2024-03-26T03:07:28.254407+0000","last_clean":"2024-03-26T03:07:28.254407+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:07:28.254407+0000","last_undegraded":"2024-03-26T03:07:28.254407+0000","last_fullsized":"2024-03-26T03:07:28.254407+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'25","last_scrub_stamp":"2024-03-26T03:07:21.717509+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:07:21.717509+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:34.423530+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.257592+0000","last_change":"2024-03-26T03:05:14.733614+0000","last_active":"2024-03-26T03:07:28.257592+0000","last_peered":"2024-03-26T03:07:28.257592+0000","last_clean":"2024-03-26T03:07:28.257592+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:07:28.257592+0000","last_undegraded":"2024-03-26T03:07:28.257592+0000","last_fullsized":"2024-03-26T03:07:28.257592+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:14.733500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:14.733500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266493+0000","last_change":"2024-03-26T03:07:07.572316+0000","last_active":"2024-03-26T03:07:29.266493+0000","last_peered":"2024-03-26T03:07:29.266493+0000","last_clean":"2024-03-26T03:07:29.266493+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:07:29.266493+0000","last_undegraded":"2024-03-26T03:07:29.266493+0000","last_fullsized":"2024-03-26T03:07:29.266493+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:07.572201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:07:07.572201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:34.957888+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":327,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.258071+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:07:29.258071+0000","last_peered":"2024-03-26T03:07:29.258071+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:07:29.258071+0000","last_undegraded":"2024-03-26T03:07:29.258071+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765451+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":405,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.256051+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:07:29.256051+0000","last_peered":"2024-03-26T03:07:29.256051+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:07:29.256051+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"212'11538","reported_seq":11636,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.084501+0000","last_change":"2024-03-26T03:06:20.350614+0000","last_active":"2024-03-26T03:07:29.084501+0000","last_peered":"2024-03-26T03:07:29.084501+0000","last_clean":"2024-03-26T03:06:19.254360+0000","last_became_active":"2024-03-26T03:06:20.326555+0000","last_became_peered":"2024-03-26T03:06:20.326555+0000","last_unstale":"2024-03-26T03:07:29.084501+0000","last_undegraded":"2024-03-26T03:06:20.279260+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"196'6500","ondisk_log_start":"196'6500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":5038,"log_dups_size":0,"ondisk_log_size":5038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.854573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11538,"num_write_kb":505536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4260}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":594,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.256709+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:07:28.256709+0000","last_peered":"2024-03-26T03:07:28.256709+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:07:28.256709+0000","last_undegraded":"2024-03-26T03:07:28.256709+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"71'10","reported_seq":347,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.247560+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:07:28.247560+0000","last_peered":"2024-03-26T03:07:28.247560+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:07:28.247560+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.272609+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.265967+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:07:29.265967+0000","last_peered":"2024-03-26T03:07:29.265967+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:07:29.265967+0000","last_undegraded":"2024-03-26T03:07:29.265967+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":448,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.257848+0000","last_change":"2024-03-26T03:06:33.798398+0000","last_active":"2024-03-26T03:07:29.257848+0000","last_peered":"2024-03-26T03:07:29.257848+0000","last_clean":"2024-03-26T03:07:29.257848+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:07:29.257848+0000","last_undegraded":"2024-03-26T03:07:29.257848+0000","last_fullsized":"2024-03-26T03:07:29.257848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:06:33.798197+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:00:20.452648+0000","last_clean_scrub_stamp":"2024-03-26T03:06:33.798197+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:50.365887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":321,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246630+0000","last_change":"2024-03-26T03:07:19.696579+0000","last_active":"2024-03-26T03:07:28.246630+0000","last_peered":"2024-03-26T03:07:28.246630+0000","last_clean":"2024-03-26T03:07:28.246630+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:07:28.246630+0000","last_undegraded":"2024-03-26T03:07:28.246630+0000","last_fullsized":"2024-03-26T03:07:28.246630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:19.696469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:19.696469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:46.435332+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"213'14966","reported_seq":15306,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.591495+0000","last_change":"2024-03-26T03:07:04.550142+0000","last_active":"2024-03-26T03:07:29.591495+0000","last_peered":"2024-03-26T03:07:29.591495+0000","last_clean":"2024-03-26T03:07:29.591495+0000","last_became_active":"2024-03-26T03:04:52.162480+0000","last_became_peered":"2024-03-26T03:04:52.162480+0000","last_unstale":"2024-03-26T03:07:29.591495+0000","last_undegraded":"2024-03-26T03:07:29.591495+0000","last_fullsized":"2024-03-26T03:07:29.591495+0000","mapping_epoch":172,"log_start":"202'10105","ondisk_log_start":"202'10105","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12971","last_scrub_stamp":"2024-03-26T03:07:04.549861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:07:04.549861+0000","objects_scrubbed":6302,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:23.386531+0000","scrub_duration":1947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277544960,"num_objects":4235,"num_object_clones":0,"num_object_copies":12705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11685,"num_write_kb":509440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":411,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.262928+0000","last_change":"2024-03-26T03:06:15.719086+0000","last_active":"2024-03-26T03:07:29.262928+0000","last_peered":"2024-03-26T03:07:29.262928+0000","last_clean":"2024-03-26T03:07:29.262928+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:07:29.262928+0000","last_undegraded":"2024-03-26T03:07:29.262928+0000","last_fullsized":"2024-03-26T03:07:29.262928+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:15.718776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:15.718776+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:34.635195+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"71'15","reported_seq":360,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.272200+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:07:29.272200+0000","last_peered":"2024-03-26T03:07:29.272200+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:07:29.272200+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781798+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":331,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.271416+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:07:29.271416+0000","last_peered":"2024-03-26T03:07:29.271416+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:07:29.271416+0000","last_undegraded":"2024-03-26T03:07:29.271416+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":436,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246517+0000","last_change":"2024-03-26T03:06:24.077578+0000","last_active":"2024-03-26T03:07:28.246517+0000","last_peered":"2024-03-26T03:07:28.246517+0000","last_clean":"2024-03-26T03:07:28.246517+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:07:28.246517+0000","last_undegraded":"2024-03-26T03:07:28.246517+0000","last_fullsized":"2024-03-26T03:07:28.246517+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:06:24.074065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:06:24.074065+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:30.486839+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":129,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266147+0000","last_change":"2024-03-26T03:06:16.213600+0000","last_active":"2024-03-26T03:07:29.266147+0000","last_peered":"2024-03-26T03:07:29.266147+0000","last_clean":"2024-03-26T03:07:29.266147+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:07:29.266147+0000","last_undegraded":"2024-03-26T03:07:29.266147+0000","last_fullsized":"2024-03-26T03:07:29.266147+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:16.213467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.213467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:39.503734+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"212'11499","reported_seq":11711,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.080638+0000","last_change":"2024-03-26T03:07:04.083653+0000","last_active":"2024-03-26T03:07:29.080638+0000","last_peered":"2024-03-26T03:07:29.080638+0000","last_clean":"2024-03-26T03:07:29.080638+0000","last_became_active":"2024-03-26T03:04:52.165202+0000","last_became_peered":"2024-03-26T03:04:52.165202+0000","last_unstale":"2024-03-26T03:07:29.080638+0000","last_undegraded":"2024-03-26T03:07:29.080638+0000","last_fullsized":"2024-03-26T03:07:29.080638+0000","mapping_epoch":172,"log_start":"196'6500","ondisk_log_start":"196'6500","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'9462","last_scrub_stamp":"2024-03-26T03:07:04.083349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:07:04.083349+0000","objects_scrubbed":6367,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:31.448181+0000","scrub_duration":1706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":12819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"71'17","reported_seq":368,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246246+0000","last_change":"2024-03-26T03:07:18.715044+0000","last_active":"2024-03-26T03:07:28.246246+0000","last_peered":"2024-03-26T03:07:28.246246+0000","last_clean":"2024-03-26T03:07:28.246246+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:07:28.246246+0000","last_undegraded":"2024-03-26T03:07:28.246246+0000","last_fullsized":"2024-03-26T03:07:28.246246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:07:18.714450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:07:18.714450+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:26.358256+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.256860+0000","last_change":"2024-03-26T03:05:17.724594+0000","last_active":"2024-03-26T03:07:28.256860+0000","last_peered":"2024-03-26T03:07:28.256860+0000","last_clean":"2024-03-26T03:07:28.256860+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:07:28.256860+0000","last_undegraded":"2024-03-26T03:07:28.256860+0000","last_fullsized":"2024-03-26T03:07:28.256860+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.724462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.724462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813405+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":133,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.247285+0000","last_change":"2024-03-26T03:06:22.001463+0000","last_active":"2024-03-26T03:07:28.247285+0000","last_peered":"2024-03-26T03:07:28.247285+0000","last_clean":"2024-03-26T03:07:28.247285+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:07:28.247285+0000","last_undegraded":"2024-03-26T03:07:28.247285+0000","last_fullsized":"2024-03-26T03:07:28.247285+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:22.001341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:06:22.001341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:43.134980+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.255513+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:07:28.255513+0000","last_peered":"2024-03-26T03:07:28.255513+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:07:28.255513+0000","last_undegraded":"2024-03-26T03:07:28.255513+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"212'11543","reported_seq":11699,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.061794+0000","last_change":"2024-03-26T03:07:15.477204+0000","last_active":"2024-03-26T03:07:29.061794+0000","last_peered":"2024-03-26T03:07:29.061794+0000","last_clean":"2024-03-26T03:07:29.061794+0000","last_became_active":"2024-03-26T03:04:52.196605+0000","last_became_peered":"2024-03-26T03:04:52.196605+0000","last_unstale":"2024-03-26T03:07:29.061794+0000","last_undegraded":"2024-03-26T03:07:29.061794+0000","last_fullsized":"2024-03-26T03:07:29.061794+0000","mapping_epoch":172,"log_start":"199'6900","ondisk_log_start":"199'6900","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'10315","last_scrub_stamp":"2024-03-26T03:07:15.476947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:07:15.476947+0000","objects_scrubbed":5477,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:32.278253+0000","scrub_duration":1700,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274464792,"num_objects":4189,"num_object_clones":0,"num_object_copies":12567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11543,"num_write_kb":503361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":141,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.265618+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:07:29.265618+0000","last_peered":"2024-03-26T03:07:29.265618+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:07:29.265618+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"71'20","reported_seq":371,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246469+0000","last_change":"2024-03-26T03:07:07.389919+0000","last_active":"2024-03-26T03:07:28.246469+0000","last_peered":"2024-03-26T03:07:28.246469+0000","last_clean":"2024-03-26T03:07:28.246469+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:07:28.246469+0000","last_undegraded":"2024-03-26T03:07:28.246469+0000","last_fullsized":"2024-03-26T03:07:28.246469+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'20","last_scrub_stamp":"2024-03-26T03:07:07.389776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:07:07.389776+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:33.320561+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":342,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.254639+0000","last_change":"2024-03-26T03:06:40.080041+0000","last_active":"2024-03-26T03:07:28.254639+0000","last_peered":"2024-03-26T03:07:28.254639+0000","last_clean":"2024-03-26T03:07:28.254639+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:07:28.254639+0000","last_undegraded":"2024-03-26T03:07:28.254639+0000","last_fullsized":"2024-03-26T03:07:28.254639+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:40.079926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:06:40.079926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:07.632094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":323,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.258066+0000","last_change":"2024-03-26T03:06:29.685173+0000","last_active":"2024-03-26T03:07:28.258066+0000","last_peered":"2024-03-26T03:07:28.258066+0000","last_clean":"2024-03-26T03:07:28.258066+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:07:28.258066+0000","last_undegraded":"2024-03-26T03:07:28.258066+0000","last_fullsized":"2024-03-26T03:07:28.258066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:29.685039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:29.685039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:55.800074+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":511,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.258861+0000","last_change":"2024-03-26T03:06:20.276636+0000","last_active":"2024-03-26T03:07:28.258861+0000","last_peered":"2024-03-26T03:07:28.258861+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:06:20.276636+0000","last_became_peered":"2024-03-26T03:06:20.276636+0000","last_unstale":"2024-03-26T03:07:28.258861+0000","last_undegraded":"2024-03-26T03:07:28.258861+0000","last_fullsized":"2024-03-26T03:06:20.268958+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:29.841533+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"211'11178","reported_seq":11267,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:25.257773+0000","last_change":"2024-03-26T03:06:20.302462+0000","last_active":"2024-03-26T03:07:25.257773+0000","last_peered":"2024-03-26T03:07:25.257773+0000","last_clean":"2024-03-26T03:06:19.253918+0000","last_became_active":"2024-03-26T03:06:20.302462+0000","last_became_peered":"2024-03-26T03:06:20.302462+0000","last_unstale":"2024-03-26T03:07:25.257773+0000","last_undegraded":"2024-03-26T03:06:20.298281+0000","last_fullsized":"2024-03-26T03:06:20.298150+0000","mapping_epoch":198,"log_start":"194'5800","ondisk_log_start":"194'5800","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296878080,"num_objects":4530,"num_object_clones":0,"num_object_copies":13590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":502656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":4530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":321,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.246843+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:07:28.246843+0000","last_peered":"2024-03-26T03:07:28.246843+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:07:28.246843+0000","last_undegraded":"2024-03-26T03:07:28.246843+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"84'42","reported_seq":143,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.257124+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:07:28.257124+0000","last_peered":"2024-03-26T03:07:28.257124+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:07:28.257124+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":338,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.267180+0000","last_change":"2024-03-26T03:06:31.300129+0000","last_active":"2024-03-26T03:07:29.267180+0000","last_peered":"2024-03-26T03:07:29.267180+0000","last_clean":"2024-03-26T03:07:29.267180+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:07:29.267180+0000","last_undegraded":"2024-03-26T03:07:29.267180+0000","last_fullsized":"2024-03-26T03:07:29.267180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:06:31.299819+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:54.411415+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":317,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.248057+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:07:28.248057+0000","last_peered":"2024-03-26T03:07:28.248057+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:07:28.248057+0000","last_undegraded":"2024-03-26T03:07:28.248057+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":409,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.253893+0000","last_change":"2024-03-26T03:06:35.171936+0000","last_active":"2024-03-26T03:07:28.253893+0000","last_peered":"2024-03-26T03:07:28.253893+0000","last_clean":"2024-03-26T03:07:28.253893+0000","last_became_active":"2024-03-26T03:04:41.967264+0000","last_became_peered":"2024-03-26T03:04:41.967264+0000","last_unstale":"2024-03-26T03:07:28.253893+0000","last_undegraded":"2024-03-26T03:07:28.253893+0000","last_fullsized":"2024-03-26T03:07:28.253893+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:06:35.171810+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:36.409817+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"213'11801","reported_seq":11896,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.433986+0000","last_change":"2024-03-26T03:06:20.315875+0000","last_active":"2024-03-26T03:07:29.433986+0000","last_peered":"2024-03-26T03:07:29.433986+0000","last_clean":"2024-03-26T03:06:19.226712+0000","last_became_active":"2024-03-26T03:06:20.315875+0000","last_became_peered":"2024-03-26T03:06:20.315875+0000","last_unstale":"2024-03-26T03:07:29.433986+0000","last_undegraded":"2024-03-26T03:06:20.301827+0000","last_fullsized":"2024-03-26T03:06:20.301710+0000","mapping_epoch":198,"log_start":"194'5800","ondisk_log_start":"194'5800","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":6001,"log_dups_size":0,"ondisk_log_size":6001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279117824,"num_objects":4259,"num_object_clones":0,"num_object_copies":12777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11801,"num_write_kb":513920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":4259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":335,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.255311+0000","last_change":"2024-03-26T03:07:17.449413+0000","last_active":"2024-03-26T03:07:28.255311+0000","last_peered":"2024-03-26T03:07:28.255311+0000","last_clean":"2024-03-26T03:07:28.255311+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:07:28.255311+0000","last_undegraded":"2024-03-26T03:07:28.255311+0000","last_fullsized":"2024-03-26T03:07:28.255311+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:17.449304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:17.449304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:29.963854+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"71'25","reported_seq":385,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.239356+0000","last_change":"2024-03-26T03:06:38.795661+0000","last_active":"2024-03-26T03:07:22.239356+0000","last_peered":"2024-03-26T03:07:22.239356+0000","last_clean":"2024-03-26T03:07:22.239356+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:07:22.239356+0000","last_undegraded":"2024-03-26T03:07:22.239356+0000","last_fullsized":"2024-03-26T03:07:22.239356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:06:38.795471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:38.795471+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:52.868003+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":335,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.261430+0000","last_change":"2024-03-26T03:06:33.716494+0000","last_active":"2024-03-26T03:07:28.261430+0000","last_peered":"2024-03-26T03:07:28.261430+0000","last_clean":"2024-03-26T03:07:28.261430+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:07:28.261430+0000","last_undegraded":"2024-03-26T03:07:28.261430+0000","last_fullsized":"2024-03-26T03:07:28.261430+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:33.716384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:33.716384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:03.535797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":325,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.271561+0000","last_change":"2024-03-26T03:05:37.842641+0000","last_active":"2024-03-26T03:07:29.271561+0000","last_peered":"2024-03-26T03:07:29.271561+0000","last_clean":"2024-03-26T03:07:29.271561+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:07:29.271561+0000","last_undegraded":"2024-03-26T03:07:29.271561+0000","last_fullsized":"2024-03-26T03:07:29.271561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:37.842529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.842529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781757+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":410,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.254909+0000","last_change":"2024-03-26T03:06:20.308944+0000","last_active":"2024-03-26T03:07:28.254909+0000","last_peered":"2024-03-26T03:07:28.254909+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:06:20.308944+0000","last_became_peered":"2024-03-26T03:06:20.308944+0000","last_unstale":"2024-03-26T03:07:28.254909+0000","last_undegraded":"2024-03-26T03:07:28.254909+0000","last_fullsized":"2024-03-26T03:06:20.270592+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"213'11864","reported_seq":12071,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.581152+0000","last_change":"2024-03-26T03:07:08.864848+0000","last_active":"2024-03-26T03:07:29.581152+0000","last_peered":"2024-03-26T03:07:29.581152+0000","last_clean":"2024-03-26T03:07:29.581152+0000","last_became_active":"2024-03-26T03:04:52.198372+0000","last_became_peered":"2024-03-26T03:04:52.198372+0000","last_unstale":"2024-03-26T03:07:29.581152+0000","last_undegraded":"2024-03-26T03:07:29.581152+0000","last_fullsized":"2024-03-26T03:07:29.581152+0000","mapping_epoch":172,"log_start":"199'7000","ondisk_log_start":"199'7000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10046","last_scrub_stamp":"2024-03-26T03:07:08.864701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:07:08.864701+0000","objects_scrubbed":6019,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:22.688247+0000","scrub_duration":2262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270008320,"num_objects":4120,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11864,"num_write_kb":511488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.251695+0000","last_change":"2024-03-26T03:07:02.696377+0000","last_active":"2024-03-26T03:07:28.251695+0000","last_peered":"2024-03-26T03:07:28.251695+0000","last_clean":"2024-03-26T03:07:28.251695+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:07:28.251695+0000","last_undegraded":"2024-03-26T03:07:28.251695+0000","last_fullsized":"2024-03-26T03:07:28.251695+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:02.696265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:02.696265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:08.815633+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"71'12","reported_seq":354,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.240697+0000","last_change":"2024-03-26T03:06:31.880248+0000","last_active":"2024-03-26T03:07:22.240697+0000","last_peered":"2024-03-26T03:07:22.240697+0000","last_clean":"2024-03-26T03:07:22.240697+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:07:22.240697+0000","last_undegraded":"2024-03-26T03:07:22.240697+0000","last_fullsized":"2024-03-26T03:07:22.240697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:06:31.880026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:31.880026+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:49.923220+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":335,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.256682+0000","last_change":"2024-03-26T03:05:28.667880+0000","last_active":"2024-03-26T03:07:28.256682+0000","last_peered":"2024-03-26T03:07:28.256682+0000","last_clean":"2024-03-26T03:07:28.256682+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:07:28.256682+0000","last_undegraded":"2024-03-26T03:07:28.256682+0000","last_fullsized":"2024-03-26T03:07:28.256682+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:28.667767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:28.667767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813468+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":343,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.241647+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:07:22.241647+0000","last_peered":"2024-03-26T03:07:22.241647+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:07:22.241647+0000","last_undegraded":"2024-03-26T03:07:22.241647+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:44.927039+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":328,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.267063+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:07:29.267063+0000","last_peered":"2024-03-26T03:07:29.267063+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:07:29.267063+0000","last_undegraded":"2024-03-26T03:07:29.267063+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:39.250104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":411,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.265884+0000","last_change":"2024-03-26T03:06:28.248735+0000","last_active":"2024-03-26T03:07:29.265884+0000","last_peered":"2024-03-26T03:07:29.265884+0000","last_clean":"2024-03-26T03:07:29.265884+0000","last_became_active":"2024-03-26T03:04:41.964807+0000","last_became_peered":"2024-03-26T03:04:41.964807+0000","last_unstale":"2024-03-26T03:07:29.265884+0000","last_undegraded":"2024-03-26T03:07:29.265884+0000","last_fullsized":"2024-03-26T03:07:29.265884+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:06:28.248568+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:51.297534+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"212'11782","reported_seq":11962,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.157776+0000","last_change":"2024-03-26T03:06:36.113550+0000","last_active":"2024-03-26T03:07:29.157776+0000","last_peered":"2024-03-26T03:07:29.157776+0000","last_clean":"2024-03-26T03:07:29.157776+0000","last_became_active":"2024-03-26T03:04:52.166591+0000","last_became_peered":"2024-03-26T03:04:52.166591+0000","last_unstale":"2024-03-26T03:07:29.157776+0000","last_undegraded":"2024-03-26T03:07:29.157776+0000","last_fullsized":"2024-03-26T03:07:29.157776+0000","mapping_epoch":172,"log_start":"195'6100","ondisk_log_start":"195'6100","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'8012","last_scrub_stamp":"2024-03-26T03:06:36.113262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:06:36.113262+0000","objects_scrubbed":7960,"log_size":5682,"log_dups_size":0,"ondisk_log_size":5682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:42.287260+0000","scrub_duration":2187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280494080,"num_objects":4280,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":327,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.259286+0000","last_change":"2024-03-26T03:05:20.645163+0000","last_active":"2024-03-26T03:07:28.259286+0000","last_peered":"2024-03-26T03:07:28.259286+0000","last_clean":"2024-03-26T03:07:28.259286+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:07:28.259286+0000","last_undegraded":"2024-03-26T03:07:28.259286+0000","last_fullsized":"2024-03-26T03:07:28.259286+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.645063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.645063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"71'16","reported_seq":456,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.255364+0000","last_change":"2024-03-26T03:05:21.760217+0000","last_active":"2024-03-26T03:07:28.255364+0000","last_peered":"2024-03-26T03:07:28.255364+0000","last_clean":"2024-03-26T03:07:28.255364+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:07:28.255364+0000","last_undegraded":"2024-03-26T03:07:28.255364+0000","last_fullsized":"2024-03-26T03:07:28.255364+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:21.760091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:21.760091+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":592,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.256788+0000","last_change":"2024-03-26T03:07:03.694914+0000","last_active":"2024-03-26T03:07:28.256788+0000","last_peered":"2024-03-26T03:07:28.256788+0000","last_clean":"2024-03-26T03:07:28.256788+0000","last_became_active":"2024-03-26T03:04:37.115068+0000","last_became_peered":"2024-03-26T03:04:37.115068+0000","last_unstale":"2024-03-26T03:07:28.256788+0000","last_undegraded":"2024-03-26T03:07:28.256788+0000","last_fullsized":"2024-03-26T03:07:28.256788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:07:03.694789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:07:03.694789+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:14.447910+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":356,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.252490+0000","last_change":"2024-03-26T03:07:01.640693+0000","last_active":"2024-03-26T03:07:28.252490+0000","last_peered":"2024-03-26T03:07:28.252490+0000","last_clean":"2024-03-26T03:07:28.252490+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:07:28.252490+0000","last_undegraded":"2024-03-26T03:07:28.252490+0000","last_fullsized":"2024-03-26T03:07:28.252490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:01.640581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:07:01.640581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:18.436364+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":61,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.253588+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:07:28.253588+0000","last_peered":"2024-03-26T03:07:28.253588+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:07:28.253588+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.854624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":324,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.251786+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:07:28.251786+0000","last_peered":"2024-03-26T03:07:28.251786+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:07:28.251786+0000","last_undegraded":"2024-03-26T03:07:28.251786+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.848365+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"212'11529","reported_seq":11621,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.056279+0000","last_change":"2024-03-26T03:06:20.306542+0000","last_active":"2024-03-26T03:07:29.056279+0000","last_peered":"2024-03-26T03:07:29.056279+0000","last_clean":"2024-03-26T03:06:19.184991+0000","last_became_active":"2024-03-26T03:06:20.306542+0000","last_became_peered":"2024-03-26T03:06:20.306542+0000","last_unstale":"2024-03-26T03:07:29.056279+0000","last_undegraded":"2024-03-26T03:06:20.262377+0000","last_fullsized":"2024-03-26T03:06:20.262239+0000","mapping_epoch":198,"log_start":"197'6600","ondisk_log_start":"197'6600","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277282816,"num_objects":4231,"num_object_clones":0,"num_object_copies":12693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11529,"num_write_kb":504320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":4231}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.258462+0000","last_change":"2024-03-26T03:07:05.893370+0000","last_active":"2024-03-26T03:07:29.258462+0000","last_peered":"2024-03-26T03:07:29.258462+0000","last_clean":"2024-03-26T03:07:29.258462+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:07:29.258462+0000","last_undegraded":"2024-03-26T03:07:29.258462+0000","last_fullsized":"2024-03-26T03:07:29.258462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:05.893254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:05.893254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:32.726633+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"71'19","reported_seq":371,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.273873+0000","last_change":"2024-03-26T03:06:30.266875+0000","last_active":"2024-03-26T03:07:29.273873+0000","last_peered":"2024-03-26T03:07:29.273873+0000","last_clean":"2024-03-26T03:07:29.273873+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:07:29.273873+0000","last_undegraded":"2024-03-26T03:07:29.273873+0000","last_fullsized":"2024-03-26T03:07:29.273873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'19","last_scrub_stamp":"2024-03-26T03:06:30.266731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:30.266731+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:35.336210+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":340,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.257355+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:07:29.257355+0000","last_peered":"2024-03-26T03:07:29.257355+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:07:29.257355+0000","last_undegraded":"2024-03-26T03:07:29.257355+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765464+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":858,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266512+0000","last_change":"2024-03-26T03:07:08.620533+0000","last_active":"2024-03-26T03:07:29.266512+0000","last_peered":"2024-03-26T03:07:29.266512+0000","last_clean":"2024-03-26T03:07:29.266512+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:07:29.266512+0000","last_undegraded":"2024-03-26T03:07:29.266512+0000","last_fullsized":"2024-03-26T03:07:29.266512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:07:08.620406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:07:08.620406+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:29.648805+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":503,"num_read_kb":498,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":326,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.261279+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:07:28.261279+0000","last_peered":"2024-03-26T03:07:28.261279+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:07:28.261279+0000","last_undegraded":"2024-03-26T03:07:28.261279+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813566+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":599,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.257251+0000","last_change":"2024-03-26T03:07:09.779059+0000","last_active":"2024-03-26T03:07:28.257251+0000","last_peered":"2024-03-26T03:07:28.257251+0000","last_clean":"2024-03-26T03:07:28.257251+0000","last_became_active":"2024-03-26T03:04:41.965091+0000","last_became_peered":"2024-03-26T03:04:41.965091+0000","last_unstale":"2024-03-26T03:07:28.257251+0000","last_undegraded":"2024-03-26T03:07:28.257251+0000","last_fullsized":"2024-03-26T03:07:28.257251+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:07:09.778930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:07:09.778930+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:32.394946+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"212'11862","reported_seq":11954,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.071560+0000","last_change":"2024-03-26T03:06:20.281135+0000","last_active":"2024-03-26T03:07:29.071560+0000","last_peered":"2024-03-26T03:07:29.071560+0000","last_clean":"2024-03-26T03:06:19.240574+0000","last_became_active":"2024-03-26T03:06:20.281135+0000","last_became_peered":"2024-03-26T03:06:20.281135+0000","last_unstale":"2024-03-26T03:07:29.071560+0000","last_undegraded":"2024-03-26T03:06:20.278998+0000","last_fullsized":"2024-03-26T03:06:20.278879+0000","mapping_epoch":198,"log_start":"197'6800","ondisk_log_start":"197'6800","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:04:51.137347+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.854597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576576,"num_objects":4266,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":516096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":4266}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.240084+0000","last_change":"2024-03-26T03:07:05.686650+0000","last_active":"2024-03-26T03:07:22.240084+0000","last_peered":"2024-03-26T03:07:22.240084+0000","last_clean":"2024-03-26T03:07:22.240084+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:07:22.240084+0000","last_undegraded":"2024-03-26T03:07:22.240084+0000","last_fullsized":"2024-03-26T03:07:22.240084+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:05.686352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:05.686352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:23.213151+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"199'46","reported_seq":511,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.265875+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:07:29.265875+0000","last_peered":"2024-03-26T03:07:29.265875+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:07:29.265875+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781718+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":129,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.265777+0000","last_change":"2024-03-26T03:05:43.795666+0000","last_active":"2024-03-26T03:07:29.265777+0000","last_peered":"2024-03-26T03:07:29.265777+0000","last_clean":"2024-03-26T03:07:29.265777+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:07:29.265777+0000","last_undegraded":"2024-03-26T03:07:29.265777+0000","last_fullsized":"2024-03-26T03:07:29.265777+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.795558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.795558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781784+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.246721+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:07:28.246721+0000","last_peered":"2024-03-26T03:07:28.246721+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:07:28.246721+0000","last_undegraded":"2024-03-26T03:07:28.246721+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":413,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.241543+0000","last_change":"2024-03-26T03:06:14.728064+0000","last_active":"2024-03-26T03:07:22.241543+0000","last_peered":"2024-03-26T03:07:22.241543+0000","last_clean":"2024-03-26T03:07:22.241543+0000","last_became_active":"2024-03-26T03:04:41.966771+0000","last_became_peered":"2024-03-26T03:04:41.966771+0000","last_unstale":"2024-03-26T03:07:22.241543+0000","last_undegraded":"2024-03-26T03:07:22.241543+0000","last_fullsized":"2024-03-26T03:07:22.241543+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:14.727935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.727935+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:27.935867+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"211'11403","reported_seq":11578,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:25.272087+0000","last_change":"2024-03-26T03:06:33.647543+0000","last_active":"2024-03-26T03:07:25.272087+0000","last_peered":"2024-03-26T03:07:25.272087+0000","last_clean":"2024-03-26T03:07:25.272087+0000","last_became_active":"2024-03-26T03:04:52.197016+0000","last_became_peered":"2024-03-26T03:04:52.197016+0000","last_unstale":"2024-03-26T03:07:25.272087+0000","last_undegraded":"2024-03-26T03:07:25.272087+0000","last_fullsized":"2024-03-26T03:07:25.272087+0000","mapping_epoch":172,"log_start":"195'6000","ondisk_log_start":"195'6000","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'7844","last_scrub_stamp":"2024-03-26T03:06:33.647113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:06:33.647113+0000","objects_scrubbed":7746,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:37.958709+0000","scrub_duration":2906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304545792,"num_objects":4647,"num_object_clones":0,"num_object_copies":13941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":446,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.239830+0000","last_change":"2024-03-26T03:07:03.684805+0000","last_active":"2024-03-26T03:07:22.239830+0000","last_peered":"2024-03-26T03:07:22.239830+0000","last_clean":"2024-03-26T03:07:22.239830+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:07:22.239830+0000","last_undegraded":"2024-03-26T03:07:22.239830+0000","last_fullsized":"2024-03-26T03:07:22.239830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:07:03.684687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:03.684687+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:31.789456+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"92'22","reported_seq":366,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266281+0000","last_change":"2024-03-26T03:07:11.716674+0000","last_active":"2024-03-26T03:07:29.266281+0000","last_peered":"2024-03-26T03:07:29.266281+0000","last_clean":"2024-03-26T03:07:29.266281+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:07:29.266281+0000","last_undegraded":"2024-03-26T03:07:29.266281+0000","last_fullsized":"2024-03-26T03:07:29.266281+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:07:11.716531+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:07:11.716531+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:14.690678+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":337,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266096+0000","last_change":"2024-03-26T03:06:26.221766+0000","last_active":"2024-03-26T03:07:29.266096+0000","last_peered":"2024-03-26T03:07:29.266096+0000","last_clean":"2024-03-26T03:07:29.266096+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:07:29.266096+0000","last_undegraded":"2024-03-26T03:07:29.266096+0000","last_fullsized":"2024-03-26T03:07:29.266096+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:26.221640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:26.221640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:48.697167+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.237646+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:07:22.237646+0000","last_peered":"2024-03-26T03:07:22.237646+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:07:22.237646+0000","last_undegraded":"2024-03-26T03:07:22.237646+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":410,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.254529+0000","last_change":"2024-03-26T03:06:20.272249+0000","last_active":"2024-03-26T03:07:28.254529+0000","last_peered":"2024-03-26T03:07:28.254529+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:06:20.272249+0000","last_became_peered":"2024-03-26T03:06:20.272249+0000","last_unstale":"2024-03-26T03:07:28.254529+0000","last_undegraded":"2024-03-26T03:07:28.254529+0000","last_fullsized":"2024-03-26T03:06:20.268693+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"212'14527","reported_seq":14824,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.126110+0000","last_change":"2024-03-26T03:07:12.354709+0000","last_active":"2024-03-26T03:07:29.126110+0000","last_peered":"2024-03-26T03:07:29.126110+0000","last_clean":"2024-03-26T03:07:29.126110+0000","last_became_active":"2024-03-26T03:04:52.164107+0000","last_became_peered":"2024-03-26T03:04:52.164107+0000","last_unstale":"2024-03-26T03:07:29.126110+0000","last_undegraded":"2024-03-26T03:07:29.126110+0000","last_fullsized":"2024-03-26T03:07:29.126110+0000","mapping_epoch":172,"log_start":"201'8801","ondisk_log_start":"201'8801","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'13032","last_scrub_stamp":"2024-03-26T03:07:12.354317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:51.137347+0000","last_clean_scrub_stamp":"2024-03-26T03:07:12.354317+0000","objects_scrubbed":5835,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:31.448525+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279642112,"num_objects":4267,"num_object_clones":0,"num_object_copies":12801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":517312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.256284+0000","last_change":"2024-03-26T03:06:16.715283+0000","last_active":"2024-03-26T03:07:29.256284+0000","last_peered":"2024-03-26T03:07:29.256284+0000","last_clean":"2024-03-26T03:07:29.256284+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:07:29.256284+0000","last_undegraded":"2024-03-26T03:07:29.256284+0000","last_fullsized":"2024-03-26T03:07:29.256284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:16.715164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.715164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765507+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"71'24","reported_seq":393,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.246651+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:07:28.246651+0000","last_peered":"2024-03-26T03:07:28.246651+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:07:28.246651+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":351,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.247191+0000","last_change":"2024-03-26T03:06:12.941656+0000","last_active":"2024-03-26T03:07:28.247191+0000","last_peered":"2024-03-26T03:07:28.247191+0000","last_clean":"2024-03-26T03:07:28.247191+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:07:28.247191+0000","last_undegraded":"2024-03-26T03:07:28.247191+0000","last_fullsized":"2024-03-26T03:07:28.247191+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:06:12.941527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:12.941527+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.257671+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:07:28.257671+0000","last_peered":"2024-03-26T03:07:28.257671+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:07:28.257671+0000","last_undegraded":"2024-03-26T03:07:28.257671+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"71'21","reported_seq":371,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:22.240803+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:07:22.240803+0000","last_peered":"2024-03-26T03:07:22.240803+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:07:22.240803+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:25.936438+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":350,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.256171+0000","last_change":"2024-03-26T03:06:22.678380+0000","last_active":"2024-03-26T03:07:29.256171+0000","last_peered":"2024-03-26T03:07:29.256171+0000","last_clean":"2024-03-26T03:07:29.256171+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:07:29.256171+0000","last_undegraded":"2024-03-26T03:07:29.256171+0000","last_fullsized":"2024-03-26T03:07:29.256171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:06:22.678209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:22.678209+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.936520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.272411+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:07:29.272411+0000","last_peered":"2024-03-26T03:07:29.272411+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:07:29.272411+0000","last_undegraded":"2024-03-26T03:07:29.272411+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":332,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.258675+0000","last_change":"2024-03-26T03:07:02.883491+0000","last_active":"2024-03-26T03:07:29.258675+0000","last_peered":"2024-03-26T03:07:29.258675+0000","last_clean":"2024-03-26T03:07:29.258675+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:07:29.258675+0000","last_undegraded":"2024-03-26T03:07:29.258675+0000","last_fullsized":"2024-03-26T03:07:29.258675+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:02.883383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:02.883383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:04.012726+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"71'12","reported_seq":75,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.259029+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:07:28.259029+0000","last_peered":"2024-03-26T03:07:28.259029+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:07:28.259029+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.256658+0000","last_change":"2024-03-26T03:05:31.682849+0000","last_active":"2024-03-26T03:07:28.256658+0000","last_peered":"2024-03-26T03:07:28.256658+0000","last_clean":"2024-03-26T03:07:28.256658+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:07:28.256658+0000","last_undegraded":"2024-03-26T03:07:28.256658+0000","last_fullsized":"2024-03-26T03:07:28.256658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:31.682729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:31.682729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":328,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.251998+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:07:28.251998+0000","last_peered":"2024-03-26T03:07:28.251998+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:07:28.251998+0000","last_undegraded":"2024-03-26T03:07:28.251998+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.854610+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":314,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.260955+0000","last_change":"2024-03-26T03:05:16.754129+0000","last_active":"2024-03-26T03:07:28.260955+0000","last_peered":"2024-03-26T03:07:28.260955+0000","last_clean":"2024-03-26T03:07:28.260955+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:07:28.260955+0000","last_undegraded":"2024-03-26T03:07:28.260955+0000","last_fullsized":"2024-03-26T03:07:28.260955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.754027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.754027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813379+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"151'29","reported_seq":270,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.266278+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:07:29.266278+0000","last_peered":"2024-03-26T03:07:29.266278+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:07:29.266278+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":344,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.253385+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:07:28.253385+0000","last_peered":"2024-03-26T03:07:28.253385+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:07:28.253385+0000","last_undegraded":"2024-03-26T03:07:28.253385+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.095183+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":445,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.252257+0000","last_change":"2024-03-26T03:06:36.601994+0000","last_active":"2024-03-26T03:07:28.252257+0000","last_peered":"2024-03-26T03:07:28.252257+0000","last_clean":"2024-03-26T03:07:28.252257+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:07:28.252257+0000","last_undegraded":"2024-03-26T03:07:28.252257+0000","last_fullsized":"2024-03-26T03:07:28.252257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:06:36.601857+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:49.917386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.240479+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:07:22.240479+0000","last_peered":"2024-03-26T03:07:22.240479+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:07:22.240479+0000","last_undegraded":"2024-03-26T03:07:22.240479+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"71'15","reported_seq":169,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.261084+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:07:28.261084+0000","last_peered":"2024-03-26T03:07:28.261084+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:07:28.261084+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.239411+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:07:22.239411+0000","last_peered":"2024-03-26T03:07:22.239411+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:07:22.239411+0000","last_undegraded":"2024-03-26T03:07:22.239411+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":75,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.259366+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:07:28.259366+0000","last_peered":"2024-03-26T03:07:28.259366+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:07:28.259366+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":322,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.247340+0000","last_change":"2024-03-26T03:07:09.469149+0000","last_active":"2024-03-26T03:07:28.247340+0000","last_peered":"2024-03-26T03:07:28.247340+0000","last_clean":"2024-03-26T03:07:28.247340+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:07:28.247340+0000","last_undegraded":"2024-03-26T03:07:28.247340+0000","last_fullsized":"2024-03-26T03:07:28.247340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:07:09.469023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:09.469023+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:20.270032+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"71'9","reported_seq":348,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266172+0000","last_change":"2024-03-26T03:07:12.685440+0000","last_active":"2024-03-26T03:07:29.266172+0000","last_peered":"2024-03-26T03:07:29.266172+0000","last_clean":"2024-03-26T03:07:29.266172+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:07:29.266172+0000","last_undegraded":"2024-03-26T03:07:29.266172+0000","last_fullsized":"2024-03-26T03:07:29.266172+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-26T03:07:12.685186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:07:12.685186+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:21.542366+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.255691+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:07:28.255691+0000","last_peered":"2024-03-26T03:07:28.255691+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:07:28.255691+0000","last_undegraded":"2024-03-26T03:07:28.255691+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.239696+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:07:22.239696+0000","last_peered":"2024-03-26T03:07:22.239696+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:07:22.239696+0000","last_undegraded":"2024-03-26T03:07:22.239696+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.237520+0000","last_change":"2024-03-26T03:06:34.729800+0000","last_active":"2024-03-26T03:07:22.237520+0000","last_peered":"2024-03-26T03:07:22.237520+0000","last_clean":"2024-03-26T03:07:22.237520+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:07:22.237520+0000","last_undegraded":"2024-03-26T03:07:22.237520+0000","last_fullsized":"2024-03-26T03:07:22.237520+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:34.729641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:34.729641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:45.620668+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"71'11","reported_seq":348,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246871+0000","last_change":"2024-03-26T03:06:33.114843+0000","last_active":"2024-03-26T03:07:28.246871+0000","last_peered":"2024-03-26T03:07:28.246871+0000","last_clean":"2024-03-26T03:07:28.246871+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:07:28.246871+0000","last_undegraded":"2024-03-26T03:07:28.246871+0000","last_fullsized":"2024-03-26T03:07:28.246871+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:06:33.114727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:33.114727+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:54.627000+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":344,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266460+0000","last_change":"2024-03-26T03:07:18.761462+0000","last_active":"2024-03-26T03:07:29.266460+0000","last_peered":"2024-03-26T03:07:29.266460+0000","last_clean":"2024-03-26T03:07:29.266460+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:07:29.266460+0000","last_undegraded":"2024-03-26T03:07:29.266460+0000","last_fullsized":"2024-03-26T03:07:29.266460+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:18.761352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:07:18.761352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:26.975870+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":323,"reported_epoch":213,"state":"active+undersized","last_fresh":"2024-03-26T03:07:29.266421+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:07:29.266421+0000","last_peered":"2024-03-26T03:07:29.266421+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:07:29.266421+0000","last_undegraded":"2024-03-26T03:07:29.266421+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781692+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.241694+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:07:22.241694+0000","last_peered":"2024-03-26T03:07:22.241694+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:07:22.241694+0000","last_undegraded":"2024-03-26T03:07:22.241694+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"71'4","reported_seq":62,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.266228+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:07:29.266228+0000","last_peered":"2024-03-26T03:07:29.266228+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:07:29.266228+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":341,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.251449+0000","last_change":"2024-03-26T03:06:28.633565+0000","last_active":"2024-03-26T03:07:28.251449+0000","last_peered":"2024-03-26T03:07:28.251449+0000","last_clean":"2024-03-26T03:07:28.251449+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:07:28.251449+0000","last_undegraded":"2024-03-26T03:07:28.251449+0000","last_fullsized":"2024-03-26T03:07:28.251449+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:28.633421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:28.633421+0000","last_clean_scrub_stamp":"2024-03-26T03:06:28.633421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:29.303917+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.256582+0000","last_change":"2024-03-26T03:07:18.502027+0000","last_active":"2024-03-26T03:07:28.256582+0000","last_peered":"2024-03-26T03:07:28.256582+0000","last_clean":"2024-03-26T03:07:28.256582+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:07:28.256582+0000","last_undegraded":"2024-03-26T03:07:28.256582+0000","last_fullsized":"2024-03-26T03:07:28.256582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:18.501806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:07:18.501806+0000","last_clean_scrub_stamp":"2024-03-26T03:07:18.501806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:46.942272+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":322,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.257341+0000","last_change":"2024-03-26T03:05:23.665275+0000","last_active":"2024-03-26T03:07:28.257341+0000","last_peered":"2024-03-26T03:07:28.257341+0000","last_clean":"2024-03-26T03:07:28.257341+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:07:28.257341+0000","last_undegraded":"2024-03-26T03:07:28.257341+0000","last_fullsized":"2024-03-26T03:07:28.257341+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.664911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.664911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813430+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"150'23","reported_seq":258,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.256979+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:07:28.256979+0000","last_peered":"2024-03-26T03:07:28.256979+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:07:28.256979+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.254544+0000","last_change":"2024-03-26T03:06:22.788827+0000","last_active":"2024-03-26T03:07:28.254544+0000","last_peered":"2024-03-26T03:07:28.254544+0000","last_clean":"2024-03-26T03:07:28.254544+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:07:28.254544+0000","last_undegraded":"2024-03-26T03:07:28.254544+0000","last_fullsized":"2024-03-26T03:07:28.254544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:22.788720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:22.788720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:42.860128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.257028+0000","last_change":"2024-03-26T03:06:30.609807+0000","last_active":"2024-03-26T03:07:28.257028+0000","last_peered":"2024-03-26T03:07:28.257028+0000","last_clean":"2024-03-26T03:07:28.257028+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:07:28.257028+0000","last_undegraded":"2024-03-26T03:07:28.257028+0000","last_fullsized":"2024-03-26T03:07:28.257028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:30.609691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:06:30.609691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:56.478786+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":33,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.239045+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:07:22.239045+0000","last_peered":"2024-03-26T03:07:22.239045+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:07:22.239045+0000","last_undegraded":"2024-03-26T03:07:22.239045+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"71'16","reported_seq":359,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:22.241223+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:07:22.241223+0000","last_peered":"2024-03-26T03:07:22.241223+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:07:22.241223+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":341,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.253016+0000","last_change":"2024-03-26T03:06:13.597058+0000","last_active":"2024-03-26T03:07:28.253016+0000","last_peered":"2024-03-26T03:07:28.253016+0000","last_clean":"2024-03-26T03:07:28.253016+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:07:28.253016+0000","last_undegraded":"2024-03-26T03:07:28.253016+0000","last_fullsized":"2024-03-26T03:07:28.253016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:13.596932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.596932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.854669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":441,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.254741+0000","last_change":"2024-03-26T03:06:15.899283+0000","last_active":"2024-03-26T03:07:28.254741+0000","last_peered":"2024-03-26T03:07:28.254741+0000","last_clean":"2024-03-26T03:07:28.254741+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:07:28.254741+0000","last_undegraded":"2024-03-26T03:07:28.254741+0000","last_fullsized":"2024-03-26T03:07:28.254741+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:06:15.899159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:06:15.899159+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.259542+0000","last_change":"2024-03-26T03:07:02.705758+0000","last_active":"2024-03-26T03:07:28.259542+0000","last_peered":"2024-03-26T03:07:28.259542+0000","last_clean":"2024-03-26T03:07:28.259542+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:07:28.259542+0000","last_undegraded":"2024-03-26T03:07:28.259542+0000","last_fullsized":"2024-03-26T03:07:28.259542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:07:02.705529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:02.705529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:24.450577+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"71'17","reported_seq":362,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.254742+0000","last_change":"2024-03-26T03:05:42.603323+0000","last_active":"2024-03-26T03:07:28.254742+0000","last_peered":"2024-03-26T03:07:28.254742+0000","last_clean":"2024-03-26T03:07:28.254742+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:07:28.254742+0000","last_undegraded":"2024-03-26T03:07:28.254742+0000","last_fullsized":"2024-03-26T03:07:28.254742+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'17","last_scrub_stamp":"2024-03-26T03:05:42.603184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:42.603184+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":43,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.256402+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:07:28.256402+0000","last_peered":"2024-03-26T03:07:28.256402+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:07:28.256402+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.241384+0000","last_change":"2024-03-26T03:06:36.734759+0000","last_active":"2024-03-26T03:07:22.241384+0000","last_peered":"2024-03-26T03:07:22.241384+0000","last_clean":"2024-03-26T03:07:22.241384+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:07:22.241384+0000","last_undegraded":"2024-03-26T03:07:22.241384+0000","last_fullsized":"2024-03-26T03:07:22.241384+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:36.734621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:06:36.734621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:06.103884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":324,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.266244+0000","last_change":"2024-03-26T03:07:14.718568+0000","last_active":"2024-03-26T03:07:29.266244+0000","last_peered":"2024-03-26T03:07:29.266244+0000","last_clean":"2024-03-26T03:07:29.266244+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:07:29.266244+0000","last_undegraded":"2024-03-26T03:07:29.266244+0000","last_fullsized":"2024-03-26T03:07:29.266244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:14.718439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:14.718439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:29.821361+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"71'15","reported_seq":360,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.260140+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:07:29.260140+0000","last_peered":"2024-03-26T03:07:29.260140+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:07:29.260140+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":342,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.255218+0000","last_change":"2024-03-26T03:06:20.817341+0000","last_active":"2024-03-26T03:07:28.255218+0000","last_peered":"2024-03-26T03:07:28.255218+0000","last_clean":"2024-03-26T03:07:28.255218+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:07:28.255218+0000","last_undegraded":"2024-03-26T03:07:28.255218+0000","last_fullsized":"2024-03-26T03:07:28.255218+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:20.817235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:20.817235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:40.086024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":336,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.267910+0000","last_change":"2024-03-26T03:07:13.692804+0000","last_active":"2024-03-26T03:07:29.267910+0000","last_peered":"2024-03-26T03:07:29.267910+0000","last_clean":"2024-03-26T03:07:29.267910+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:07:29.267910+0000","last_undegraded":"2024-03-26T03:07:29.267910+0000","last_fullsized":"2024-03-26T03:07:29.267910+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:13.692676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:07:13.692676+0000","last_clean_scrub_stamp":"2024-03-26T03:07:13.692676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:30.678644+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":330,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246356+0000","last_change":"2024-03-26T03:07:15.641949+0000","last_active":"2024-03-26T03:07:28.246356+0000","last_peered":"2024-03-26T03:07:28.246356+0000","last_clean":"2024-03-26T03:07:28.246356+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:07:28.246356+0000","last_undegraded":"2024-03-26T03:07:28.246356+0000","last_fullsized":"2024-03-26T03:07:28.246356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:15.641712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:15.641712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:38.398818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"71'18","reported_seq":91,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.267716+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:07:29.267716+0000","last_peered":"2024-03-26T03:07:29.267716+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:07:29.267716+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:31.781606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":35,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.257878+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:07:28.257878+0000","last_peered":"2024-03-26T03:07:28.257878+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:07:28.257878+0000","last_undegraded":"2024-03-26T03:07:28.257878+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":320,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.247809+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:07:28.247809+0000","last_peered":"2024-03-26T03:07:28.247809+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:07:28.247809+0000","last_undegraded":"2024-03-26T03:07:28.247809+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.254853+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:07:28.254853+0000","last_peered":"2024-03-26T03:07:28.254853+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:07:28.254853+0000","last_undegraded":"2024-03-26T03:07:28.254853+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"188'27","reported_seq":378,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.247636+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:07:28.247636+0000","last_peered":"2024-03-26T03:07:28.247636+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:07:28.247636+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.259105+0000","last_change":"2024-03-26T03:05:18.717654+0000","last_active":"2024-03-26T03:07:28.259105+0000","last_peered":"2024-03-26T03:07:28.259105+0000","last_clean":"2024-03-26T03:07:28.259105+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:07:28.259105+0000","last_undegraded":"2024-03-26T03:07:28.259105+0000","last_fullsized":"2024-03-26T03:07:28.259105+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:18.717525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:18.717525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.252067+0000","last_change":"2024-03-26T03:06:18.558568+0000","last_active":"2024-03-26T03:07:28.252067+0000","last_peered":"2024-03-26T03:07:28.252067+0000","last_clean":"2024-03-26T03:07:28.252067+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:07:28.252067+0000","last_undegraded":"2024-03-26T03:07:28.252067+0000","last_fullsized":"2024-03-26T03:07:28.252067+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.558456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.558456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:42.205217+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":328,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.240929+0000","last_change":"2024-03-26T03:06:35.723021+0000","last_active":"2024-03-26T03:07:22.240929+0000","last_peered":"2024-03-26T03:07:22.240929+0000","last_clean":"2024-03-26T03:07:22.240929+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:07:22.240929+0000","last_undegraded":"2024-03-26T03:07:22.240929+0000","last_fullsized":"2024-03-26T03:07:22.240929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:35.722885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:35.722885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:58.146384+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"71'11","reported_seq":346,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:28.247040+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:07:28.247040+0000","last_peered":"2024-03-26T03:07:28.247040+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:07:28.247040+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":334,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.262608+0000","last_change":"2024-03-26T03:06:26.605629+0000","last_active":"2024-03-26T03:07:28.262608+0000","last_peered":"2024-03-26T03:07:28.262608+0000","last_clean":"2024-03-26T03:07:28.262608+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:07:28.262608+0000","last_undegraded":"2024-03-26T03:07:28.262608+0000","last_fullsized":"2024-03-26T03:07:28.262608+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:06:26.605507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:46.950357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":326,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.257113+0000","last_change":"2024-03-26T03:05:21.649497+0000","last_active":"2024-03-26T03:07:28.257113+0000","last_peered":"2024-03-26T03:07:28.257113+0000","last_clean":"2024-03-26T03:07:28.257113+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:07:28.257113+0000","last_undegraded":"2024-03-26T03:07:28.257113+0000","last_fullsized":"2024-03-26T03:07:28.257113+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:21.649373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:21.649373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813529+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":324,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.251840+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:07:28.251840+0000","last_peered":"2024-03-26T03:07:28.251840+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:07:28.251840+0000","last_undegraded":"2024-03-26T03:07:28.251840+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:32.854637+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"71'15","reported_seq":357,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.246171+0000","last_change":"2024-03-26T03:06:32.059302+0000","last_active":"2024-03-26T03:07:28.246171+0000","last_peered":"2024-03-26T03:07:28.246171+0000","last_clean":"2024-03-26T03:07:28.246171+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:07:28.246171+0000","last_undegraded":"2024-03-26T03:07:28.246171+0000","last_fullsized":"2024-03-26T03:07:28.246171+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:32.059132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:32.059132+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:39.685621+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":33,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-26T03:07:22.241023+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:07:22.241023+0000","last_peered":"2024-03-26T03:07:22.241023+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:07:22.241023+0000","last_undegraded":"2024-03-26T03:07:22.241023+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-03-26T03:07:29.271451+0000","last_change":"2024-03-26T03:07:19.789243+0000","last_active":"2024-03-26T03:07:29.271451+0000","last_peered":"2024-03-26T03:07:29.271451+0000","last_clean":"2024-03-26T03:07:29.271451+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:07:29.271451+0000","last_undegraded":"2024-03-26T03:07:29.271451+0000","last_fullsized":"2024-03-26T03:07:29.271451+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:19.789091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:19.789091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:46.540724+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":321,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.247531+0000","last_change":"2024-03-26T03:06:20.991465+0000","last_active":"2024-03-26T03:07:28.247531+0000","last_peered":"2024-03-26T03:07:28.247531+0000","last_clean":"2024-03-26T03:07:28.247531+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:07:28.247531+0000","last_undegraded":"2024-03-26T03:07:28.247531+0000","last_fullsized":"2024-03-26T03:07:28.247531+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:20.991238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:20.991238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:45.569024+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"71'40","reported_seq":130,"reported_epoch":213,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:07:29.257092+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:07:29.257092+0000","last_peered":"2024-03-26T03:07:29.257092+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:07:29.257092+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:33.765478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":61,"num_read_kb":48,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":341,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-26T03:07:22.240452+0000","last_change":"2024-03-26T03:06:40.638182+0000","last_active":"2024-03-26T03:07:22.240452+0000","last_peered":"2024-03-26T03:07:22.240452+0000","last_clean":"2024-03-26T03:07:22.240452+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:07:22.240452+0000","last_undegraded":"2024-03-26T03:07:22.240452+0000","last_fullsized":"2024-03-26T03:07:22.240452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:40.638045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:06:40.638045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:44.425437+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":333,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-26T03:07:28.247162+0000","last_change":"2024-03-26T03:07:14.591670+0000","last_active":"2024-03-26T03:07:28.247162+0000","last_peered":"2024-03-26T03:07:28.247162+0000","last_clean":"2024-03-26T03:07:28.247162+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:07:28.247162+0000","last_undegraded":"2024-03-26T03:07:28.247162+0000","last_fullsized":"2024-03-26T03:07:28.247162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:14.591549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:07:14.591549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:08:30.935882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-26T03:07:28.257423+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:07:28.257423+0000","last_peered":"2024-03-26T03:07:28.257423+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:07:28.257423+0000","last_undegraded":"2024-03-26T03:07:28.257423+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:07:28.813442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4504682520,"num_objects":68737,"num_object_clones":0,"num_object_copies":206211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68737,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186041,"num_write_kb":8152833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14324674560,"data_stored":14324662344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82704,"ondisk_log_size":82704,"up":42,"acting":42,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":28,"acting":28,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":503,"num_read_kb":498,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":981,"num_read_kb":706,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":648,"ondisk_log_size":648,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103275,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1683240,"kb_used_data":1363292,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92061912,"statfs":{"total":95995035648,"available":94271397888,"internally_reserved":0,"allocated":1396011008,"data_stored":1395618913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":327612874},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430319,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1687708,"kb_used_data":1385808,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92057444,"statfs":{"total":95995035648,"available":94266822656,"internally_reserved":0,"allocated":1419067392,"data_stored":1418686943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309131718},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2224832,"kb_used_data":1909044,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91520320,"statfs":{"total":95995035648,"available":93716807680,"internally_reserved":0,"allocated":1954861056,"data_stored":1954445563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738508,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2118620,"kb_used_data":1776468,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91626532,"statfs":{"total":95995035648,"available":93825568768,"internally_reserved":0,"allocated":1819103232,"data_stored":1818712990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411458,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1934116,"kb_used_data":1632216,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91811036,"statfs":{"total":95995035648,"available":94014500864,"internally_reserved":0,"allocated":1671389184,"data_stored":1671002168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986297,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1571632,"kb_used_data":1373860,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92173520,"statfs":{"total":95995035648,"available":94385684480,"internally_reserved":0,"allocated":1406832640,"data_stored":1406432238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691957,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2460904,"kb_used_data":2181148,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91284248,"statfs":{"total":95995035648,"available":93475069952,"internally_reserved":0,"allocated":2233495552,"data_stored":2233103461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1809645568,"data_stored":1809645568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1924268032,"data_stored":1924268032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1650790400,"data_stored":1650786328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2537816064,"data_stored":2537816064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1395195904,"data_stored":1395195904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2223050752,"data_stored":2223046680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1396899840,"data_stored":1396899840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1387008000,"data_stored":1387003928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:07:33.201 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-26T03:07:33.210 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:33.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:33.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:33.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:33.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:33.591 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:33 smithi066 ceph-mon[29666]: pgmap v373: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 25420/202866 objects degraded (12.530%) 2024-03-26T03:07:33.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:33 smithi066 ceph-mon[37920]: pgmap v373: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 25420/202866 objects degraded (12.530%) 2024-03-26T03:07:33.636 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:33 smithi192 ceph-mon[33110]: pgmap v373: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 25420/202866 objects degraded (12.530%) 2024-03-26T03:07:33.737 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:33.780 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:33.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:33.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:33.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:33.964 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:07:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:07:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:07:34.024 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:34.067 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:34.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:34.171 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:07:34.171 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:07:34.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:34.302 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:34.343 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:34.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:34.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:34.582 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:34.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 25420/202866 objects degraded (12.530%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 25420/202866 objects degraded (12.530%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:34.627 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:34.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 25420/202866 objects degraded (12.530%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:34.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:34.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:34.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:34.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:35.001 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:35.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:35.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:35.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:35.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:35.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:35.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:35 smithi066 ceph-mon[29666]: pgmap v374: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 24972/197643 objects degraded (12.635%) 2024-03-26T03:07:35.401 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:35 smithi066 ceph-mon[37920]: pgmap v374: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 24972/197643 objects degraded (12.635%) 2024-03-26T03:07:35.417 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:35.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:35.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:35.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:35.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:35.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:35 smithi192 ceph-mon[33110]: pgmap v374: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 24972/197643 objects degraded (12.635%) 2024-03-26T03:07:35.836 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:35.870 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:35.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:35.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:36.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:36.106 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:36.133 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:36.235 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:36.285 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:36.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:36.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:36.516 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.565 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:36.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:36.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:36.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:36.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:36.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:36.845 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:07:36.893 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:07:36.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:36.937 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:07:36.973 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:36.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:36.991 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:07:37.035 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:07:37.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:37.079 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:07:37.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:37.130 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:07:37.130 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:07:37.130 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:07:37.130 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:07:37.130 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:07:37.130 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:07:37.131 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:07:37.131 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:07:37.131 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:07:37.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:37.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:07:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:37.307 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:07:37.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:37.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:37.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:37 smithi066 ceph-mon[29666]: pgmap v375: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 22822/182118 objects degraded (12.531%) 2024-03-26T03:07:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3318017789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2788602126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:37 smithi066 ceph-mon[37920]: pgmap v375: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 22822/182118 objects degraded (12.531%) 2024-03-26T03:07:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3318017789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2788602126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:37.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:37.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:37 smithi192 ceph-mon[33110]: pgmap v375: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 22822/182118 objects degraded (12.531%) 2024-03-26T03:07:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3318017789' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2788602126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:07:37.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:37.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:37.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:37.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:37.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:38.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:38.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:38.076 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:07:38.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:07:38.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:38.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:38.309 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:38.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:38 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:38.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2788602126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:07:38.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:38 smithi066 ceph-mon[37920]: osdmap e214: 8 total, 7 up, 8 in 2024-03-26T03:07:38.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:38 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:38.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2788602126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:07:38.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:38 smithi066 ceph-mon[29666]: osdmap e214: 8 total, 7 up, 8 in 2024-03-26T03:07:38.360 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:38.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:38.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:38.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:38.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:38 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2788602126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:07:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:38 smithi192 ceph-mon[33110]: osdmap e214: 8 total, 7 up, 8 in 2024-03-26T03:07:38.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:38.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:38.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:38.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:38.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:39.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:39.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:39.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:39.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:39.293 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:39.336 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:39.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:39.476 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:39.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:39 smithi066 ceph-mon[29666]: pgmap v377: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 22822/182118 objects degraded (12.531%) 2024-03-26T03:07:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 22822/182118 objects degraded (12.531%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:39 smithi066 ceph-mon[37920]: pgmap v377: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 22822/182118 objects degraded (12.531%) 2024-03-26T03:07:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 22822/182118 objects degraded (12.531%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:39.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:39.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:39.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:07:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:07:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:39 smithi192 ceph-mon[33110]: pgmap v377: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 22822/182118 objects degraded (12.531%) 2024-03-26T03:07:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 22822/182118 objects degraded (12.531%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:39.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:39.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:39.890 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:39.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:39.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.036 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:40.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:40.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:07:40.137 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:40.208 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:40.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.356 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:40.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:40.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:40.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:40.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:40.552 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:40.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:40.609 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:40.614 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:07:40.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:40.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:40.688 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:40.721 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:40.775 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:40.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:40.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:40.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:41.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:41.062 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:41.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:41.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:41.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:41.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:41.425 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:41 smithi066 ceph-mon[29666]: pgmap v378: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 20733/165903 objects degraded (12.497%) 2024-03-26T03:07:41.425 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:41 smithi066 ceph-mon[37920]: pgmap v378: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 20733/165903 objects degraded (12.497%) 2024-03-26T03:07:41.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:41.545 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:41 smithi192 ceph-mon[33110]: pgmap v378: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s; 20733/165903 objects degraded (12.497%) 2024-03-26T03:07:41.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:41.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:41.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:41.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:41.863 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:41.903 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:41.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:42.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:42.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:42.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:42.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:42.284 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:42.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:42.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:42.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:42.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:42.607 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:42 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:07:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:42 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:07:42.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:42 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:07:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:42 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:07:42.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:42 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:07:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:42 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:07:42.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:42.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:42.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:42.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:42.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:43.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:43.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:43.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:43.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:43.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:43.364 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 1 pgp_num to 11 2024-03-26T03:07:43.387 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:43.437 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:43.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:43.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[29666]: pgmap v379: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 20340/162258 objects degraded (12.536%) 2024-03-26T03:07:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[29666]: osdmap e215: 8 total, 7 up, 8 in 2024-03-26T03:07:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1684692127' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:07:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[29666]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:07:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[37920]: pgmap v379: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 20340/162258 objects degraded (12.536%) 2024-03-26T03:07:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[37920]: osdmap e215: 8 total, 7 up, 8 in 2024-03-26T03:07:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1684692127' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:07:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-mon[37920]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:07:43.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:43.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:43 smithi192 ceph-mon[33110]: pgmap v379: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 20340/162258 objects degraded (12.536%) 2024-03-26T03:07:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:43 smithi192 ceph-mon[33110]: osdmap e215: 8 total, 7 up, 8 in 2024-03-26T03:07:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:43 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1684692127' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:07:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:43 smithi192 ceph-mon[33110]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T03:07:43.807 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:43.868 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:43.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:43.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:44.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:44.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:07:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:07:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:07:44.149 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:44.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:44.250 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:44.296 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:44.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:44.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:44 smithi066 ceph-mon[37920]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T03:07:44.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:44 smithi066 ceph-mon[37920]: osdmap e216: 8 total, 7 up, 8 in 2024-03-26T03:07:44.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 20340/162258 objects degraded (12.536%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:44.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:44 smithi066 ceph-mon[29666]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T03:07:44.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:44 smithi066 ceph-mon[29666]: osdmap e216: 8 total, 7 up, 8 in 2024-03-26T03:07:44.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 20340/162258 objects degraded (12.536%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:44.439 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:44.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:44.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:44.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:44.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:44 smithi192 ceph-mon[33110]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T03:07:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:44 smithi192 ceph-mon[33110]: osdmap e216: 8 total, 7 up, 8 in 2024-03-26T03:07:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 20340/162258 objects degraded (12.536%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:44.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:44.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:44.957 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:44.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:44.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:45.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:45.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:45.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:45.140 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:45.238 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:07:45.238 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75525 7.4 7:229e4d32:::benchmark_data_smithi066_76917_object75524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.308357+0000", 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 354.92565000299999, 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.010940199, 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75525 2024-03-26T03:07:45.239 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308357+0000", 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308357+0000", 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.240 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308360+0000", 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6729999999999998e-06 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308401+0000", 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1057999999999999e-05 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308412+0000", 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0577000000000001e-05 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.241 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308421+0000", 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8289999999999997e-06 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495402+0000", 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.18698124099999999 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495411+0000", 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.285e-06 2024-03-26T03:07:45.242 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.312028+0000", 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.81661686300000003 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.314010+0000", 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0019819579999999998 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.319214+0000", 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.005204017 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.243 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.319297+0000", 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2698e-05 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75527 7.16 7:68ebb84d:::benchmark_data_smithi066_76917_object75526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.308433+0000", 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 354.92557310299998, 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.057818116, 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.244 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75527 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308433+0000", 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.245 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308433+0000", 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308434+0000", 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2200000000000003e-07 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308458+0000", 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3436999999999999e-05 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.246 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308464+0000", 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.404e-06 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308470+0000", 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4179999999999996e-06 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.517883+0000", 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20941363299999999 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.247 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.517904+0000", 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0769999999999999e-05 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.351182+0000", 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.83327831100000005 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.355526+0000", 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043432890000000002 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.248 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.366164+0000", 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010638409 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.366252+0000", 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7622999999999998e-05 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.249 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75535 7.14 7:296efe21:::benchmark_data_smithi066_76917_object75534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.322254+0000", 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 354.91175306500003, 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.013142381, 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75535 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.250 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322254+0000", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322254+0000", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322257+0000", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8410000000000002e-06 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322294+0000", 2024-03-26T03:07:45.251 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6766e-05 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322313+0000", 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9043999999999999e-05 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.322328+0000", 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4816e-05 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495341+0000", 2024-03-26T03:07:45.252 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.17301283000000001 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.495351+0000", 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0040999999999999e-05 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.325257+0000", 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.82990634299999999 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.253 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.325442+0000", 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000184842 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.325617+0000", 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000175184 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.334181+0000", 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0085639180000000002 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.254 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.334235+0000", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3418999999999998e-05 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.335090+0000", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00085500099999999998 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.335349+0000", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025913800000000001 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.335396+0000", 2024-03-26T03:07:45.255 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7197999999999998e-05 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76596 7.18 7.dbf6958 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:52.302604+0000", 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 352.931402381, 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99306915299999998, 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.256 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76596 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302604+0000", 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302604+0000", 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.257 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302607+0000", 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3630000000000001e-06 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302631+0000", 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4168999999999998e-05 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302644+0000", 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3159999999999999e-05 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.258 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302655+0000", 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.066e-05 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302731+0000", 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6691000000000001e-05 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.302740+0000", 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1570000000000006e-06 2024-03-26T03:07:45.259 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295660+0000", 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99291966399999998 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295673+0000", 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3288999999999999e-05 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.260 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76601 7.18 7.a9915598 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:52.309002+0000", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 352.92500436099999, 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98671072100000001, 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76601 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309002+0000", 2024-03-26T03:07:45.261 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309002+0000", 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309004+0000", 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0080000000000001e-06 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.262 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309028+0000", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3490999999999999e-05 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309038+0000", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0372e-05 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309046+0000", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0590000000000003e-06 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309124+0000", 2024-03-26T03:07:45.263 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.805e-05 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.309134+0000", 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0190000000000001e-05 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295703+0000", 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98656848200000002 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.264 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:45.265 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295713+0000", 2024-03-26T03:07:45.265 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0069e-05 2024-03-26T03:07:45.265 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76606 7.18 7.ca314098 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:52.313649+0000", 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 352.92035733900002, 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98228015700000004, 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.266 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76606 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313649+0000", 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313649+0000", 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.267 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313652+0000", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.317e-06 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313672+0000", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9953e-05 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313689+0000", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7540000000000001e-05 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.268 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313696+0000", 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5479999999999998e-06 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313741+0000", 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5008e-05 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:52.313747+0000", 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9039999999999997e-06 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.269 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295917+0000", 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.982170816 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.295929+0000", 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2071e-05 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76573 7.14 7:2ce71561:::benchmark_data_smithi066_76917_object76572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289155+0000", 2024-03-26T03:07:45.270 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94485156600001, 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.463067562, 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76573 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289155+0000", 2024-03-26T03:07:45.271 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289155+0000", 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289157+0000", 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.79e-06 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289186+0000", 2024-03-26T03:07:45.272 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.9400999999999999e-05 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289197+0000", 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0499000000000001e-05 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289207+0000", 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0324e-05 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.273 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289718+0000", 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00051074300000000003 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289726+0000", 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8490000000000006e-06 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.294959+0000", 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0052334759999999999 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.274 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.294968+0000", 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7739999999999993e-06 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.675546+0000", 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3805785420000001 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.676666+0000", 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001119461 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.275 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752119+0000", 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.075453352000000001 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752223+0000", 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010335099999999999 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76576 7.17 7:ea6e24b1:::benchmark_data_smithi066_76917_object76575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289318+0000", 2024-03-26T03:07:45.276 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94468858200003, 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.528118935, 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76576 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289318+0000", 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.277 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289318+0000", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289319+0000", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8299999999999995e-07 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289351+0000", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2016e-05 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.278 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289360+0000", 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4590000000000004e-06 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289368+0000", 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8369999999999997e-06 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290510+0000", 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0011418000000000001 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.279 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290517+0000", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8959999999999997e-06 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293379+0000", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002861948 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293388+0000", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9959999999999999e-06 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.801084+0000", 2024-03-26T03:07:45.280 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.507696285 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.802350+0000", 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00126542 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.817272+0000", 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.01492253 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.817437+0000", 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016476500000000001 2024-03-26T03:07:45.281 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76577 7.17 7:ef576187:::benchmark_data_smithi066_76917_object76576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289379+0000", 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94462747300003, 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.536755898, 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.282 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76577 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289379+0000", 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289379+0000", 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289380+0000", 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8800000000000001e-07 2024-03-26T03:07:45.283 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289406+0000", 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6115e-05 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289414+0000", 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5669999999999997e-06 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289421+0000", 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3370000000000002e-06 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.284 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290544+0000", 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001122861 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290549+0000", 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3990000000000003e-06 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293410+0000", 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028609740000000001 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.285 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293417+0000", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9650000000000002e-06 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.802413+0000", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.508995978 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.803792+0000", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001378659 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.286 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.825676+0000", 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.021884596999999999 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.826135+0000", 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00045855800000000002 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76589 7.17 7:e926eddf:::benchmark_data_smithi066_76917_object76588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289429+0000", 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94457728399999, 2024-03-26T03:07:45.287 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5438101280000001, 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76589 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289429+0000", 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.288 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289429+0000", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289430+0000", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7400000000000002e-07 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289455+0000", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4635999999999999e-05 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289462+0000", 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9290000000000001e-06 2024-03-26T03:07:45.289 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289469+0000", 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9770000000000003e-06 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.290 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290574+0000", 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0011057549999999999 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290579+0000", 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7600000000000002e-06 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.291 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293441+0000", 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002861363 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293448+0000", 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.187e-06 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.803843+0000", 2024-03-26T03:07:45.292 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5103952920000001 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.805763+0000", 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00192022 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.832867+0000", 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.027103308 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.293 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833239+0000", 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000372927 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76605 7.14 7:2d4fabea:::benchmark_data_smithi066_76917_object76604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289476+0000", 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94453032899997, 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.476329665, 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.294 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76605 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289476+0000", 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289476+0000", 2024-03-26T03:07:45.295 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289477+0000", 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8000000000000005e-07 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289503+0000", 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5534999999999999e-05 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.296 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289510+0000", 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2790000000000001e-06 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289516+0000", 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2740000000000004e-06 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289771+0000", 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025493100000000002 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.297 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289776+0000", 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7539999999999997e-06 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.298866+0000", 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0090900359999999993 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.298874+0000", 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3480000000000004e-06 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.298 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752310+0000", 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4534362270000001 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.752461+0000", 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000150419 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,3,5", 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.753330+0000", 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00086932899999999996 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.299 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759667+0000", 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0063369380000000003 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759732+0000", 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4414999999999999e-05 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759810+0000", 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8844000000000006e-05 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.300 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.759830+0000", 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.963e-05 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.760377+0000", 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00054731500000000004 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.765806+0000", 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0054286109999999999 2024-03-26T03:07:45.301 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76610 7.19 7:9b2fb385:::benchmark_data_smithi066_76917_object76609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.289524+0000", 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94448219100002, 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4842921849999999, 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.302 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76610 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289524+0000", 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289524+0000", 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.303 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289525+0000", 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3399999999999998e-07 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289581+0000", 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6276999999999997e-05 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289590+0000", 2024-03-26T03:07:45.304 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0149999999999992e-06 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.289598+0000", 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4760000000000001e-06 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290791+0000", 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001193295 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:07:45.305 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290798+0000", 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7519999999999999e-06 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.292610+0000", 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018123169999999999 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.292620+0000", 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7599999999999997e-06 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.306 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.664560+0000", 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3719395089999999 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.664739+0000", 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017908299999999999 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,6,7", 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.664948+0000", 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000209796 2024-03-26T03:07:45.307 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771112+0000", 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.106163639 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771767+0000", 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00065495799999999997 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771816+0000", 2024-03-26T03:07:45.308 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9298000000000001e-05 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771862+0000", 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6029000000000003e-05 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.771981+0000", 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011827 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.773817+0000", 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018359769999999999 2024-03-26T03:07:45.309 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76628 7.17 7:efdc1178:::benchmark_data_smithi066_76917_object76627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.290595+0000", 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94341165600002, 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5449126049999999, 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.310 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76628 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290595+0000", 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290595+0000", 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.311 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290596+0000", 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.392e-06 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290660+0000", 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3312999999999995e-05 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290668+0000", 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0469999999999994e-06 2024-03-26T03:07:45.312 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290676+0000", 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.259e-06 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291402+0000", 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00072557899999999996 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291407+0000", 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8470000000000001e-06 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.313 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293472+0000", 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020646879999999999 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293479+0000", 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5169999999999996e-06 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.805808+0000", 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.512329349 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.314 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.806173+0000", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00036477100000000001 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5,7", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.806375+0000", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020274600000000001 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.813734+0000", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0073585120000000002 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.315 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833675+0000", 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019941496999999999 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833728+0000", 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2692000000000003e-05 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833774+0000", 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5426000000000002e-05 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.316 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.833903+0000", 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000129529 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835508+0000", 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0016044410000000001 2024-03-26T03:07:45.317 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.318 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.318 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.318 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.318 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.318 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76630 7.17 7:e9c213e7:::benchmark_data_smithi066_76917_object76629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:07:45.318 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.290970+0000", 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 351.94303702000002, 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.559619444, 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76630 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290970+0000", 2024-03-26T03:07:45.319 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290970+0000", 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.290971+0000", 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0839999999999999e-06 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.320 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291026+0000", 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5686999999999997e-05 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291036+0000", 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.594e-06 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291044+0000", 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1610000000000003e-06 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.321 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291608+0000", 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00056420000000000005 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting_for_map not empty", 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.291615+0000", 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3790000000000003e-06 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293502+0000", 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018876349999999999 2024-03-26T03:07:45.322 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.293509+0000", 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5270000000000004e-06 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835554+0000", 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5420450429999999 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835739+0000", 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018524399999999999 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.323 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5,7", 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.835962+0000", 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00022237799999999999 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.839842+0000", 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038806880000000002 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.839859+0000", 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7227e-05 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.324 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.844365+0000", 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0045059039999999998 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.848761+0000", 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043960520000000001 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.848823+0000", 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1501999999999995e-05 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.325 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.850589+0000", 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017661389999999999 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:112611 7.13 7:c88931eb:::benchmark_data_smithi066_76917_object12428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 292.87048098000002, 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.85905683200000005, 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.326 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 112611 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:07:45.327 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363526+0000", 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6499999999999999e-07 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363529+0000", 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7949999999999999e-06 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363534+0000", 2024-03-26T03:07:45.328 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1560000000000003e-06 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363539+0000", 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2329999999999997e-06 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363678+0000", 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000138412 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.363710+0000", 2024-03-26T03:07:45.329 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.1860999999999999e-05 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.169363+0000", 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.80565370300000005 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.169491+0000", 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000127441 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.170058+0000", 2024-03-26T03:07:45.330 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00056694500000000001 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.177633+0000", 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0075750779999999998 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.181265+0000", 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003632298 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.181318+0000", 2024-03-26T03:07:45.331 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2575999999999999e-05 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.222533+0000", 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.041215214 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:53.222582+0000", 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9555000000000002e-05 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.332 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142695 7.18 7:1a9a85f2:::benchmark_data_smithi066_76917_object42512:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.371156+0000", 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 255.862850673, 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0339076030000001, 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142695 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.333 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371156+0000", 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371156+0000", 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.334 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371157+0000", 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4389999999999999e-06 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371162+0000", 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3810000000000001e-06 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371170+0000", 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.038e-06 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.335 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371176+0000", 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3609999999999997e-06 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371422+0000", 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000245386 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.371428+0000", 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2110000000000003e-06 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.336 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.374547+0000", 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031191000000000001 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.374558+0000", 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0951000000000001e-05 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405007+0000", 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.030449669 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.337 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405064+0000", 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6066999999999997e-05 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142780 7.18 7:1931f3fd:::benchmark_data_smithi066_76917_object42597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.442439+0000", 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 255.79156777599999, 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97209616700000001, 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.338 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142780 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442439+0000", 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.339 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442439+0000", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442441+0000", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7409999999999999e-06 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442447+0000", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4930000000000003e-06 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442457+0000", 2024-03-26T03:07:45.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0236e-05 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442466+0000", 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.354e-06 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442563+0000", 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7499999999999998e-05 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.442574+0000", 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0658e-05 2024-03-26T03:07:45.341 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405672+0000", 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96309829800000002 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.405817+0000", 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000145026 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.406214+0000", 2024-03-26T03:07:45.342 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000396473 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414323+0000", 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0081098179999999995 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414379+0000", 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5500000000000001e-05 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414425+0000", 2024-03-26T03:07:45.343 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5986000000000003e-05 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414494+0000", 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8732000000000005e-05 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.344 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414535+0000", 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1352e-05 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.345 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142786 7.18 7:18a9b3ae:::benchmark_data_smithi066_76917_object42603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.446346+0000", 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 255.78766057799999, 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98757069600000003, 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142786 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.346 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446346+0000", 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446346+0000", 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446348+0000", 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0320000000000002e-06 2024-03-26T03:07:45.347 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446354+0000", 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6389999999999997e-06 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446365+0000", 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1208e-05 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446374+0000", 2024-03-26T03:07:45.348 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9870000000000005e-06 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446439+0000", 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4834999999999998e-05 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.446451+0000", 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.222e-05 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.349 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.408158+0000", 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96170670800000002 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.408280+0000", 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012194 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.408566+0000", 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00028633599999999999 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.350 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414573+0000", 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0060071839999999996 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414703+0000", 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000129443 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433781+0000", 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019078504 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.351 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433855+0000", 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.436e-05 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433917+0000", 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1299999999999999e-05 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142796 7.18 7:1c624d3a:::benchmark_data_smithi066_76917_object42613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.453769+0000", 2024-03-26T03:07:45.352 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 255.780237322, 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98029207799999996, 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142796 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453769+0000", 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.353 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453769+0000", 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453771+0000", 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0949999999999998e-06 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453776+0000", 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.216e-06 2024-03-26T03:07:45.354 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453783+0000", 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4510000000000003e-06 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.453789+0000", 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8730000000000004e-06 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.454575+0000", 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00078626300000000004 2024-03-26T03:07:45.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.454586+0000", 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.044e-05 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.412024+0000", 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.957438758 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.412162+0000", 2024-03-26T03:07:45.356 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013784299999999999 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.412494+0000", 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000332076 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414745+0000", 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002250232 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.414782+0000", 2024-03-26T03:07:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7766999999999999e-05 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.433978+0000", 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019195796000000001 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434018+0000", 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0212000000000001e-05 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434061+0000", 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3056000000000002e-05 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:142818 7.18 7:1ebdfa01:::benchmark_data_smithi066_76917_object42635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 255.64718596700001, 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.84743222200000001, 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:07:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 142818 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586821+0000", 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8999999999999996e-07 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586824+0000", 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.9119999999999998e-06 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586830+0000", 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6559999999999997e-06 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586835+0000", 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5809999999999996e-06 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586975+0000", 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013987400000000001 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:29.586984+0000", 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8950000000000004e-06 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.419246+0000", 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.83226167200000001 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.419365+0000", 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118733 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 6,7", 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.419698+0000", 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00033390999999999998 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.421858+0000", 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0021593250000000001 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.421901+0000", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2904000000000001e-05 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434168+0000", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.012267659 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434214+0000", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5793e-05 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:07:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:03:30.434253+0000", 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8717999999999998e-05 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:07:45.366 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:07:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:45.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:45.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:45.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:45.506 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:45.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:45.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:45 smithi066 ceph-mon[29666]: pgmap v382: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 1.22k op/s; 19928/157176 objects degraded (12.679%) 2024-03-26T03:07:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:45 smithi066 ceph-mon[37920]: pgmap v382: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 1.22k op/s; 19928/157176 objects degraded (12.679%) 2024-03-26T03:07:45.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:45.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:45 smithi192 ceph-mon[33110]: pgmap v382: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 1.22k op/s; 19928/157176 objects degraded (12.679%) 2024-03-26T03:07:45.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:45.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:45.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:45.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:45.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:45.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:46.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:46.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:46.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:46.191 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:46.293 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.335 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:46.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:46.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:46.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:46.574 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:46.617 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:46.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:46.758 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:46.841 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:07:46.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:46.880 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:07:46.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:46.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:46.916 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:07:46.959 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:07:46.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:46.998 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:07:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:47.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:47.040 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:07:47.079 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:07:47.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:07:47.124 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.163 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:47.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:47.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:47.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:47.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:47.439 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:47.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:47.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:47 smithi066 ceph-mon[29666]: pgmap v383: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 17818/142047 objects degraded (12.544%) 2024-03-26T03:07:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:47 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1241562823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:47.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:47 smithi066 ceph-mon[37920]: pgmap v383: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 17818/142047 objects degraded (12.544%) 2024-03-26T03:07:47.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:47 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1241562823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:47.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:47.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:47 smithi192 ceph-mon[33110]: pgmap v383: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 17818/142047 objects degraded (12.544%) 2024-03-26T03:07:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:47 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1241562823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:47.811 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:47.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:47.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:47.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:47.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:48.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:48.129 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:48.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:48.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:48.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:48.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:48.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:48.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:48.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:07:48.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:07:48.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:48.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:48.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:48.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:48.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:48.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:48.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:48.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:48.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:48.913 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:07:48.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:49.011 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:49.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:49.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:49.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:49.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:49.392 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.462 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:49.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:49.563 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:49.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:49 smithi066 ceph-mon[29666]: pgmap v384: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s; 17818/142047 objects degraded (12.544%) 2024-03-26T03:07:49.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 17818/142047 objects degraded (12.544%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:49.596 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:49 smithi066 ceph-mon[37920]: pgmap v384: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s; 17818/142047 objects degraded (12.544%) 2024-03-26T03:07:49.596 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 17818/142047 objects degraded (12.544%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:49.610 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:49.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:49 smithi192 ceph-mon[33110]: pgmap v384: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s; 17818/142047 objects degraded (12.544%) 2024-03-26T03:07:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 17818/142047 objects degraded (12.544%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:49.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:07:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:07:49.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:49.847 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:49.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:07:49.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:07:49.885 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-26T03:07:49.885 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-26T03:07:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:49.899 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:49.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:49.993 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:50.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.056 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:50.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:50.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:50.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:50.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:50.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:50.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:50.443 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:07:50.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:50 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:50.462 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:50 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.486 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:50.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:50.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:50.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:50.722 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:50 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:07:50.775 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:50.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:50.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:50.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:51.011 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:51.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:51.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:51.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:51.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:51.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:51.409 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:07:51.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:51.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:51.573 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:51.616 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:51.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:51 smithi192 ceph-mon[33110]: pgmap v385: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 15641/124737 objects degraded (12.539%) 2024-03-26T03:07:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:51 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2270449585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:51 smithi192 ceph-mon[33110]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:51.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:51.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:51 smithi066 ceph-mon[29666]: pgmap v385: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 15641/124737 objects degraded (12.539%) 2024-03-26T03:07:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:51 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2270449585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:51 smithi066 ceph-mon[29666]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:51.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:51 smithi066 ceph-mon[37920]: pgmap v385: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 15641/124737 objects degraded (12.539%) 2024-03-26T03:07:51.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:51 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2270449585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:51.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:51 smithi066 ceph-mon[37920]: from='client.35093 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:07:51.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:51.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:51.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:52.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:52.137 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.181 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:52.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:52.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:52.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:52.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:52.431 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:52.431 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:52.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:52.526 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:07:52.526 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:07:52.551 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:52.601 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[37920]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:07:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[37920]: osdmap e217: 8 total, 7 up, 8 in 2024-03-26T03:07:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:07:52.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:07:52.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:52.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[29666]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:07:52.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[29666]: osdmap e217: 8 total, 7 up, 8 in 2024-03-26T03:07:52.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:07:52.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:52 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:07:52.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:52 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:07:52.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:52 smithi192 ceph-mon[33110]: from='client.35093 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:07:52.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:52 smithi192 ceph-mon[33110]: osdmap e217: 8 total, 7 up, 8 in 2024-03-26T03:07:52.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:52 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:07:52.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:52 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:07:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:52.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:52.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:52.936 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:52.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:53.036 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.078 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:53.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:53.219 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:53.321 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:53.359 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:53.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:53.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:53.597 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[29666]: pgmap v387: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.57k op/s; 15222/121062 objects degraded (12.574%) 2024-03-26T03:07:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:07:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[37920]: pgmap v387: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.57k op/s; 15222/121062 objects degraded (12.574%) 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:07:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:07:53.638 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:53.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:07:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:53 smithi192 ceph-mon[33110]: pgmap v387: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.57k op/s; 15222/121062 objects degraded (12.574%) 2024-03-26T03:07:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:53 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:07:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:53 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:07:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:53 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:07:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:53 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:07:53.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:53.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:53.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:53.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:53.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:54.010 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:54.054 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:54.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:07:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:07:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:07:54.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:54.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:54.289 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:54.320 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:54.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:54.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:54.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:54.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 8.6 scrub starts 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 15222/121062 objects degraded (12.574%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:07:54.665 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 8.6 scrub starts 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 15222/121062 objects degraded (12.574%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:07:54.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:54.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:54.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 8.6 scrub starts 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 15222/121062 objects degraded (12.574%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:07:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:07:54.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:54.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:54.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:54.966 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:55.005 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:55.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:55.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:55.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:07:55.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:55.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:55.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:55.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:55.504 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.543 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:55.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:55.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 8.6 scrub ok 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: pgmap v388: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 15222/119913 objects degraded (12.694%) 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 1.5 scrub starts 2024-03-26T03:07:55.668 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 1.5 scrub ok 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 8.6 scrub ok 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: pgmap v388: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 15222/119913 objects degraded (12.694%) 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:07:55.669 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:07:55.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 1.5 scrub starts 2024-03-26T03:07:55.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 1.5 scrub ok 2024-03-26T03:07:55.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:07:55.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:07:55.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:07:55.670 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:55 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:07:55.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 8.6 scrub ok 2024-03-26T03:07:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: pgmap v388: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 15222/119913 objects degraded (12.694%) 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 1.5 scrub starts 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 1.5 scrub ok 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:07:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:55 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:07:55.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:55.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:55.924 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:55.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:55.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:56.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:56.102 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:56.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.261 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:56.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:56.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:56.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:56.502 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.551 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:56.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:07:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:56 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:07:56.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.692 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:56.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:56 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:07:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:56 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:07:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:56 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:07:56.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:56 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:07:56.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:56.837 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:56.846 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:56.863 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:07:56.932 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:07:56.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:56.961 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:07:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:56.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:56.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:57.001 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:07:57.045 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:07:57.079 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:07:57.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:57.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:57.119 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:07:57.159 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:07:57.160 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:07:57.160 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:07:57.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:57.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:57.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:57.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:57.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:57.537 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:57.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:57 smithi066 ceph-mon[37920]: pgmap v389: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s; 12785/101142 objects degraded (12.641%) 2024-03-26T03:07:57.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/797699302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:57.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:57 smithi066 ceph-mon[29666]: pgmap v389: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s; 12785/101142 objects degraded (12.641%) 2024-03-26T03:07:57.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/797699302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:57.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:57.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:57 smithi192 ceph-mon[33110]: pgmap v389: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s; 12785/101142 objects degraded (12.641%) 2024-03-26T03:07:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/797699302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:07:57.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:57.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:07:57.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:57.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:57.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:58.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:58.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:58.185 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:58.227 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:58.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:58.363 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:58.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:58.510 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:58.612 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:07:58.652 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:07:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:58 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:07:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:58 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:07:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:58 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:07:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:58 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:07:58.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:58.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:07:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:58 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:07:58.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:58.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:58.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:58.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:59.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:59.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:59.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:07:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:59.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:07:59.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:07:59.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:07:59.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:07:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.497 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:07:59.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:07:59.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[29666]: pgmap v390: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s; 12785/101142 objects degraded (12.641%) 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 12785/101142 objects degraded (12.641%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[37920]: pgmap v390: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s; 12785/101142 objects degraded (12.641%) 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:07:59.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:07:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 12785/101142 objects degraded (12.641%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:59.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:07:59.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:07:59.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:07:59.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:59 smithi192 ceph-mon[33110]: pgmap v390: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s; 12785/101142 objects degraded (12.641%) 2024-03-26T03:07:59.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:59 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:07:59.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:59 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:07:59.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:07:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 12785/101142 objects degraded (12.641%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:07:59.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:07:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:07:59.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:07:59.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:07:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:07:59.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:07:59.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:00.023 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:00.064 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:00.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:00.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:00.300 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:00.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:00.442 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:00.459 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:08:00.493 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:00.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:00.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:08:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:08:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:00 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:08:00.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:08:00.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:08:00.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:00 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:08:00.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:00.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:00.926 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:00.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:01.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:01.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:01.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:01.202 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:01.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:01.356 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:01.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:01.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:01.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:01.728 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:01 smithi192 ceph-mon[33110]: pgmap v391: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 10583/84294 objects degraded (12.555%) 2024-03-26T03:08:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:01 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:08:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:01 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:08:01.772 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:01 smithi066 ceph-mon[29666]: pgmap v391: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 10583/84294 objects degraded (12.555%) 2024-03-26T03:08:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:01 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:08:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:01 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:08:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:01 smithi066 ceph-mon[37920]: pgmap v391: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 10583/84294 objects degraded (12.555%) 2024-03-26T03:08:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:01 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:08:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:01 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:08:01.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:01.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:01.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:02.012 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:02.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:02.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:02.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:02.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:02.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:02.434 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:02.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:08:02.476 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:02.480 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:02 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:08:02.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:02.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:02.717 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:02 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:08:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:02 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:08:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:02 smithi192 ceph-mon[33110]: pgmap v392: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.26k op/s; 10146/80496 objects degraded (12.604%) 2024-03-26T03:08:02.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:02 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:08:02.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:02 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:08:02.741 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:02 smithi066 ceph-mon[37920]: pgmap v392: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.26k op/s; 10146/80496 objects degraded (12.604%) 2024-03-26T03:08:02.741 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:02 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:08:02.741 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:02 smithi066 ceph-mon[29666]: pgmap v392: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.26k op/s; 10146/80496 objects degraded (12.604%) 2024-03-26T03:08:02.748 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:02.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:02.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:02.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:02.985 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:03.030 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:03.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:03.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:03.266 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.315 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:03.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:03.417 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:03.467 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:08:03.565 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:08:03.566 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:08:03.566 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:08:03.566 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:08:03.569 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:08:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:08:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:08:03.569 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:08:03.569 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:08:03.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.603 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:03.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:08:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:08:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:08:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:08:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:08:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:08:03.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:08:03.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:08:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:08:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:08:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:08:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:08:03.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:08:03.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:08:03.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:08:03.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:08:03.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:08:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:08:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:08:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:08:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:08:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:08:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:08:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:03 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:08:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:03.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:03.840 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:03.877 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:03.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:03.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:03.979 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:04.022 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:04.088 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:08:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:08:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:08:04.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:04.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:04.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:04.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:04.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:04.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:04.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:04.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:04.678 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:04.720 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 10146/80496 objects degraded (12.604%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:04 smithi192 ceph-mon[33110]: pgmap v393: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 10146/79194 objects degraded (12.812%) 2024-03-26T03:08:04.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 10146/80496 objects degraded (12.604%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:04.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:04 smithi066 ceph-mon[37920]: pgmap v393: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 10146/79194 objects degraded (12.812%) 2024-03-26T03:08:04.787 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 10146/80496 objects degraded (12.604%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:04.787 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:04 smithi066 ceph-mon[29666]: pgmap v393: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 10146/79194 objects degraded (12.812%) 2024-03-26T03:08:04.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:04.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:04.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:04.957 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:08:04.993 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:08:04.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-26T03:08:05.003 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:05.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:05.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:05.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:05.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:05.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:05.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:05.500 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:08:05.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:05.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:05.537 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:08:05.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:05.681 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1332588098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:08:05.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:08:05.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:08:05.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:08:05.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:08:05.760 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:05 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1332588098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:05.775 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:05.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.847 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:05.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:05.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:05.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:05.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:08:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:08:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:08:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:08:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:08:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:08:05.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:05 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1332588098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:06.084 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:06.125 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:06.227 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:06.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:06.304 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:06.371 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:06.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:06.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:06.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:08:06.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:08:06.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:08:06.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1332588098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:08:06.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: osdmap e218: 8 total, 7 up, 8 in 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: 5.d deep-scrub starts 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: 5.d deep-scrub ok 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[29666]: pgmap v395: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 7510/59400 objects degraded (12.643%) 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:08:06.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1332588098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:08:06.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: osdmap e218: 8 total, 7 up, 8 in 2024-03-26T03:08:06.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:08:06.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:08:06.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: 5.d deep-scrub starts 2024-03-26T03:08:06.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: 5.d deep-scrub ok 2024-03-26T03:08:06.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:06 smithi066 ceph-mon[37920]: pgmap v395: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 7510/59400 objects degraded (12.643%) 2024-03-26T03:08:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:06.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:06.672 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.718 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:06.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:06.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:06.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:06.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:08:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1332588098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: osdmap e218: 8 total, 7 up, 8 in 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: 5.d deep-scrub starts 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: 5.d deep-scrub ok 2024-03-26T03:08:06.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:06 smithi192 ceph-mon[33110]: pgmap v395: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 7510/59400 objects degraded (12.643%) 2024-03-26T03:08:06.997 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:06.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:07.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:07.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:07.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:07.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:07.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:07.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:07.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.558 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:07.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:07.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:07.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:07.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:07.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:07.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[29666]: 8.9 scrub starts 2024-03-26T03:08:07.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[29666]: 8.9 scrub ok 2024-03-26T03:08:07.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:08:07.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:08:07.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[37920]: 8.9 scrub starts 2024-03-26T03:08:07.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[37920]: 8.9 scrub ok 2024-03-26T03:08:07.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:08:07.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:07 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:08:07.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:07.960 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:07.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:07 smithi192 ceph-mon[33110]: 8.9 scrub starts 2024-03-26T03:08:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:07 smithi192 ceph-mon[33110]: 8.9 scrub ok 2024-03-26T03:08:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:07 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:08:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:07 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:08:08.062 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:08.100 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:08.202 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:08.243 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:08.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:08:08.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:08.394 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:08.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:08.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:08.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: 8.e scrub starts 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: 8.e scrub ok 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[29666]: pgmap v396: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 7510/59400 objects degraded (12.643%) 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: 8.e scrub starts 2024-03-26T03:08:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: 8.e scrub ok 2024-03-26T03:08:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:08:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:08:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:08:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:08:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:08 smithi066 ceph-mon[37920]: pgmap v396: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 7510/59400 objects degraded (12.643%) 2024-03-26T03:08:08.633 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:08.674 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:08.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:08.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:08.907 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:08.935 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:08.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: 8.e scrub starts 2024-03-26T03:08:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: 8.e scrub ok 2024-03-26T03:08:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:08:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:08:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:08:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:08:08.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:08 smithi192 ceph-mon[33110]: pgmap v396: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 1.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.39k op/s; 7510/59400 objects degraded (12.643%) 2024-03-26T03:08:09.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:09.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:09.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:09.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:09.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:09.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:09.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.491 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:09.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:09.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:09.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:09.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:09.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:08:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:08:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:08:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: osdmap e219: 8 total, 7 up, 8 in 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 2.1c deep-scrub starts 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 8.4 scrub starts 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 2.1c deep-scrub ok 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 6.14 deep-scrub starts 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: 6.14 deep-scrub ok 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 7510/59400 objects degraded (12.643%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:09 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:09.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: osdmap e219: 8 total, 7 up, 8 in 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 2.1c deep-scrub starts 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 8.4 scrub starts 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 2.1c deep-scrub ok 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 6.14 deep-scrub starts 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: 6.14 deep-scrub ok 2024-03-26T03:08:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 7510/59400 objects degraded (12.643%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: osdmap e219: 8 total, 7 up, 8 in 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 2.1c deep-scrub starts 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 8.4 scrub starts 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 2.1c deep-scrub ok 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 6.14 deep-scrub starts 2024-03-26T03:08:09.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: 6.14 deep-scrub ok 2024-03-26T03:08:09.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 7510/59400 objects degraded (12.643%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:09.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:09.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:09.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:09 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:09.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:09.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:09.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:09.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:10.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:10.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:10.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:10.269 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:10.309 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:10.351 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:10.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:10.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:10.548 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.591 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:10.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[29666]: 8.4 scrub ok 2024-03-26T03:08:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[29666]: pgmap v398: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 888 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 5403/43281 objects degraded (12.484%) 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[37920]: 8.4 scrub ok 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:08:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:10 smithi066 ceph-mon[37920]: pgmap v398: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 888 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 5403/43281 objects degraded (12.484%) 2024-03-26T03:08:10.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:10.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:10.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:10.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:10.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:10 smithi192 ceph-mon[33110]: 8.4 scrub ok 2024-03-26T03:08:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:10 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:08:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:10 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:08:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:10 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:08:10.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:10 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:08:10.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:10 smithi192 ceph-mon[33110]: pgmap v398: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 888 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 5403/43281 objects degraded (12.484%) 2024-03-26T03:08:10.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:11.022 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:11.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:11.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:11.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:11.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:11.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:11.496 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:08:11.532 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:08:11.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:11.565 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:08:11.565 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:11 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:08:11.565 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:11 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:08:11.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1696231609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:11.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:11 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:08:11.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:11 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:08:11.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1696231609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:11.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:11.608 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:08:11.656 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:08:11.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:11.696 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:08:11.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:11.737 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:08:11.738 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:08:11.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:11.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:11.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:11 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:08:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:11 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:08:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1696231609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:11.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:11.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:12.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:12.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:12.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:12.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:12.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:12.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:12.505 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:12.548 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:12.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:12.654 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:08:12.654 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:08:12.654 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:08:12.654 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:08:12.654 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:08:12.655 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: 4.6 deep-scrub starts 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: 4.6 deep-scrub ok 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:08:12.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[29666]: pgmap v399: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 809 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 5005/39492 objects degraded (12.673%) 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: 4.6 deep-scrub starts 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: 4.6 deep-scrub ok 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:08:12.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:12 smithi066 ceph-mon[37920]: pgmap v399: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 809 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 5005/39492 objects degraded (12.673%) 2024-03-26T03:08:12.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:12.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:12.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:12.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:12.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:12.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: 4.6 deep-scrub starts 2024-03-26T03:08:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: 4.6 deep-scrub ok 2024-03-26T03:08:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:08:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:08:12.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:08:12.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:08:12.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:08:12.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:08:12.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:12 smithi192 ceph-mon[33110]: pgmap v399: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 809 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 5005/39492 objects degraded (12.673%) 2024-03-26T03:08:13.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:13.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:13.187 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:13.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:13.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:13.326 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:13.368 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:13.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:13.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:13.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:13 smithi066 ceph-mon[29666]: osdmap e220: 8 total, 7 up, 8 in 2024-03-26T03:08:13.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:13 smithi066 ceph-mon[37920]: osdmap e220: 8 total, 7 up, 8 in 2024-03-26T03:08:13.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:13.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:13.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:13.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:13.847 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:08:13.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:13.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:13.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:13 smithi192 ceph-mon[33110]: osdmap e220: 8 total, 7 up, 8 in 2024-03-26T03:08:14.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.060 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:14.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:14.074 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:08:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:08:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:08:14.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:14.201 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:14.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:14.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:14.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:14.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:14.574 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:14.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:14.720 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:14.756 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:14.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:08:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:08:14.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 5005/39492 objects degraded (12.673%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:14.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[29666]: pgmap v401: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 783 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 5005/38271 objects degraded (13.078%) 2024-03-26T03:08:14.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:08:14.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:08:14.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 5005/39492 objects degraded (12.673%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:14.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:14 smithi066 ceph-mon[37920]: pgmap v401: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 783 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 5005/38271 objects degraded (13.078%) 2024-03-26T03:08:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:14.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:14.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:14 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:08:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:14 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:08:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 5005/39492 objects degraded (12.673%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:14 smithi192 ceph-mon[33110]: pgmap v401: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 783 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 880 op/s; 5005/38271 objects degraded (13.078%) 2024-03-26T03:08:15.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:15.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:15.140 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:15.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:15.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:15.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:15.363 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:08:15.363 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:08:15.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:15.478 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:15.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:15.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:15.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:15.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:15.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:15.848 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:08:15.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:15 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:08:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:15 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:08:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:15 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:08:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:15 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:08:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:15.911 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:15.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:15.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:15 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:08:15.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:15 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:08:16.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:16.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:16.147 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:16.205 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:16.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:16.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:16.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:16.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:16.584 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:16.634 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:08:16.715 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:08:16.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-26T03:08:16.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:16.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:16.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:08:16.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:08:16.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:08:16.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[29666]: pgmap v402: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 380 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 2398/18894 objects degraded (12.692%) 2024-03-26T03:08:16.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:08:16.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:08:16.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:08:16.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:16 smithi066 ceph-mon[37920]: pgmap v402: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 380 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 2398/18894 objects degraded (12.692%) 2024-03-26T03:08:16.876 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:16.940 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:16.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:16 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:08:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:16 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:08:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:16 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:08:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:16 smithi192 ceph-mon[33110]: pgmap v402: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 380 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 2398/18894 objects degraded (12.692%) 2024-03-26T03:08:17.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:17.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:17.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:17.222 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:17.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:17.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:17.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:17.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:17.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:17.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:17.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:17.793 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:17.846 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:08:17.846 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:08:17.846 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:08:17.846 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:08:17.846 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[29666]: osdmap e221: 8 total, 7 up, 8 in 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:08:17.847 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:17 smithi066 ceph-mon[37920]: osdmap e221: 8 total, 7 up, 8 in 2024-03-26T03:08:17.854 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:17.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:17.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:08:17.939 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:17.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:08:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:08:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:08:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:08:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:08:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:08:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:17 smithi192 ceph-mon[33110]: osdmap e221: 8 total, 7 up, 8 in 2024-03-26T03:08:18.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:18.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:18.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.222 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:18.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:18.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:18.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:18.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:18.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:18.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:18.644 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:18.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:18.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[29666]: pgmap v404: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 380 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 2398/18894 objects degraded (12.692%) 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:08:18.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:08:18.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:18.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:18 smithi066 ceph-mon[37920]: pgmap v404: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 380 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 2398/18894 objects degraded (12.692%) 2024-03-26T03:08:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:18.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:18.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:18.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:18.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:18 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:08:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:18 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:08:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:18 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:08:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:18 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:08:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:18 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:18 smithi192 ceph-mon[33110]: pgmap v404: 158 pgs: 28 active+undersized+degraded, 38 active+undersized, 92 active+clean; 380 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.02k op/s; 2398/18894 objects degraded (12.692%) 2024-03-26T03:08:19.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:19.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:19.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:19.172 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Removed 127388 objects 2024-03-26T03:08:19.172 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Clean up completed and total clean up time :101.651 2024-03-26T03:08:19.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:19.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:19.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:19.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:19.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:19.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:19.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:19.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:19.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:08:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:08:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:08:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 2398/18894 objects degraded (12.692%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/12528936' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:08:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-mon[33110]: from='client.35129 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:08:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:19 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:19.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:19.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:19.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:19.942 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:19.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 2398/18894 objects degraded (12.692%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/12528936' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[29666]: from='client.35129 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:08:19.964 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 2398/18894 objects degraded (12.692%), 28 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:19.965 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/12528936' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:08:19.965 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[37920]: from='client.35129 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:08:19.965 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:19 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:20.020 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:08:20.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:20.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:20.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:20.199 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:20.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:20.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:20.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:20.458 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:20.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:20.587 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:20.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:20.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:20.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:20.847 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:20.948 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:20.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:20.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:08:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:08:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:08:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:08:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:08:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: osdmap e222: 8 total, 7 up, 8 in 2024-03-26T03:08:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:20 smithi192 ceph-mon[33110]: pgmap v406: 158 pgs: 26 active+undersized+degraded, 40 active+undersized, 92 active+clean; 59 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s; 404/3492 objects degraded (11.569%) 2024-03-26T03:08:21.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:21.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:08:21.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: osdmap e222: 8 total, 7 up, 8 in 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[29666]: pgmap v406: 158 pgs: 26 active+undersized+degraded, 40 active+undersized, 92 active+clean; 59 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s; 404/3492 objects degraded (11.569%) 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:08:21.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:08:21.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:21.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: from='client.35129 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:08:21.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: osdmap e222: 8 total, 7 up, 8 in 2024-03-26T03:08:21.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:20 smithi066 ceph-mon[37920]: pgmap v406: 158 pgs: 26 active+undersized+degraded, 40 active+undersized, 92 active+clean; 59 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.59k op/s; 404/3492 objects degraded (11.569%) 2024-03-26T03:08:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:21.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:21.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:21.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:21.341 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:21.371 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:21.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:21.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:21.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:21.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:21.700 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-26T03:08:21.701 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-26T03:08:21.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:21.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:21.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:21.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:21.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:21.937 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:21 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:08:21.937 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:21 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:08:21.943 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:21.974 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:21 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:08:21.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:22.020 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:22.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:22.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:22.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:22.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:22.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:22.404 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:22.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:22.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:22.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:22.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:22.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:22.746 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:22 smithi066 ceph-mon[29666]: pgmap v407: 158 pgs: 25 active+undersized+degraded, 41 active+undersized, 92 active+clean; 571 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 115/702 objects degraded (16.382%) 2024-03-26T03:08:22.746 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:22 smithi066 ceph-mon[37920]: pgmap v407: 158 pgs: 25 active+undersized+degraded, 41 active+undersized, 92 active+clean; 571 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 115/702 objects degraded (16.382%) 2024-03-26T03:08:22.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:22.786 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:08:22.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:22.806 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:08:22.825 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:08:22.846 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:08:22.867 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:08:22.886 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:08:22.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:08:22.907 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:08:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:22.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:22.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:22 smithi192 ceph-mon[33110]: pgmap v407: 158 pgs: 25 active+undersized+degraded, 41 active+undersized, 92 active+clean; 571 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 115/702 objects degraded (16.382%) 2024-03-26T03:08:23.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:23.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:23.148 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:23.176 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:23.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.306 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:23.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:23.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:23.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:23.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:23.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:23.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.695 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:23.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:23.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:23.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2965744218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:23.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:23 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2965744218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:23.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:23.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:23.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:23.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:23 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2965744218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:24.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:24.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:08:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:08:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:08:24.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:24.213 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:24.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:08:24.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:24.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:24.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:24.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:24.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:24.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:24.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:24.733 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:24.753 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:08:24.753 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:08:24.753 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:08:24.753 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:08:24.754 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:08:24.835 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 115/702 objects degraded (16.382%), 25 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[29666]: pgmap v408: 158 pgs: 25 active+undersized+degraded, 41 active+undersized, 92 active+clean; 443 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s; 115/693 objects degraded (16.595%) 2024-03-26T03:08:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 115/702 objects degraded (16.382%), 25 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:24 smithi066 ceph-mon[37920]: pgmap v408: 158 pgs: 25 active+undersized+degraded, 41 active+undersized, 92 active+clean; 443 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s; 115/693 objects degraded (16.595%) 2024-03-26T03:08:24.863 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:24.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 115/702 objects degraded (16.382%), 25 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:24 smithi192 ceph-mon[33110]: pgmap v408: 158 pgs: 25 active+undersized+degraded, 41 active+undersized, 92 active+clean; 443 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s; 115/693 objects degraded (16.595%) 2024-03-26T03:08:24.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:24.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:25.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.123 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:25.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:25.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:25.251 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:25.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:25.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:25.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:25.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:25.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:25.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:25.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:25.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:25.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:25.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:25.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:26.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.034 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:26.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:26.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:26.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:26.198 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:26.265 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:26.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:26.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:08:26.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:26.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:26.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:26.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:26.656 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:26.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:08:26.689 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:08:26.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-26T03:08:26.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:26.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:26.918 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:26.947 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:26.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:27.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:27.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:27.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:27.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:27.215 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:08:27.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:27.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:27.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:27.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:27 smithi192 ceph-mon[33110]: pgmap v409: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 636 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:27.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:27 smithi066 ceph-mon[29666]: pgmap v409: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 636 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:27.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:27 smithi066 ceph-mon[37920]: pgmap v409: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 636 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:27.569 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:27.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:27.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:27.726 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:27.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:27.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:27.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:27.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:27.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:28.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:28.116 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:28.141 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:28.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:28.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:28.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:28.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:28.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:28.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:28.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:28.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:28.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:28.632 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:08:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:08:28.632 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 0 2024-03-26T03:08:28.632 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:08:28.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:28.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:28.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:28.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:28.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:28.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:29.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:29.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:29.247 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:29.267 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:29 smithi066 ceph-mon[29666]: pgmap v410: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 607 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:29.267 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 114/687 objects degraded (16.594%), 22 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:29.267 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:29 smithi066 ceph-mon[37920]: pgmap v410: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 607 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:29.267 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 114/687 objects degraded (16.594%), 22 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:29.275 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:29.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:29.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:29 smithi192 ceph-mon[33110]: pgmap v410: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 607 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:29.458 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 114/687 objects degraded (16.594%), 22 pgs degraded, 66 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:29.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:29.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:29.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:29.666 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:29.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:29.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:08:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:08:29.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:29.796 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:29.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:29.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:29.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:29.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:08:30.027 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:30.062 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:30.164 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:30.194 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:30.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:30.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:30.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:30.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:30.484 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:08:30.503 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:08:30.526 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:08:30.526 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:30.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:08:30.554 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:30.569 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:08:30.583 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:30.593 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:08:30.614 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:08:30.614 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:08:30.615 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:08:30.686 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:30.714 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:30.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:30.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:30.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:30.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:30.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:31.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:31.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:31.165 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:31.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:31.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:31.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:31.460 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:31.466 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:31.466 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:31 smithi066 ceph-mon[29666]: pgmap v411: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1011 B/s rd, 0 B/s wr, 93 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:31.466 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:31 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1636131954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:31.467 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:31.467 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:31 smithi066 ceph-mon[37920]: pgmap v411: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1011 B/s rd, 0 B/s wr, 93 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:31.467 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:31 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1636131954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:31 smithi192 ceph-mon[33110]: pgmap v411: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1011 B/s rd, 0 B/s wr, 93 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:31 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1636131954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.488 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:31.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:31.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:31.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:31.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:31.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:31.847 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:31.875 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:31.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:31.977 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:32.005 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:32.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:32.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:32.213 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:32.214 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_1' removed 2024-03-26T03:08:32.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:32.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:32.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:32.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:32 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:08:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:32 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:08:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:32 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:08:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:32 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:08:32.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/94954672' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:08:32.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:32.514 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:08:32.599 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/94954672' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:08:32.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:08:32.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:08:32.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:08:32.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:08:32.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/94954672' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:08:32.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:32.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:32.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:08:32.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:32.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:32.874 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:32.903 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:32.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:33.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:33.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:33.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:33.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:33.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:33.283 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:33.385 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:33.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:33 smithi192 ceph-mon[33110]: pgmap v412: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 78 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:33 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/94954672' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:08:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:33 smithi192 ceph-mon[33110]: osdmap e223: 8 total, 7 up, 8 in 2024-03-26T03:08:33.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:33.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-mon[29666]: pgmap v412: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 78 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/94954672' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:08:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-mon[29666]: osdmap e223: 8 total, 7 up, 8 in 2024-03-26T03:08:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-mon[37920]: pgmap v412: 158 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 44 active+undersized, 91 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 78 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/94954672' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:08:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-mon[37920]: osdmap e223: 8 total, 7 up, 8 in 2024-03-26T03:08:33.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:33.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:33.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:33.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:33.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:33.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:33.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:33.942 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:08:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:08:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:08:34.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:34.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:34.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:34.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:34.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:34.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:34.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:34.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:34.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:34.549 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:34.650 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:34.680 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:08:34.680 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:08:34.681 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-26T03:08:34.681 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-26T03:08:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.682 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:34.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:34.732 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:08:34.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:34.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:34.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:34.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:34.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:35.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:35.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:35.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:35.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:35.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:35.295 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:35 smithi066 ceph-mon[29666]: pgmap v414: 142 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:35.295 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:35 smithi066 ceph-mon[37920]: pgmap v414: 142 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:35.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:35.410 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:35.438 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:35 smithi192 ceph-mon[33110]: pgmap v414: 142 pgs: 1 active+clean+scrubbing, 22 active+undersized+degraded, 38 active+undersized, 81 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:35.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:35.567 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:35.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:35.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:35.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:35.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:35.926 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:35.954 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:35.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:36.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:36.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:36.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:36.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:36.237 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:36 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 114/687 objects degraded (16.594%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:36.238 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:36 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 114/687 objects degraded (16.594%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:36.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:36.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:36.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:36 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 114/687 objects degraded (16.594%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:36.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:36.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:36.643 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:08:36.693 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.723 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:36.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:36.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:36.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:36.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:36.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:36.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:37.039 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:08:37.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:37.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:37.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:37.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:37.332 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:37.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:37 smithi066 ceph-mon[29666]: pgmap v415: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:37.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3439878462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:37.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:37 smithi066 ceph-mon[29666]: from='client.35159 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:37.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:37 smithi066 ceph-mon[37920]: pgmap v415: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:37.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3439878462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:37.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:37 smithi066 ceph-mon[37920]: from='client.35159 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:37.360 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:37.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:37.477 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:37.485 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:08:37.485 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:08:37.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:37 smithi192 ceph-mon[33110]: pgmap v415: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3439878462' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:37 smithi192 ceph-mon[33110]: from='client.35159 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:08:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:37.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:37.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:37.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:37.744 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:37.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:37.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:37.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:37.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:38.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:38.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:38.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:38.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:38.237 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:38 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:38.237 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:38 smithi066 ceph-mon[29666]: from='client.35159 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:08:38.237 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:38 smithi066 ceph-mon[29666]: osdmap e224: 8 total, 7 up, 8 in 2024-03-26T03:08:38.237 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:38 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:38.237 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:38 smithi066 ceph-mon[37920]: from='client.35159 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:08:38.237 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:38 smithi066 ceph-mon[37920]: osdmap e224: 8 total, 7 up, 8 in 2024-03-26T03:08:38.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:38.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:38.388 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:38 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:38 smithi192 ceph-mon[33110]: from='client.35159 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:08:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:38 smithi192 ceph-mon[33110]: osdmap e224: 8 total, 7 up, 8 in 2024-03-26T03:08:38.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:38.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:38.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:38.649 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:38.672 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:08:38.673 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-26T03:08:38.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:38.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:38.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:38.909 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:38.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:39.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:39.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:39.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:39.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:39.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:39.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:39.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:39 smithi066 ceph-mon[29666]: pgmap v417: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:39.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:39 smithi066 ceph-mon[37920]: pgmap v417: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:39.399 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:39.429 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:39.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:39 smithi192 ceph-mon[33110]: pgmap v417: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:39.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:39.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:39.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:39.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:39.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:08:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:08:39.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:39.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:39.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:08:39.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:08:39.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:39.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:39.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:39.949 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:39.950 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:40.046 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:40.074 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:40.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:40.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:40.176 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:40.203 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:40.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:40.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:40.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:40.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:40.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:40.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:40.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:40.725 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:40.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:40.854 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:40.956 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:40.984 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:40.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:41.012 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:41.013 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:41.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:41.114 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:41.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:41.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:41.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:41 smithi066 ceph-mon[29666]: pgmap v418: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:41 smithi066 ceph-mon[37920]: pgmap v418: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:41.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:41.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:41 smithi192 ceph-mon[33110]: pgmap v418: 142 pgs: 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:41.500 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:41.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:41.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:41.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:41.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:41.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:41.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:41.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:41.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:42.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:42.053 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_2' created 2024-03-26T03:08:42.118 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:42.147 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:42.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:42.276 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:42.293 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1767386063' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:08:42.293 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:42 smithi066 ceph-mon[29666]: from='client.35171 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:08:42.293 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:42 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1767386063' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:08:42.293 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:42 smithi066 ceph-mon[37920]: from='client.35171 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:08:42.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:42.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:42 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1767386063' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:08:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:42 smithi192 ceph-mon[33110]: from='client.35171 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:08:42.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:42.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:42.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:42.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:42.766 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:42.794 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:42.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:42.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:42.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:43.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:43.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:43.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-mon[29666]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T03:08:43.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-mon[29666]: osdmap e225: 8 total, 7 up, 8 in 2024-03-26T03:08:43.093 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-mon[29666]: pgmap v420: 158 pgs: 16 unknown, 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:43.093 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-mon[37920]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T03:08:43.093 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-mon[37920]: osdmap e225: 8 total, 7 up, 8 in 2024-03-26T03:08:43.093 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-mon[37920]: pgmap v420: 158 pgs: 16 unknown, 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:43.156 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:43.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:43.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:43.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:43.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:43.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:43 smithi192 ceph-mon[33110]: from='client.35171 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T03:08:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:43 smithi192 ceph-mon[33110]: osdmap e225: 8 total, 7 up, 8 in 2024-03-26T03:08:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:43 smithi192 ceph-mon[33110]: pgmap v420: 158 pgs: 16 unknown, 22 active+undersized+degraded, 38 active+undersized, 82 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:43.543 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.571 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:43.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:43.672 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:43.699 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:43.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:43.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:43.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:43.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:43.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:44.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-26T03:08:44.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:44.088 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:44.088 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:44 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:08:44.088 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:44 smithi066 ceph-mon[29666]: osdmap e226: 8 total, 7 up, 8 in 2024-03-26T03:08:44.088 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:44 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:08:44.088 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:44 smithi066 ceph-mon[37920]: osdmap e226: 8 total, 7 up, 8 in 2024-03-26T03:08:44.089 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:08:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:08:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:08:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:44.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:44.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:44.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:44.349 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:44.450 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:44.477 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:44 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:08:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:44 smithi192 ceph-mon[33110]: osdmap e226: 8 total, 7 up, 8 in 2024-03-26T03:08:44.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:44.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:44.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:44.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:44.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:44.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:44.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:44.965 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:44.994 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:44.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:45.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:45.123 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:45.148 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:08:45.167 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:08:45.168 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:45 smithi066 ceph-mon[29666]: osdmap e227: 8 total, 7 up, 8 in 2024-03-26T03:08:45.168 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:45 smithi066 ceph-mon[29666]: pgmap v423: 158 pgs: 12 unknown, 22 active+undersized+degraded, 40 active+undersized, 84 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:45.168 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:45 smithi066 ceph-mon[37920]: osdmap e227: 8 total, 7 up, 8 in 2024-03-26T03:08:45.168 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:45 smithi066 ceph-mon[37920]: pgmap v423: 158 pgs: 12 unknown, 22 active+undersized+degraded, 40 active+undersized, 84 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:45.187 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:08:45.208 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:08:45.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:45.229 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:08:45.250 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:08:45.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:45.271 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:08:45.272 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:08:45.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:26626 7.e 7:77b70000:::benchmark_data_smithi066_76917_object26625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:14.072938+0000", 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 451.28145532000002, 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14604862399999999, 2024-03-26T03:08:45.357 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 26626 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.072938+0000", 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.358 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.072938+0000", 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.072940+0000", 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.838e-06 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.359 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.072973+0000", 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3806000000000001e-05 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.072984+0000", 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1003e-05 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.072993+0000", 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2579999999999995e-06 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.073063+0000", 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0643999999999997e-05 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.073194+0000", 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013037699999999999 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.073340+0000", 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014682599999999999 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.075230+0000", 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018896900000000001 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.075455+0000", 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000225305 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.218870+0000", 2024-03-26T03:08:45.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14341498 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.218943+0000", 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2087000000000003e-05 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:14.218986+0000", 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3810000000000002e-05 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:30132 7.e 7:735f32a3:::benchmark_data_smithi066_76917_object30131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:16.592452+0000", 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 448.76194134899998, 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.164402568, 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 30132 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.364 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592452+0000", 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592452+0000", 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592453+0000", 2024-03-26T03:08:45.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3209999999999999e-06 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592477+0000", 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4068999999999999e-05 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592486+0000", 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1200000000000008e-06 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.366 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592494+0000", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0020000000000006e-06 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592606+0000", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000111706 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592717+0000", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011123199999999999 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.592973+0000", 2024-03-26T03:08:45.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025556500000000002 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.594956+0000", 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001983542 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.595328+0000", 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000371694 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.368 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.756748+0000", 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.16142050799999999 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.756807+0000", 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8696000000000002e-05 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.756854+0000", 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7113000000000001e-05 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.369 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:30492 7.8 7:142cea31:::benchmark_data_smithi066_76917_object30491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:16.928472+0000", 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 448.42592105199998, 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.15103614400000001, 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.370 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 30492 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.928472+0000", 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.928472+0000", 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.371 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.928474+0000", 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.578e-06 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.928498+0000", 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4802000000000001e-05 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.928507+0000", 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9789999999999999e-06 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.372 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.928515+0000", 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6650000000000003e-06 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.992546+0000", 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.064031432999999999 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.992735+0000", 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018902 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:08:45.373 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:16.995597+0000", 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028617880000000001 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:17.079269+0000", 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.083671998999999997 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:17.079339+0000", 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9369999999999995e-05 2024-03-26T03:08:45.374 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:17.079406+0000", 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6993e-05 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:17.079460+0000", 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3999000000000003e-05 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:17.079508+0000", 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8517999999999999e-05 2024-03-26T03:08:45.375 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:70270 7.8 7:1cea7b20:::benchmark_data_smithi066_76917_object70269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:45.094050+0000", 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 420.260342987, 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.12378848200000001, 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.376 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 70270 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094050+0000", 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094050+0000", 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.377 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094052+0000", 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.564e-06 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094077+0000", 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5296e-05 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094087+0000", 2024-03-26T03:08:45.378 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0365e-05 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094096+0000", 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3489999999999992e-06 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094160+0000", 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4758999999999994e-05 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.379 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094287+0000", 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012624399999999999 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.094371+0000", 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4689000000000002e-05 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.096258+0000", 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018866499999999999 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.380 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.096574+0000", 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000316273 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.217704+0000", 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.121129404 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.217797+0000", 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3315999999999999e-05 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.381 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:45.217839+0000", 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1573000000000003e-05 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75519 7.e 7:7efeb81d:::benchmark_data_smithi066_76917_object75518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.265486+0000", 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 415.08890682600003, 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.28001805699999999, 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.382 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75519 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.265486+0000", 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.383 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.265486+0000", 2024-03-26T03:08:45.384 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.384 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.384 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.385 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.385 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.265487+0000", 2024-03-26T03:08:45.385 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2869999999999999e-06 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.265510+0000", 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2863000000000001e-05 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.265520+0000", 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3659999999999998e-06 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.386 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.265527+0000", 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5900000000000002e-06 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.306403+0000", 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.040875947000000003 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.306764+0000", 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00036116099999999998 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.387 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.307232+0000", 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000467911 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.312466+0000", 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0052334010000000004 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.327175+0000", 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014708938 2024-03-26T03:08:45.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.545398+0000", 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21822354199999999 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.545455+0000", 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6804e-05 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.545504+0000", 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9246999999999998e-05 2024-03-26T03:08:45.389 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75520 7.8 7:1c9c4f40:::benchmark_data_smithi066_76917_object75519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.267784+0000", 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 415.08660921299997, 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19460102200000001, 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.390 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75520 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267784+0000", 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267784+0000", 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267786+0000", 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9989999999999998e-06 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267811+0000", 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5347e-05 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267823+0000", 2024-03-26T03:08:45.392 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1800000000000001e-05 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267831+0000", 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4840000000000004e-06 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.267900+0000", 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8582000000000002e-05 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.268031+0000", 2024-03-26T03:08:45.393 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000131027 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.268200+0000", 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016875599999999999 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.272394+0000", 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041939660000000004 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.394 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.273030+0000", 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00063582299999999999 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.462365+0000", 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.18933565099999999 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.462385+0000", 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9587000000000001e-05 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.395 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:75526 7.3 e110/96)", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.278632+0000", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 415.07576052600001, 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.17406073999999999, 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.278632+0000", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.278632+0000", 2024-03-26T03:08:45.396 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.278635+0000", 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7819999999999998e-06 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.278670+0000", 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4465999999999998e-05 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.278682+0000", 2024-03-26T03:08:45.397 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2619999999999999e-05 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.278693+0000", 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0284e-05 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.452668+0000", 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.173975819 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.398 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.452693+0000", 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4769e-05 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.14841.0:75530 7.5 e110/96)", 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.287075+0000", 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 415.06731792199997, 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.202767895, 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.399 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.287075+0000", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.287075+0000", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.287077+0000", 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9089999999999998e-06 2024-03-26T03:08:45.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.287131+0000", 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3862e-05 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.287146+0000", 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5107e-05 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.287157+0000", 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0659000000000001e-05 2024-03-26T03:08:45.401 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.489825+0000", 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20266798999999999 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.489843+0000", 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8368000000000001e-05 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75520 7.18 7:1c9c4f40:::benchmark_data_smithi066_76917_object75519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.308510+0000", 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 415.04588267700001, 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0118079440000001, 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75520 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.403 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308510+0000", 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308510+0000", 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308513+0000", 2024-03-26T03:08:45.404 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5000000000000002e-06 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308540+0000", 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6883e-05 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308559+0000", 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9474999999999999e-05 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.405 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.308569+0000", 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5419999999999994e-06 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.463020+0000", 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.15445110100000001 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.463030+0000", 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0091999999999999e-05 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.315609+0000", 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.85257883199999995 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for degraded object", 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.318082+0000", 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024734309999999999 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.320226+0000", 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0021440399999999998 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:51.320318+0000", 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2047999999999997e-05 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:75534 7.e 7:7492f940:::benchmark_data_smithi066_76917_object75533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:50.317713+0000", 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 415.03668030699998, 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.22790139400000001, 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.408 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 75534 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317713+0000", 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.409 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317713+0000", 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317716+0000", 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2789999999999999e-06 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317750+0000", 2024-03-26T03:08:45.410 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3861e-05 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317773+0000", 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3584999999999999e-05 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317787+0000", 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3331e-05 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.411 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.317858+0000", 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1223000000000006e-05 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.318027+0000", 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016932000000000001 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.318195+0000", 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016811300000000001 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.327213+0000", 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0090173040000000003 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.328717+0000", 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0015046720000000001 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.545542+0000", 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.216824933 2024-03-26T03:08:45.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.545583+0000", 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0658000000000003e-05 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:50.545614+0000", 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.1115000000000003e-05 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76596 7.18 7:1a96fdb0:::benchmark_data_smithi066_76917_object76595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.311018+0000", 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 412.043375137, 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4098060610000001, 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76596 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311018+0000", 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311018+0000", 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311020+0000", 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.77e-06 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311048+0000", 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7951000000000001e-05 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311058+0000", 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0553e-05 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311068+0000", 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0351e-05 2024-03-26T03:08:45.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311395+0000", 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00032647299999999998 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311404+0000", 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4129999999999995e-06 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.418 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.697207+0000", 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3858024929999999 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.697362+0000", 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000154723 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,6,7", 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.697548+0000", 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000185918 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.710971+0000", 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013423740999999999 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.711185+0000", 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021397300000000001 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.715253+0000", 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004067569 2024-03-26T03:08:45.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.717533+0000", 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022803229999999999 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.717677+0000", 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014351300000000001 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.720824+0000", 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031472969999999999 2024-03-26T03:08:45.421 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76601 7.18 7:19aa8995:::benchmark_data_smithi066_76917_object76600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.311083+0000", 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 412.04330954599999, 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4186896609999999, 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.422 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76601 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311083+0000", 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.423 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311083+0000", 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311084+0000", 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3300000000000001e-07 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311124+0000", 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9854999999999997e-05 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311133+0000", 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3479999999999993e-06 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311139+0000", 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0639999999999999e-06 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311428+0000", 2024-03-26T03:08:45.425 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00028874400000000001 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311435+0000", 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3339999999999998e-06 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.711348+0000", 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.399912993 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.711718+0000", 2024-03-26T03:08:45.426 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000370727 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,6,7", 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.712236+0000", 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00051796999999999998 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.720933+0000", 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0086963240000000001 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.427 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.727781+0000", 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00684799 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.727878+0000", 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7081000000000001e-05 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.727929+0000", 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1298000000000003e-05 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.428 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.727999+0000", 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0351000000000005e-05 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.729773+0000", 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017738490000000001 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76606 7.18 7:19028c53:::benchmark_data_smithi066_76917_object76605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e114)", 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:53.311146+0000", 2024-03-26T03:08:45.429 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 412.043246598, 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4201247420000001, 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76606 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311146+0000", 2024-03-26T03:08:45.430 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311146+0000", 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311147+0000", 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4399999999999999e-07 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311170+0000", 2024-03-26T03:08:45.431 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3195000000000001e-05 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311176+0000", 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8839999999999999e-06 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311181+0000", 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2159999999999997e-06 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.432 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311456+0000", 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00027444900000000001 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:53.311462+0000", 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9279999999999998e-06 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.713815+0000", 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4023533269999999 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.433 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.713990+0000", 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000174976 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,6,7", 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.714236+0000", 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00024607699999999998 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.720980+0000", 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0067442070000000003 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.434 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.727832+0000", 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0068516150000000001 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.729848+0000", 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020156319999999998 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.729990+0000", 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014241899999999999 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.435 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.730069+0000", 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9354e-05 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.731271+0000", 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001201719 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.436 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:76634 7.e 7:7d07fe8e:::benchmark_data_smithi066_76917_object76633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:01:54.461237+0000", 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 410.89315554199999, 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.144727946, 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 76634 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.437 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.461237+0000", 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.461237+0000", 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.461241+0000", 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3100000000000001e-06 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.461275+0000", 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3958000000000003e-05 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.462706+0000", 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001431119 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.462725+0000", 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9582999999999999e-05 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.462810+0000", 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4726000000000002e-05 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.462995+0000", 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000185203 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.463161+0000", 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000165277 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.466106+0000", 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0029450750000000001 2024-03-26T03:08:45.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.469313+0000", 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0032072239999999998 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.605843+0000", 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13652963700000001 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.605915+0000", 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2312999999999997e-05 2024-03-26T03:08:45.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:01:54.605965+0000", 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0521e-05 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:109861 7.e 7:7e39e292:::benchmark_data_smithi066_76917_object9678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:48.041963+0000", 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 357.31242962699997, 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.130038182, 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.444 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 109861 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.041963+0000", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.041963+0000", 2024-03-26T03:08:45.445 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.041965+0000", 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6700000000000001e-06 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.041972+0000", 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7739999999999999e-06 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.446 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.041984+0000", 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1899000000000001e-05 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.041993+0000", 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.251e-06 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.046106+0000", 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004112531 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.447 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.046281+0000", 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000175556 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.046419+0000", 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013799999999999999 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.052773+0000", 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0063537710000000002 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.448 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.065784+0000", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013010704999999999 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.171909+0000", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10612558800000001 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.171961+0000", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1487999999999999e-05 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.172002+0000", 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0948999999999997e-05 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:110048 7.13 7:cfe73ebd:::benchmark_data_smithi066_76917_object9865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:48.322625+0000", 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 357.03176835599999, 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.123160705, 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.450 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 110048 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.322625+0000", 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.322625+0000", 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.451 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.322626+0000", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.271e-06 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.322631+0000", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7820000000000001e-06 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.322639+0000", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0639999999999994e-06 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.452 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.322645+0000", 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7100000000000001e-06 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.428472+0000", 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.105826499 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.428603+0000", 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013116899999999999 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:08:45.453 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.428904+0000", 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00030035800000000003 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.437769+0000", 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0088657969999999999 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.455 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.439150+0000", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013805480000000001 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.445408+0000", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0062577020000000004 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.445468+0000", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0359000000000003e-05 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:48.445785+0000", 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00031744599999999998 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:111988 7.13 7:cd92f64e:::benchmark_data_smithi066_76917_object11805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:50.993368+0000", 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 354.36102490500002, 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.127073942, 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.457 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 111988 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993368+0000", 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993368+0000", 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993370+0000", 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.855e-06 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993377+0000", 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0509999999999999e-06 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993387+0000", 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0125999999999999e-05 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.459 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993395+0000", 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7540000000000006e-06 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.993949+0000", 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00055424900000000004 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.994082+0000", 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013310199999999999 2024-03-26T03:08:45.460 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.994407+0000", 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00032461499999999999 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:50.996049+0000", 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001642131 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.001487+0000", 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0054383230000000001 2024-03-26T03:08:45.461 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.120161+0000", 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11867327599999999 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.120233+0000", 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2212000000000006e-05 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.120442+0000", 2024-03-26T03:08:45.462 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020924800000000001 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:111995 7.e 7:78047173:::benchmark_data_smithi066_76917_object11812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:51.000205+0000", 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 354.35418767200002, 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13649815200000001, 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.463 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 111995 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.000205+0000", 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.000205+0000", 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.000207+0000", 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8449999999999999e-06 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.000213+0000", 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2609999999999999e-06 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.000223+0000", 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0047000000000001e-05 2024-03-26T03:08:45.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.000231+0000", 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.503e-06 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.001014+0000", 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00078299100000000003 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.001148+0000", 2024-03-26T03:08:45.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013359 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.001262+0000", 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011397599999999999 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.003959+0000", 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002697942 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.004590+0000", 2024-03-26T03:08:45.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00063084199999999995 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.136615+0000", 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13202496599999999 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.136660+0000", 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4836999999999998e-05 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:51.136703+0000", 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3352000000000001e-05 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "MOSDPGPush(7.10 131/114 [PushOp(7:09daf299:::benchmark_data_smithi066_76917_object13117:head, version: 99'796, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:09daf299:::benchmark_data_smithi066_76917_object13117:head@99'796, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:52.137163+0000", 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 353.21722979800001, 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.144391089, 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:08:45.469 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.137163+0000", 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.137163+0000", 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.137165+0000", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3200000000000001e-06 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.137195+0000", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.0562999999999999e-05 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.274269+0000", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13707381099999999 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.274305+0000", 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.5695000000000003e-05 2024-03-26T03:08:45.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.280051+0000", 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0057462600000000004 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.280066+0000", 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5125e-05 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.281554+0000", 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001488315 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.472 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.14841.0:112711 7.e 7:7ab3d34e:::benchmark_data_smithi066_76917_object12528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:02:52.515541+0000", 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 352.83885225300003, 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.16869362199999999, 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.14841", 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 112711 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.473 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515541+0000", 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515541+0000", 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515543+0000", 2024-03-26T03:08:45.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2790000000000001e-06 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515549+0000", 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1809999999999998e-06 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515561+0000", 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2228999999999999e-05 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:08:45.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515570+0000", 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3149999999999992e-06 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515625+0000", 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5217999999999999e-05 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515747+0000", 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000121611 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,5", 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.515872+0000", 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012566899999999999 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.520801+0000", 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049282850000000001 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.520816+0000", 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5038999999999999e-05 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.682133+0000", 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.16131754800000001 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.684177+0000", 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020439450000000001 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:02:52.684234+0000", 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7302999999999997e-05 2024-03-26T03:08:45.478 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.479 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.479 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.479 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:08:45.479 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:08:45.479 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:08:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:45.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:45 smithi192 ceph-mon[33110]: osdmap e227: 8 total, 7 up, 8 in 2024-03-26T03:08:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:45 smithi192 ceph-mon[33110]: pgmap v423: 158 pgs: 12 unknown, 22 active+undersized+degraded, 40 active+undersized, 84 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:45.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:45.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:45.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:45.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:45.869 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:45.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:45.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:45.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:46.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:46.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/474419364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:46.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/474419364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:46.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:46.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:46.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:46.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:46.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:08:46.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:46.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:46.482 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/474419364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.511 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:46.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:46.611 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:46.638 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:46.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:46.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:46.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:46.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:46.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:46.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:47.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:47.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.158 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:47.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:47.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:47.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:47.311 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:47 smithi066 ceph-mon[29666]: pgmap v424: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:47.311 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:47 smithi066 ceph-mon[37920]: pgmap v424: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:47.392 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.421 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:47.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:47 smithi192 ceph-mon[33110]: pgmap v424: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:47.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:47.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:47.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:08:47.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:47.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:47.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:47.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:47.881 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:08:47.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:47.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:47.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:48.052 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:48.080 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:48.182 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:48.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:48.288 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:48.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:08:48.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:48.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:48.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:48.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:48.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:48.593 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:48.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:48.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:08:48.734 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:08:48.735 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:08:48.735 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:08:48.735 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:08:48.735 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:08:48.735 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:08:48.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:48.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:48.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:48.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:48.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:49.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:49.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:49.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:49.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:49.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:49 smithi192 ceph-mon[33110]: pgmap v425: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:49.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:49.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:49.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:49.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:49.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:49 smithi066 ceph-mon[29666]: pgmap v425: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:49.533 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:49 smithi066 ceph-mon[37920]: pgmap v425: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:49.557 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:08:49.557 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:08:49.557 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:08:49.557 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:08:49.557 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:08:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:08:49.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:49.605 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:49.707 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:49.734 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:08:49.832 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:08:49.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-26T03:08:49.837 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:49.902 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:49.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:50.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:50.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:50.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:50.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:08:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:50.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:50.259 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:50.286 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:50.374 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:50.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:50.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:50.415 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:08:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:50.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:50.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:50.642 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:50.671 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:50.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:50.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:50.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:50.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:50.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:51.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:51.058 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:51.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:51.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:51.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:51.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:51.349 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:51.349 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:51 smithi066 ceph-mon[29666]: pgmap v426: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:51.349 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:51.350 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:51 smithi066 ceph-mon[37920]: pgmap v426: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:51.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:51.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:51.456 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:08:51.457 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:51 smithi192 ceph-mon[33110]: pgmap v426: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:51.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:51.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:51.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:51.692 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:51.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:51.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:51.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:51.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:51.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:52.037 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:08:52.037 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:08:52.037 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:08:52.037 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:08:52.037 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:08:52.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:52.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:52.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:52.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:52.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:52.422 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:52.451 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:52.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:52.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:52.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:08:52.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:52.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:52.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:52.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:52.875 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:08:52.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:52.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:52.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:53.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:53.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:53.181 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:53.210 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-26T03:08:53.212 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:53.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:53.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:53 smithi192 ceph-mon[33110]: pgmap v427: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:53.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1555081177' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:08:53.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:53.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:53.465 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:53.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:53.560 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:53 smithi066 ceph-mon[29666]: pgmap v427: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:53.560 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1555081177' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:08:53.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:53 smithi066 ceph-mon[37920]: pgmap v427: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:53.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1555081177' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:08:53.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:53.593 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:53.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:53.722 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:53.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:53.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:53.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:53.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:53.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:54.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:54.082 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:08:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:08:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:08:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:54.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:54.207 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:54.271 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:08:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.303 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:54.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:54.351 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1555081177' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:08:54.351 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[29666]: osdmap e228: 8 total, 7 up, 8 in 2024-03-26T03:08:54.351 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/873555738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1555081177' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[37920]: osdmap e228: 8 total, 7 up, 8 in 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/873555738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:54.352 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:54.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:54.427 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:08:54.427 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:08:54.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:54 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1555081177' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:08:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:54 smithi192 ceph-mon[33110]: osdmap e228: 8 total, 7 up, 8 in 2024-03-26T03:08:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:54 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/873555738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:08:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:08:54.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:08:54.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:54.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:08:54.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:08:54.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:08:54.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:08:54.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:08:54.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:08:54.549 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:08:54.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:54.658 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:54.686 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:54.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:54.816 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:54.870 DEBUG:teuthology.orchestra.run.smithi066:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-26T03:08:54.873 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:08:54.874 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:08:54.918 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:54.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:hints = 1 2024-03-26T03:08:54.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:08:54.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object prefix: benchmark_data_smithi066_125102 2024-03-26T03:08:54.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:08:54.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:08:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:54.968 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:54.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:55.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:55.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:08:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.113 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:55.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:55.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:55.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:55.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:55.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: pgmap v429: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/873555738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: osdmap e229: 8 total, 7 up, 8 in 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/498901602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:08:55.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:08:55.470 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: pgmap v429: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:55.470 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:55.470 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/873555738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:08:55.473 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: osdmap e229: 8 total, 7 up, 8 in 2024-03-26T03:08:55.473 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/498901602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:55.473 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:08:55.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:08:55.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:55 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:08:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: pgmap v429: 158 pgs: 22 active+undersized+degraded, 46 active+undersized, 90 active+clean; 315 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 114/687 objects degraded (16.594%) 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/873555738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: osdmap e229: 8 total, 7 up, 8 in 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/498901602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:08:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:55 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:08:55.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:55.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:55.542 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:08:55.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:55.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:55.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:55.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:08:55.852 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:08:55.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:55.921 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 1 15 4088 4073 254.55 254.562 0.00244719 0.00391672 2024-03-26T03:08:55.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:55.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:56.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:08:56.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:08:56.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:56.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:56.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:56.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:56.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:56.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:56.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:56 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 239/2025 objects degraded (11.802%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:56.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:56 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 239/2025 objects degraded (11.802%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:56.647 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:56.713 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:56 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 239/2025 objects degraded (11.802%), 23 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:08:56.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:56.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:56.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:56.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 2 16 7949 7933 247.89 241.25 0.00266293 0.00402617 2024-03-26T03:08:56.935 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:56.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:57.037 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.085 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:57.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:57.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.231 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:57.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:57.319 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:57 smithi066 ceph-mon[29666]: pgmap v431: 158 pgs: 23 active+undersized+degraded, 45 active+undersized, 90 active+clean; 28 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 56 op/s; 239/2025 objects degraded (11.802%) 2024-03-26T03:08:57.319 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:57 smithi066 ceph-mon[37920]: pgmap v431: 158 pgs: 23 active+undersized+degraded, 45 active+undersized, 90 active+clean; 28 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 56 op/s; 239/2025 objects degraded (11.802%) 2024-03-26T03:08:57.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:57.348 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:08:57.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:57.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:57.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:57.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:57 smithi192 ceph-mon[33110]: pgmap v431: 158 pgs: 23 active+undersized+degraded, 45 active+undersized, 90 active+clean; 28 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 56 op/s; 239/2025 objects degraded (11.802%) 2024-03-26T03:08:57.601 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:57.654 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:57.655 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:08:57.655 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:08:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:57.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:57.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:57.853 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:08:57.875 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:08:57.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:57.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 3 16 11121 11105 231.337 198.25 0.00406489 0.00431594 2024-03-26T03:08:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:57.964 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:57.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:58.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:58.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:58.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:58.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:58.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:08:58.342 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.377 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:58.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:58.478 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:58.521 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:08:58.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:08:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:08:58.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:58 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:08:58.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:58.665 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:58.726 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:08:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:08:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:08:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:08:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:08:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:08:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:08:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:08:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:58 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:08:58.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:08:58.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:08:58.913 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:58.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 4 16 14181 14165 221.312 191.25 0.00695604 0.00451252 2024-03-26T03:08:58.956 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:58.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:08:58.991 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:08:58.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-26T03:08:59.018 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:59.058 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:08:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.124 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:59.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:08:59.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:59.277 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:59.379 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:08:59.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:08:59.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:08:59 smithi192 ceph-mon[33110]: pgmap v432: 158 pgs: 23 active+undersized+degraded, 45 active+undersized, 90 active+clean; 28 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 3.5 MiB/s wr, 56 op/s; 239/2025 objects degraded (11.802%) 2024-03-26T03:08:59.508 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:08:59 smithi066 ceph-mon[29666]: pgmap v432: 158 pgs: 23 active+undersized+degraded, 45 active+undersized, 90 active+clean; 28 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 3.5 MiB/s wr, 56 op/s; 239/2025 objects degraded (11.802%) 2024-03-26T03:08:59.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:08:59 smithi066 ceph-mon[37920]: pgmap v432: 158 pgs: 23 active+undersized+degraded, 45 active+undersized, 90 active+clean; 28 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 3.5 MiB/s wr, 56 op/s; 239/2025 objects degraded (11.802%) 2024-03-26T03:08:59.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:08:59.573 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:08:59.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:59.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:59.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:08:59.821 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:08:59.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:08:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:08:59.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:08:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:08:59.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 5 16 17175 17159 214.465 187.125 0.00366935 0.00465636 2024-03-26T03:08:59.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:08:59.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:08:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:08:59.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:00.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:00.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:00.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:00.273 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:00.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:09:00.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:00 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:09:00.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:00.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:00.475 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:00 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:09:00.475 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:00 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:09:00.475 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:00 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:09:00.475 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:00 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:09:00.476 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:00 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:09:00.476 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:00 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:09:00.510 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:00.552 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:00.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:00.689 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:09:00.689 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:09:00.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:00.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:00.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:00.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 6 16 20049 20033 208.656 179.625 0.00230356 0.00478752 2024-03-26T03:09:00.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:00.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:00.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:01.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:01.114 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:01.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:01.255 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:01.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.399 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:01.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:01.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:01.536 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[29666]: pgmap v433: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 748 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s; 6877/36588 objects degraded (18.796%) 2024-03-26T03:09:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:09:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:09:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[37920]: pgmap v433: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 748 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s; 6877/36588 objects degraded (18.796%) 2024-03-26T03:09:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:09:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:01 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:09:01.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:01.679 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:01 smithi192 ceph-mon[33110]: pgmap v433: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 748 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s; 6877/36588 objects degraded (18.796%) 2024-03-26T03:09:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:01 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:09:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:01 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:09:01.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:01.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:01.885 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:01.918 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:01.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 7 15 22782 22767 203.254 170.875 0.00947701 0.00491485 2024-03-26T03:09:01.970 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:01.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:02.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:02.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:02.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:09:02.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:02.250 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:02.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:02.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:02.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:02 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:09:02.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:02 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:09:02.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:02.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:02.607 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:02 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:09:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:02 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:09:02.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:02 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:09:02.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:02 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:09:02.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:02.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:02.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:02.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:02.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 8 16 25435 25419 198.565 165.75 0.00587211 0.00502974 2024-03-26T03:09:02.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:02.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:02.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:03.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:03.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:03.209 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:03.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:03.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:03.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:03.495 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:03.534 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:03.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:03 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:09:03.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:03 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:09:03.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:03 smithi192 ceph-mon[33110]: pgmap v434: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1009 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.80k op/s; 7850/49089 objects degraded (15.991%) 2024-03-26T03:09:03.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:03 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 7850/49089 objects degraded (15.991%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:03.575 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:03 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:09:03.576 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:03 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:09:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:09:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:09:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[29666]: pgmap v434: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1009 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.80k op/s; 7850/49089 objects degraded (15.991%) 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 7850/49089 objects degraded (15.991%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[37920]: pgmap v434: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1009 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.80k op/s; 7850/49089 objects degraded (15.991%) 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 7850/49089 objects degraded (15.991%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:09:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:09:03.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:03.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:03.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:03.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:03.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:03.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:03.915 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:09:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:09:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:09:03.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 9 16 27565 27549 191.293 133.125 0.0146154 0.00522115 2024-03-26T03:09:03.960 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:03.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:04.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:04.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:04.195 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.239 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:04.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:04.340 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:04.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[29666]: osdmap e230: 8 total, 7 up, 8 in 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:09:04.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[37920]: osdmap e230: 8 total, 7 up, 8 in 2024-03-26T03:09:04.488 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:09:04.488 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:04 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:09:04.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:04.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:04.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:04 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:09:04.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:04 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:09:04.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:04 smithi192 ceph-mon[33110]: osdmap e230: 8 total, 7 up, 8 in 2024-03-26T03:09:04.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:04 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:09:04.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:04 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:09:04.611 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:09:04.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:04.656 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:09:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.670 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:04.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:04.695 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:09:04.731 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:09:04.771 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:09:04.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:04.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:04.808 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:09:04.858 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:09:04.858 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:09:04.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:09:04.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:04.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 10 16 29728 29712 185.682 135.188 0.00696833 0.00538008 2024-03-26T03:09:04.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:04.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:05.042 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:05.087 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:05.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:05.187 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:05.227 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:05.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:09:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:09:05.261 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:09:05.261 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:09:05.261 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:09:05.261 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:09:05.326 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:09:05.326 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:09:05.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:27534 7.0 7:0c401e1e:::benchmark_data_smithi066_76917_object27533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:14.797704+0000", 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 470.52323638600001, 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.172682524, 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.327 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 27534 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797704+0000", 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797704+0000", 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797706+0000", 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797760+0000", 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3449000000000003e-05 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797769+0000", 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.149e-06 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797776+0000", 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.379e-06 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797885+0000", 2024-03-26T03:09:05.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010837500000000001 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.797960+0000", 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4871999999999997e-05 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.798055+0000", 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5304999999999996e-05 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799033+0000", 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00097775100000000001 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799202+0000", 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000169798 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.970321+0000", 2024-03-26T03:09:05.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.171118767 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.970357+0000", 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5686999999999999e-05 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.970387+0000", 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9873e-05 2024-03-26T03:09:05.333 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:27540 7.0 7:0b0e0d9b:::benchmark_data_smithi066_76917_object27539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:14.799714+0000", 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 470.52122624200001, 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17334993200000001, 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 27540 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799714+0000", 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799714+0000", 2024-03-26T03:09:05.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799716+0000", 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4020000000000001e-06 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799778+0000", 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2278000000000002e-05 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799788+0000", 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0219e-05 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799795+0000", 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4749999999999998e-06 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799847+0000", 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1823000000000002e-05 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.799974+0000", 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127353 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.800120+0000", 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000145852 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.800972+0000", 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085239699999999996 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.801046+0000", 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3663000000000003e-05 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.973003+0000", 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17195730000000001 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.973039+0000", 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5882e-05 2024-03-26T03:09:05.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:14.973064+0000", 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5287999999999999e-05 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28224 7.0 7:0aa9b815:::benchmark_data_smithi066_76917_object28223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.289009+0000", 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 470.03193180800002, 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.119753289, 2024-03-26T03:09:05.340 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.341 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.341 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.341 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.341 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:05.342 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28224 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289009+0000", 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289009+0000", 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289010+0000", 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.474e-06 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289095+0000", 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4634000000000003e-05 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289110+0000", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4963999999999999e-05 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289118+0000", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5850000000000002e-06 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289233+0000", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115214 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289350+0000", 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011717700000000001 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.289479+0000", 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012877200000000001 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.290665+0000", 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011862450000000001 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291515+0000", 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085024500000000002 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408656+0000", 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.117140335 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408715+0000", 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9061999999999998e-05 2024-03-26T03:09:05.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408762+0000", 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7581999999999999e-05 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28227 7.0 7:0aa0c260:::benchmark_data_smithi066_76917_object28226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.291818+0000", 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 470.02912278100001, 2024-03-26T03:09:05.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.117099724, 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28227 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291818+0000", 2024-03-26T03:09:05.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291818+0000", 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291819+0000", 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.457e-06 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291883+0000", 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3862000000000006e-05 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291892+0000", 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1179999999999998e-06 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291900+0000", 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6960000000000005e-06 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.291976+0000", 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5934999999999997e-05 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292055+0000", 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9031999999999998e-05 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292144+0000", 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8855000000000002e-05 2024-03-26T03:09:05.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.294043+0000", 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001898765 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295519+0000", 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014764419999999999 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408823+0000", 2024-03-26T03:09:05.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.113304047 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408877+0000", 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3791e-05 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408918+0000", 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0723999999999998e-05 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28236 7.0 7:0e3637c8:::benchmark_data_smithi066_76917_object28235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.292650+0000", 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 470.02829108499998, 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.116428117, 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28236 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.355 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292650+0000", 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292650+0000", 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292651+0000", 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2839999999999999e-06 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292743+0000", 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2077999999999995e-05 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292758+0000", 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5021000000000001e-05 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292766+0000", 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.405e-06 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292872+0000", 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105282 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.292988+0000", 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116025 2024-03-26T03:09:05.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.293099+0000", 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011086600000000001 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.294187+0000", 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00108884 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.295696+0000", 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015085540000000001 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.408973+0000", 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11327743699999999 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.409042+0000", 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8937999999999996e-05 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.409078+0000", 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5386999999999998e-05 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28771 7.0 7:019680ad:::benchmark_data_smithi066_76917_object28770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.642646+0000", 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 469.67829437199998, 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.143119834, 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.361 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28771 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642646+0000", 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642646+0000", 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642649+0000", 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2120000000000002e-06 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642706+0000", 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7716000000000001e-05 2024-03-26T03:09:05.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642722+0000", 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5274999999999999e-05 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642729+0000", 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3069999999999997e-06 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642798+0000", 2024-03-26T03:09:05.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9510000000000004e-05 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.642935+0000", 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013685600000000001 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643041+0000", 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010560000000000001 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645176+0000", 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021349870000000001 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645798+0000", 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00062242299999999999 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.785647+0000", 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13984848999999999 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.785710+0000", 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3287000000000006e-05 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.785766+0000", 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6171000000000001e-05 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:28776 7.0 7:06c4cb01:::benchmark_data_smithi066_76917_object28775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:15.643359+0000", 2024-03-26T03:09:05.367 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 469.67758186100002, 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.144621905, 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 28776 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643359+0000", 2024-03-26T03:09:05.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643359+0000", 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643360+0000", 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4869999999999999e-06 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643400+0000", 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9524000000000003e-05 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643415+0000", 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4773999999999999e-05 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643422+0000", 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7409999999999992e-06 2024-03-26T03:09:05.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.643921+0000", 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000498925 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644054+0000", 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013238799999999999 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.644202+0000", 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000148825 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645323+0000", 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001120081 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.645911+0000", 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00058858000000000001 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787861+0000", 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.141949363 2024-03-26T03:09:05.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787923+0000", 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2137000000000005e-05 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:15.787981+0000", 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.808e-05 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:29450 7.0 7:0cfae8a6:::benchmark_data_smithi066_76917_object29449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:16.097295+0000", 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 469.22364576199999, 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17839643499999999, 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 29450 2024-03-26T03:09:05.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097295+0000", 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097295+0000", 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097297+0000", 2024-03-26T03:09:05.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6700000000000001e-06 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097352+0000", 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4947000000000001e-05 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097362+0000", 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0404000000000001e-05 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097368+0000", 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5890000000000002e-06 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097412+0000", 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4308999999999999e-05 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097514+0000", 2024-03-26T03:09:05.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102359 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.097613+0000", 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8985000000000002e-05 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.098601+0000", 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00098787399999999991 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.098978+0000", 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037711100000000002 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.275600+0000", 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17662185899999999 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.275639+0000", 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9023000000000002e-05 2024-03-26T03:09:05.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:16.275691+0000", 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2305000000000002e-05 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30716 7.0 7:0202be7f:::benchmark_data_smithi066_76917_object30715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.300707+0000", 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 468.02023369199998, 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11267152599999999, 2024-03-26T03:09:05.380 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30716 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300707+0000", 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300707+0000", 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300709+0000", 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0499999999999999e-06 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300761+0000", 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2040000000000003e-05 2024-03-26T03:09:05.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300776+0000", 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4598e-05 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300783+0000", 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6009999999999997e-06 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300850+0000", 2024-03-26T03:09:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7045000000000006e-05 2024-03-26T03:09:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.300993+0000", 2024-03-26T03:09:05.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014279000000000001 2024-03-26T03:09:05.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.301135+0000", 2024-03-26T03:09:05.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014195700000000001 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302196+0000", 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010605759999999999 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.303765+0000", 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001568975 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413269+0000", 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.109504431 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413327+0000", 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8439000000000001e-05 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413379+0000", 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1023999999999998e-05 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:30717 7.0 7:0d0913d8:::benchmark_data_smithi066_76917_object30716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.302714+0000", 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 468.01822697599999, 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.110809716, 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 30717 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.388 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302714+0000", 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302714+0000", 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302718+0000", 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4900000000000002e-06 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302775+0000", 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6983e-05 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302785+0000", 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0084999999999999e-05 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302793+0000", 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5299999999999999e-06 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302840+0000", 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7524000000000001e-05 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.302960+0000", 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120081 2024-03-26T03:09:05.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.303069+0000", 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108811 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.303877+0000", 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080767800000000004 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.304274+0000", 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000396769 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413423+0000", 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.109148934 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413480+0000", 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7497000000000002e-05 2024-03-26T03:09:05.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.413523+0000", 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3334000000000002e-05 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:31281 7.0 7:06258b42:::benchmark_data_smithi066_76917_object31280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:17.738133+0000", 2024-03-26T03:09:05.394 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 467.58280778699998, 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14132778900000001, 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31281 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738133+0000", 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738133+0000", 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738135+0000", 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.091e-06 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738200+0000", 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4888000000000006e-05 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738212+0000", 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1803e-05 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738218+0000", 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2659999999999998e-06 2024-03-26T03:09:05.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738275+0000", 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7308000000000001e-05 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738414+0000", 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000138329 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.738551+0000", 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013751200000000001 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.739875+0000", 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001323942 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.740111+0000", 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: pgmap v436: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 9489/59142 objects degraded (16.044%) 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3227350699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1483298674' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:09:05.400 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: pgmap v436: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 9489/59142 objects degraded (16.044%) 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3227350699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:09:05.401 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1483298674' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:05 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: pgmap v436: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 122 MiB/s wr, 1.96k op/s; 9489/59142 objects degraded (16.044%) 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3227350699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:09:05.402 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1483298674' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:09:05.403 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:09:05.403 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:05 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000235658 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.879381+0000", 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13927038999999999 2024-03-26T03:09:05.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.879419+0000", 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7885000000000001e-05 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:17.879461+0000", 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1717000000000001e-05 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75536 7.0 7:000359a7:::benchmark_data_smithi066_76917_object75535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 434.99753520000002, 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.004732272, 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75536 2024-03-26T03:09:05.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323409+0000", 2024-03-26T03:09:05.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3119999999999998e-06 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323453+0000", 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4338999999999997e-05 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323474+0000", 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1109000000000001e-05 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323488+0000", 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3397000000000001e-05 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.363800+0000", 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040311903000000003 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.363816+0000", 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6526000000000001e-05 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313106+0000", 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94928997599999998 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313268+0000", 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016208999999999999 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313443+0000", 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174558 2024-03-26T03:09:05.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.316322+0000", 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028792119999999999 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.317555+0000", 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012334970000000001 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328022+0000", 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010466392999999999 2024-03-26T03:09:05.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328092+0000", 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9900000000000005e-05 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328138+0000", 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6060000000000003e-05 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75538 7.0 7:0739b178:::benchmark_data_smithi066_76917_object75537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 434.77345618999999, 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78758199900000003, 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75538 2024-03-26T03:09:05.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547488+0000", 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0359999999999998e-06 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547625+0000", 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013731600000000001 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547645+0000", 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9944000000000001e-05 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547656+0000", 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1610000000000001e-05 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547729+0000", 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2249999999999994e-05 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547738+0000", 2024-03-26T03:09:05.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5040000000000008e-06 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315069+0000", 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.76733058799999998 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315202+0000", 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132769 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315298+0000", 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6464000000000003e-05 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.316448+0000", 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011497969999999999 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.319454+0000", 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003006075 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.334937+0000", 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015482902 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.335019+0000", 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1930999999999999e-05 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.335066+0000", 2024-03-26T03:09:05.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7812999999999998e-05 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:111974 7.0 7:06bf99cb:::benchmark_data_smithi066_76917_object11791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 374.34081620500001, 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14525418500000001, 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111974 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980127+0000", 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0849999999999999e-06 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980134+0000", 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1810000000000004e-06 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980149+0000", 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5702999999999999e-05 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980159+0000", 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7969999999999995e-06 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980225+0000", 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5648000000000006e-05 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980350+0000", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125227 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980649+0000", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029928300000000001 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.982219+0000", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001569916 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.983019+0000", 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00079931200000000005 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125310+0000", 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14229117699999999 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125347+0000", 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7212000000000002e-05 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125379+0000", 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1643999999999997e-05 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112720 7.0 7:054d20dd:::benchmark_data_smithi066_76917_object12537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 372.80533589100003, 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13726976599999999, 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.426 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112720 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515607+0000", 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1119999999999999e-06 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515613+0000", 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1709999999999999e-06 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515629+0000", 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5914e-05 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515638+0000", 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6910000000000003e-06 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515696+0000", 2024-03-26T03:09:05.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7989999999999999e-05 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515813+0000", 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116997 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.516136+0000", 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032339800000000002 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.518072+0000", 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001936173 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.521492+0000", 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034199230000000001 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652808+0000", 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.131315819 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652843+0000", 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4805000000000003e-05 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652875+0000", 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1773000000000002e-05 2024-03-26T03:09:05.432 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:142785 7.0 7:05427235:::benchmark_data_smithi066_76917_object42602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 335.874690375, 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.121159576, 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.433 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 142785 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:09:05.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446253+0000", 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3099999999999999e-06 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446259+0000", 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2999999999999998e-06 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446275+0000", 2024-03-26T03:09:05.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6047999999999999e-05 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446284+0000", 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446354+0000", 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0160999999999995e-05 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446480+0000", 2024-03-26T03:09:05.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000126036 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446868+0000", 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038751299999999997 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.449292+0000", 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024239219999999998 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.452247+0000", 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029555039999999999 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567305+0000", 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.115057495 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567363+0000", 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8637000000000001e-05 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567410+0000", 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6641000000000003e-05 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:31077 8.8 8:1e9c3a95:::benchmark_data_smithi066_100612_object31076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:05:19.544642+0000", 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 225.77629846299999, 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10534709, 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 31077 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544642+0000", 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544642+0000", 2024-03-26T03:09:05.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544654+0000", 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1685000000000001e-05 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544685+0000", 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1161000000000002e-05 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544707+0000", 2024-03-26T03:09:05.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1625999999999999e-05 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544714+0000", 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4830000000000002e-06 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.544918+0000", 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000204012 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.545054+0000", 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000135747 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.545722+0000", 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066763200000000003 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.548064+0000", 2024-03-26T03:09:05.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023426850000000002 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.548255+0000", 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019107299999999999 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.649883+0000", 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.101627308 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.649935+0000", 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2469999999999997e-05 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:05:19.649989+0000", 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4208e-05 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.445 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:148920 8.0 8:0ef07389:::benchmark_data_smithi066_100612_object21529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:07:00.249718+0000", 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 125.071222918, 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11961746099999999, 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148920 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.446 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249718+0000", 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249718+0000", 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249719+0000", 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.531e-06 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249727+0000", 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9480000000000008e-06 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249737+0000", 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0019e-05 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249755+0000", 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7821000000000001e-05 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.305827+0000", 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056072151000000001 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.305959+0000", 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013224800000000001 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.306094+0000", 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013411900000000001 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369101+0000", 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.063007103999999994 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369151+0000", 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0281999999999997e-05 2024-03-26T03:09:05.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369242+0000", 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1469000000000001e-05 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369335+0000", 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2769000000000006e-05 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:148929 8.0 8:077252a2:::benchmark_data_smithi066_100612_object21538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-03-26T03:09:05.451 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:07:00.256786+0000", 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 125.064154657, 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.113071699, 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148929 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256786+0000", 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256786+0000", 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256788+0000", 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256795+0000", 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4099999999999996e-06 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.300719+0000", 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.043924419999999999 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.300732+0000", 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3134e-05 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.308563+0000", 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078310540000000005 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.308677+0000", 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011394499999999999 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.308790+0000", 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112948 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369107+0000", 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.060316579000000002 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369384+0000", 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000277681 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369648+0000", 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026355600000000002 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369858+0000", 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000209903 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:232862 8.0 8:0deccfb0:::benchmark_data_smithi066_100612_object105471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:08:02.696603+0000", 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 62.624337844000003, 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10821460400000001, 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:05.457 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 232862 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696603+0000", 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696603+0000", 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696604+0000", 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.654e-06 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:05.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696609+0000", 2024-03-26T03:09:05.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.746e-06 2024-03-26T03:09:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:05.460 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:05.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696620+0000", 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0749e-05 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696626+0000", 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.896e-06 2024-03-26T03:09:05.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696673+0000", 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6931999999999997e-05 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.696773+0000", 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100453 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:09:05.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.697122+0000", 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034874500000000002 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.793632+0000", 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.096510167999999993 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.804634+0000", 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011001533000000001 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.804723+0000", 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8858999999999997e-05 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:08:02.804817+0000", 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4869000000000003e-05 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:05.464 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:09:05.561 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:05.604 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:05.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:05.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:05.748 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:05.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:05.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:05.886 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:05.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:05.922 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 11 16 31974 31958 181.562 140.375 0.00708258 0.00550215 2024-03-26T03:09:05.988 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.030 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:06.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:06.131 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:06.171 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:09:06.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:09:06.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:06.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:09:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.320 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:06.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:06.420 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:06.462 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:06.488 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:09:06.488 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1483298674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:09:06.488 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[29666]: osdmap e231: 8 total, 7 up, 8 in 2024-03-26T03:09:06.488 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:09:06.488 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:09:06.488 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:09:06.489 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1483298674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:09:06.489 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[37920]: osdmap e231: 8 total, 7 up, 8 in 2024-03-26T03:09:06.489 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:09:06.489 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:06 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:09:06.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:06.603 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:06.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:06.727 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:06 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:09:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:06 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1483298674' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:09:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:06 smithi192 ceph-mon[33110]: osdmap e231: 8 total, 7 up, 8 in 2024-03-26T03:09:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:06 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:09:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:06 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:09:06.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:09:06.772 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:09:06.772 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-26T03:09:06.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:06.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:06.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:06.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 12 15 34158 34143 177.808 136.562 0.00892654 0.00561961 2024-03-26T03:09:06.984 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.036 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:07.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:07.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:07.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:07.276 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:07.319 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:07.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:07.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:07.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:07.585 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:07.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[37920]: pgmap v438: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 223 MiB/s wr, 3.59k op/s; 14287/87648 objects degraded (16.300%) 2024-03-26T03:09:07.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:09:07.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:09:07.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:09:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:09:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[29666]: pgmap v438: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 223 MiB/s wr, 3.59k op/s; 14287/87648 objects degraded (16.300%) 2024-03-26T03:09:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:09:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:09:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:09:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:07 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:09:07.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:07.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:07 smithi192 ceph-mon[33110]: pgmap v438: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 223 MiB/s wr, 3.59k op/s; 14287/87648 objects degraded (16.300%) 2024-03-26T03:09:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:07 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:09:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:07 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:09:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:07 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:09:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:07 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:09:07.823 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:07.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:07.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 13 16 35866 35850 172.336 106.688 0.00204906 0.0057864 2024-03-26T03:09:07.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:07.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:07.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:08.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:08.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:08.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:08.239 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:09:08.266 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:08.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:08.397 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:08.499 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:08.531 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:08.633 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:08.674 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:08.776 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:08.803 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:08.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:08.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 14 15 37608 37593 167.808 108.938 0.0168362 0.00595429 2024-03-26T03:09:08.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:08.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:09.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:09.061 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:09.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:09.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:09.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:09.251 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:09.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:09.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:09.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:09.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:09:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: pgmap v439: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 133 MiB/s wr, 2.15k op/s; 14287/87648 objects degraded (16.300%) 2024-03-26T03:09:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14287/87648 objects degraded (16.300%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:09.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:09.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:09.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:09:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: pgmap v439: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 133 MiB/s wr, 2.15k op/s; 14287/87648 objects degraded (16.300%) 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14287/87648 objects degraded (16.300%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: pgmap v439: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 1.8 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 14 KiB/s rd, 133 MiB/s wr, 2.15k op/s; 14287/87648 objects degraded (16.300%) 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:09:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:09:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:09:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:09:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:09:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14287/87648 objects degraded (16.300%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:09.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:09.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:09.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:09.843 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:09.873 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:09.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:09.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 15 16 39198 39182 163.241 99.3125 0.0147899 0.00612144 2024-03-26T03:09:09.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:09.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:09:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:09:10.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:10.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:10.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:10.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:10.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:10.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:10.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:09:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:09:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:09:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:09:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:09:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:09:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:09:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:10 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:09:10.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:10 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:09:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:10 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:09:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:10 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:09:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:10 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:09:10.791 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:10.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:10.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 16 16 40971 40955 159.963 110.812 0.00522336 0.00623072 2024-03-26T03:09:10.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:10.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:10.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:10.973 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:09:11.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:11.112 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:11.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:11.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:11.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:11.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:11.480 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:11 smithi192 ceph-mon[33110]: pgmap v440: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 54 KiB/s rd, 158 MiB/s wr, 2.62k op/s; 18668/109899 objects degraded (16.987%) 2024-03-26T03:09:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:11 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:09:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:11 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:09:11.521 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[29666]: pgmap v440: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 54 KiB/s rd, 158 MiB/s wr, 2.62k op/s; 18668/109899 objects degraded (16.987%) 2024-03-26T03:09:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:09:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:09:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[37920]: pgmap v440: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 54 KiB/s rd, 158 MiB/s wr, 2.62k op/s; 18668/109899 objects degraded (16.987%) 2024-03-26T03:09:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:09:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:11 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:09:11.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:11.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:11.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:11.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:11.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:11.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 17 16 42778 42762 157.196 112.938 0.0092906 0.00635649 2024-03-26T03:09:11.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:11.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:12.040 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:12.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:12.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:12.214 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:12.316 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:12.353 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:12.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:12 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:09:12.449 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:12 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:09:12.455 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:12.491 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:12.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:12.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:44034 9.b e231/225 9:de9b2e21:::benchmark_data_smithi066_125102_object44033:head v 231'2706, mlcod=231'2704)", 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:12.654033+0000", 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0069729529999999996, 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070036849999999999, 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:12.663 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654033+0000", 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654033+0000", 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654035+0000", 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.632e-06 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654069+0000", 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4539000000000003e-05 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654082+0000", 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2653999999999999e-05 2024-03-26T03:09:12.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654090+0000", 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8140000000000001e-06 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654160+0000", 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0582999999999998e-05 2024-03-26T03:09:12.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.654183+0000", 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.234e-05 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:44036 9.b e231/225 9:d507e978:::benchmark_data_smithi066_125102_object44035:head v 231'2707, mlcod=231'2704)", 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:12.655206+0000", 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0058005610000000001, 2024-03-26T03:09:12.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058980760000000004, 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.655206+0000", 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.655206+0000", 2024-03-26T03:09:12.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.655207+0000", 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3400000000000001e-06 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.655376+0000", 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000168787 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.655382+0000", 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6980000000000001e-06 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.655388+0000", 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5520000000000004e-06 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.656007+0000", 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00061898699999999999 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.656029+0000", 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1698000000000001e-05 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:12.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:44045 9.1 9:82fb8e0c:::benchmark_data_smithi066_125102_object44044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:12.658787+0000", 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0022193960000000002, 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002383287, 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:12.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 44045 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.658787+0000", 2024-03-26T03:09:12.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.658787+0000", 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.658797+0000", 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.041e-05 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.658861+0000", 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3969999999999999e-05 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.658875+0000", 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.398e-05 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.658884+0000", 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2879999999999998e-06 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659035+0000", 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015035 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659175+0000", 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000140693 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-03-26T03:09:12.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659319+0000", 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000143172 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:44048 9.b e231/225)", 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:12.659736+0000", 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0012698130000000001, 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001502296, 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:12.678 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659736+0000", 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659736+0000", 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:12.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659738+0000", 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.392e-06 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659786+0000", 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8708999999999997e-05 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:12.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659797+0000", 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0210000000000001e-05 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:12.659804+0000", 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0650000000000001e-06 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:09:12.681 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 4 2024-03-26T03:09:12.682 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:09:12.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:12.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:12.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:12 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:09:12.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:12 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:09:12.782 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:12 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:09:12.782 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:12 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:09:12.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:12.920 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:12.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:12.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 18 16 44513 44497 154.487 108.438 0.00487358 0.0064677 2024-03-26T03:09:13.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:13.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:13.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:13.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:13.308 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:13.352 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:13.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:13.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:13.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:13.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:09:13.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:13.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:13 smithi192 ceph-mon[33110]: pgmap v441: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 50 KiB/s rd, 160 MiB/s wr, 2.64k op/s; 19241/116772 objects degraded (16.477%) 2024-03-26T03:09:13.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:13.784 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:13 smithi066 ceph-mon[29666]: pgmap v441: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 50 KiB/s rd, 160 MiB/s wr, 2.64k op/s; 19241/116772 objects degraded (16.477%) 2024-03-26T03:09:13.784 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:13 smithi066 ceph-mon[37920]: pgmap v441: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 50 KiB/s rd, 160 MiB/s wr, 2.64k op/s; 19241/116772 objects degraded (16.477%) 2024-03-26T03:09:13.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:13.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:13.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 19 16 46410 46394 152.596 118.562 0.00441189 0.00654744 2024-03-26T03:09:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:13.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:13.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:13.994 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:14.057 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:14.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:09:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:09:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:09:14.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:14.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:14.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:14.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:14.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:14 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:09:14.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:14 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:09:14.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:14 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:09:14.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:14 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:09:14.460 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 19241/116772 objects degraded (16.477%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:14.474 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:14.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:14.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:14.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:14.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:14.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 19241/116772 objects degraded (16.477%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:09:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 19241/116772 objects degraded (16.477%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:14.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:14.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:09:14.923969+0000 min lat: 0.00130673 max lat: 0.113131 avg lat: 0.00663946 2024-03-26T03:09:14.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:09:14.924 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 20 16 48145 48129 150.387 108.438 0.00477192 0.00663946 2024-03-26T03:09:14.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:14.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:15.027 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:15.067 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:15.169 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.213 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:15.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:15.266 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:09:15.307 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:09:15.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:15.347 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:09:15.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:15.386 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:09:15.426 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:09:15.456 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:15.465 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:09:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:09:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:09:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: pgmap v442: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 44 KiB/s rd, 120 MiB/s wr, 1.99k op/s; 19241/116772 objects degraded (16.477%) 2024-03-26T03:09:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: 5.1d deep-scrub starts 2024-03-26T03:09:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: 5.1d deep-scrub ok 2024-03-26T03:09:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:09:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:09:15.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:15 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2748525747' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:15.500 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:09:15.502 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:09:15.602 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:15.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:09:15.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: pgmap v442: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 44 KiB/s rd, 120 MiB/s wr, 1.99k op/s; 19241/116772 objects degraded (16.477%) 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: 5.1d deep-scrub starts 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: 5.1d deep-scrub ok 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2748525747' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: pgmap v442: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 44 KiB/s rd, 120 MiB/s wr, 1.99k op/s; 19241/116772 objects degraded (16.477%) 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: 5.1d deep-scrub starts 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: 5.1d deep-scrub ok 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:09:15.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:15 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2748525747' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:15.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:15.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:15.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:15.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:no valid command found; 10 closest matches: 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:0 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:1 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:2 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:abort 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:assert 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:bench [] [] [] [] 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs debug_inject_read_zeros 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs files list 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:bluefs stats 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:bluestore allocator dump block 2024-03-26T03:09:15.885 INFO:teuthology.orchestra.run.smithi066.stderr:Error EINVAL: invalid command 2024-03-26T03:09:15.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:15.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:15.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 21 16 50018 50002 148.8 117.062 0.00484169 0.00671493 2024-03-26T03:09:16.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:16.063 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:16.155 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:09:16.155 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:09:16.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:16.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:16.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:16.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:16.447 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:16 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:09:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:16 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:09:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.489 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:16.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:16.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:16 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:09:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:16 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:09:16.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:16 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:09:16.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:16 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:09:16.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:16.729 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:09:16.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:09:16.780 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:16.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:16.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:09:16.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:16.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:16.923 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 22 16 51865 51849 147.283 115.438 0.00911108 0.00678466 2024-03-26T03:09:17.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:17.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:17.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:17.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:09:17.242 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-26T03:09:17.259 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:17.316 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:17.319 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:17.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:17.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:17.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:17.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:17.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:17.703 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:17 smithi066 ceph-mon[29666]: pgmap v443: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 36 KiB/s rd, 125 MiB/s wr, 2.06k op/s; 23720/143487 objects degraded (16.531%) 2024-03-26T03:09:17.703 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:17 smithi066 ceph-mon[37920]: pgmap v443: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 36 KiB/s rd, 125 MiB/s wr, 2.06k op/s; 23720/143487 objects degraded (16.531%) 2024-03-26T03:09:17.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:17 smithi192 ceph-mon[33110]: pgmap v443: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 36 KiB/s rd, 125 MiB/s wr, 2.06k op/s; 23720/143487 objects degraded (16.531%) 2024-03-26T03:09:17.750 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.790 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:17.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:17.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:17.924 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 23 16 53607 53591 145.613 108.875 0.0126752 0.00686197 2024-03-26T03:09:17.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:17.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:17.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:18.036 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.089 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:18.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:18.190 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:18.237 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:18.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:18.380 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:18.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:18.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:18.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:18.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:09:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:09:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:09:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:09:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:09:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:09:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:18 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:09:18.767 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:09:18.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:09:18.812 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:09:18.813 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:09:18.813 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:09:18.813 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:09:18.813 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:18 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:09:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:18.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:18.925 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 24 16 55206 55190 143.707 99.9375 0.0083895 0.00695348 2024-03-26T03:09:18.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:18.980 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:18.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:19.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:19.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:19.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:19.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:19.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:19.277 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:19.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:19.340 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:09:19.340 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:09:19.378 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:19.434 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: pgmap v444: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 28 KiB/s rd, 97 MiB/s wr, 1.60k op/s; 23720/143487 objects degraded (16.531%) 2024-03-26T03:09:19.440 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:19.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:19.460 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:19.541 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:19.601 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:19.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:19.683 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: pgmap v444: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 28 KiB/s rd, 97 MiB/s wr, 1.60k op/s; 23720/143487 objects degraded (16.531%) 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: 2.16 deep-scrub starts 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: 2.16 deep-scrub ok 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:09:19.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:09:19.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:19.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 23720/143487 objects degraded (16.531%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:19.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:19.750 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:19.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: 2.16 deep-scrub starts 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: 2.16 deep-scrub ok 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:09:19.770 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 23720/143487 objects degraded (16.531%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: pgmap v444: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 28 KiB/s rd, 97 MiB/s wr, 1.60k op/s; 23720/143487 objects degraded (16.531%) 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: 2.16 deep-scrub starts 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: 2.16 deep-scrub ok 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:09:19.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:09:19.772 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:19.772 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 23720/143487 objects degraded (16.531%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:19.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:19.895 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:19.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:19.924 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 25 16 56703 56687 141.701 93.5625 0.00466574 0.00705242 2024-03-26T03:09:19.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:20.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:20.134 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:20.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:09:20.176 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:20.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.321 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:20.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:20.422 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:20.456 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:09:20.467 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:20.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:20.596 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:20.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:20.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:20 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:09:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:20 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:09:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:20 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4165186144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:09:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:20 smithi192 ceph-mon[33110]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:09:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:20 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:09:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:20 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:09:20.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4165186144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[29666]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4165186144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:09:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[37920]: from='client.25255 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:09:20.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:09:20.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:20 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:09:20.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:20.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:20.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:20.924 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 26 16 58340 58324 140.185 102.312 0.00805052 0.00712828 2024-03-26T03:09:21.023 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.070 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:21.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:21.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:21.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:21.312 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:21.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:21.457 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:21.496 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:21.598 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:21.644 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: pgmap v445: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 28 KiB/s rd, 131 MiB/s wr, 2.14k op/s; 27511/163134 objects degraded (16.864%) 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[29666]: osdmap e232: 8 total, 7 up, 8 in 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: pgmap v445: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 28 KiB/s rd, 131 MiB/s wr, 2.14k op/s; 27511/163134 objects degraded (16.864%) 2024-03-26T03:09:21.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:09:21.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:09:21.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:09:21.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:21.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:09:21.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:21 smithi066 ceph-mon[37920]: osdmap e232: 8 total, 7 up, 8 in 2024-03-26T03:09:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: pgmap v445: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 28 KiB/s rd, 131 MiB/s wr, 2.14k op/s; 27511/163134 objects degraded (16.864%) 2024-03-26T03:09:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:09:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:09:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:09:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: from='client.25255 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:09:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:21 smithi192 ceph-mon[33110]: osdmap e232: 8 total, 7 up, 8 in 2024-03-26T03:09:21.746 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:21.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:21.820 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:09:21.857 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:09:21.885 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:09:21.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:21.924 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 27 16 60073 60057 139.005 108.312 0.00780932 0.00718913 2024-03-26T03:09:21.929 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:09:21.937 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:21.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:21.969 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:09:22.006 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:09:22.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:22.049 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:09:22.049 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:09:22.049 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:09:22.049 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:09:22.049 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:09:22.050 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:09:22.050 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:09:22.050 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:09:22.050 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:09:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.078 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:22.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:22.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:22.217 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:22.243 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:09:22.243 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:09:22.244 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-26T03:09:22.244 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:09:22.244 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:09:22.244 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-26T03:09:22.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:22.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:22.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:22.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:22.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:22.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:22 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3562547729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:22.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:22 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3562547729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:22.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:22.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:22 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3562547729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:22.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:09:22.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:09:22.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:22.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:22.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:22.925 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 28 16 61576 61560 137.395 93.9375 0.0159038 0.00727315 2024-03-26T03:09:22.981 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:23.008 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:23.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:23.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:23.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:23.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:23.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:23.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61239 9.e 9:7dd88abe:::benchmark_data_smithi066_125102_object61238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.673172+0000", 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.004527423, 2024-03-26T03:09:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045647639999999998, 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61239 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673172+0000", 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673172+0000", 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673174+0000", 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8700000000000001e-06 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673217+0000", 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2892999999999997e-05 2024-03-26T03:09:23.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673231+0000", 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4531e-05 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673239+0000", 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8029999999999997e-06 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673299+0000", 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9735000000000003e-05 2024-03-26T03:09:23.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673977+0000", 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067779099999999996 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.674137+0000", 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016066299999999999 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.430 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61240 9.e 9:774bd9db:::benchmark_data_smithi066_125102_object61239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.673254+0000", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0044457469999999999, 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045766670000000004, 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61240 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673254+0000", 2024-03-26T03:09:23.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673254+0000", 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673255+0000", 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1699999999999997e-07 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673289+0000", 2024-03-26T03:09:23.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3902000000000002e-05 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673294+0000", 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9190000000000004e-06 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673300+0000", 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6080000000000003e-06 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676166+0000", 2024-03-26T03:09:23.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028658139999999999 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676309+0000", 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014343099999999999 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676459+0000", 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014990000000000001 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61241 9.3 9:ca4950cd:::benchmark_data_smithi066_125102_object61240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.673308+0000", 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0043912509999999997, 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046205760000000004, 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61241 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673308+0000", 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673308+0000", 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673309+0000", 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9100000000000002e-07 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673338+0000", 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8666999999999998e-05 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673345+0000", 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9149999999999999e-06 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673350+0000", 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1959999999999999e-06 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673445+0000", 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5397999999999997e-05 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673584+0000", 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013888099999999999 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673761+0000", 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000177228 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677340+0000", 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035782819999999999 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61244 9.d 9:b2f1a716:::benchmark_data_smithi066_125102_object61243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.673655+0000", 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0040449140000000001, 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043774290000000004, 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:23.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61244 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673655+0000", 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673655+0000", 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673656+0000", 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4360000000000001e-06 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673712+0000", 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5714999999999998e-05 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673739+0000", 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7406000000000002e-05 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673747+0000", 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9079999999999995e-06 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673797+0000", 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9997999999999998e-05 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.673927+0000", 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013007900000000001 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.674087+0000", 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000159631 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61208 9.7 9:e3c554db:::benchmark_data_smithi066_125102_object61207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.443 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.650721+0000", 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.026978966, 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027398807000000001, 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61208 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.650721+0000", 2024-03-26T03:09:23.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.650721+0000", 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.650722+0000", 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2470000000000001e-06 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.650750+0000", 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7919000000000002e-05 2024-03-26T03:09:23.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.650759+0000", 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7030000000000001e-06 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.650767+0000", 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0740000000000004e-06 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.656356+0000", 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055890380000000002 2024-03-26T03:09:23.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.656480+0000", 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012420199999999999 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5", 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.656606+0000", 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012660699999999999 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.659151+0000", 2024-03-26T03:09:23.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025448760000000002 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61247 9.d 9.71459c6d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.675021+0000", 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0026789829999999998, 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031769569999999998, 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.448 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61247 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.675021+0000", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.675021+0000", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.675022+0000", 2024-03-26T03:09:23.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5489999999999999e-06 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.675053+0000", 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0383e-05 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.675063+0000", 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0698e-05 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.675071+0000", 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8050000000000007e-06 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:61248 9.5 e232/225 9:a60e406a:::benchmark_data_smithi066_125102_object61247:head v 232'3775, mlcod=232'3773)", 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.676452+0000", 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.001247923, 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001802145, 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.451 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676452+0000", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676452+0000", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676453+0000", 2024-03-26T03:09:23.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4869999999999999e-06 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676553+0000", 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100318 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676563+0000", 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5380000000000008e-06 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676570+0000", 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6290000000000004e-06 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676660+0000", 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0370000000000004e-05 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.676678+0000", 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7728e-05 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15360.0:61208 9.7 e232/225)", 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.677359+0000", 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00034017500000000002, 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00095206800000000003, 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677359+0000", 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677359+0000", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677360+0000", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0529999999999999e-06 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677365+0000", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7199999999999997e-06 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677373+0000", 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0779999999999996e-06 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677379+0000", 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9939999999999997e-06 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:61249 9.a e232/225 9:554d7006:::benchmark_data_smithi066_125102_object61248:head v 232'3791, mlcod=232'3789)", 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.677394+0000", 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00030536400000000003, 2024-03-26T03:09:23.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096486300000000005, 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677394+0000", 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677394+0000", 2024-03-26T03:09:23.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677395+0000", 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1300000000000001e-07 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677427+0000", 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2576000000000001e-05 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677433+0000", 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9379999999999997e-06 2024-03-26T03:09:23.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677438+0000", 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9139999999999999e-06 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677547+0000", 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108671 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677566+0000", 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9746999999999999e-05 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.460 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15360.0:61224 9.7 e232/225)", 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.677451+0000", 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.000248402, 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096275699999999996, 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677451+0000", 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677451+0000", 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677452+0000", 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3000000000000001e-07 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677454+0000", 2024-03-26T03:09:23.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.199e-06 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677458+0000", 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3150000000000002e-06 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.677462+0000", 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1869999999999999e-06 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:61224 9.7 9:eab03c41:::benchmark_data_smithi066_125102_object61223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:22.661001+0000", 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016698283000000001, 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017466063, 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61224 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.661001+0000", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.661001+0000", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.661002+0000", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.037e-06 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.661040+0000", 2024-03-26T03:09:23.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7459000000000002e-05 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.661048+0000", 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0479999999999999e-06 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.661056+0000", 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9789999999999993e-06 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.662422+0000", 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013666819999999999 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.662531+0000", 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010867800000000001 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5", 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.662661+0000", 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000129436 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:09:23.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:22.668286+0000", 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056258339999999997 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 11 2024-03-26T03:09:23.468 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:09:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:23.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:23.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:23.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:23.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:23 smithi192 ceph-mon[33110]: pgmap v447: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s; 28014/169257 objects degraded (16.551%) 2024-03-26T03:09:23.752 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:23.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:23 smithi066 ceph-mon[29666]: pgmap v447: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s; 28014/169257 objects degraded (16.551%) 2024-03-26T03:09:23.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:23 smithi066 ceph-mon[37920]: pgmap v447: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s; 28014/169257 objects degraded (16.551%) 2024-03-26T03:09:23.854 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:23.895 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:23.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:23.925 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 29 16 63103 63087 135.948 95.4375 0.00937795 0.00735102 2024-03-26T03:09:23.997 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.038 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:24.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:24.099 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:09:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:09:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:09:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:09:24.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:24.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:24.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:24.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:24.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:09:24.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:24.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:24.548 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:24.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:24.688 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:24.725 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 28014/169257 objects degraded (16.551%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:24 smithi192 ceph-mon[33110]: pgmap v448: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s; 28014/169257 objects degraded (16.551%) 2024-03-26T03:09:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:24.766 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:24.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 28014/169257 objects degraded (16.551%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:24.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[37920]: pgmap v448: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s; 28014/169257 objects degraded (16.551%) 2024-03-26T03:09:24.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:24.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:24.803 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 28014/169257 objects degraded (16.551%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:24.803 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[29666]: pgmap v448: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.75k op/s; 28014/169257 objects degraded (16.551%) 2024-03-26T03:09:24.803 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:09:24.803 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:24.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:24.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:09:24.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:24.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:24.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:24.955 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 30 16 64589 64573 134.503 92.875 0.0101041 0.00742986 2024-03-26T03:09:25.056 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:25.096 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:25.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:25.200 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:25.236 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:25.338 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:25.379 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:25.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:25.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:25.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:25.661 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:25.763 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:25.806 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:25.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:25.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 31 16 66177 66161 133.366 99.25 0.0148962 0.00749161 2024-03-26T03:09:25.943 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:25.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:26.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:26.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:26.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:26.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:26.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:26.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:26.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:26.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:26.519 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:09:26.519 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "" 2024-03-26T03:09:26.520 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:09:26.597 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:26.632 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:26.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:26.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:26.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:26.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:26.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:26.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 32 16 67623 67607 132.022 90.375 0.00935586 0.00757009 2024-03-26T03:09:27.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:27.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:27.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:27.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:27.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:27.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:27.419 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:27.448 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:27 smithi192 ceph-mon[33110]: pgmap v449: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.66k op/s; 31993/193137 objects degraded (16.565%) 2024-03-26T03:09:27.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:27.585 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:27 smithi066 ceph-mon[29666]: pgmap v449: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.66k op/s; 31993/193137 objects degraded (16.565%) 2024-03-26T03:09:27.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:27 smithi066 ceph-mon[37920]: pgmap v449: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.66k op/s; 31993/193137 objects degraded (16.565%) 2024-03-26T03:09:27.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:27.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:27.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:27.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:27.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 33 15 68953 68938 130.542 83.1875 0.00448396 0.00765537 2024-03-26T03:09:27.951 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:27.990 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:27.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:28.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:28.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:28.228 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:28.267 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:28.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:28.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:28.456 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:09:28.457 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:09:28.457 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:09:28.457 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:09:28.457 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:09:28.505 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:28.544 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:28.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:28.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:28.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:28.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:28.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:28.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 34 15 70398 70383 129.359 90.3125 0.00589806 0.00772575 2024-03-26T03:09:28.958 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:28.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:29.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:29.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:29.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:29.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:29.247 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:29.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:29.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:29.384 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:29 smithi066 ceph-mon[29666]: pgmap v450: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.66k op/s; 31993/193137 objects degraded (16.565%) 2024-03-26T03:09:29.384 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 31993/193137 objects degraded (16.565%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:29.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:29 smithi066 ceph-mon[37920]: pgmap v450: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.66k op/s; 31993/193137 objects degraded (16.565%) 2024-03-26T03:09:29.384 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 31993/193137 objects degraded (16.565%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:29.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:29 smithi192 ceph-mon[33110]: pgmap v450: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.66k op/s; 31993/193137 objects degraded (16.565%) 2024-03-26T03:09:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 31993/193137 objects degraded (16.565%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.530 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:29.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:29.631 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:29.668 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:29.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.812 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:29.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:29.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:29.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 35 16 71914 71898 128.368 94.6875 0.0106803 0.00778569 2024-03-26T03:09:29.937 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:09:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:09:29.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:29.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:29.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:09:30.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:30.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:30.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:30.247 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:09:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:30.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:30.360 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:30.390 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:30.412 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:09:30.492 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.539 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:30.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:30.556 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:30 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2020821454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:09:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:30 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2020821454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:09:30.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:30 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2020821454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:09:30.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:30.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:30.782 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:30.820 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:30.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:30.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 36 16 73286 73270 127.184 85.75 0.0075778 0.00785779 2024-03-26T03:09:30.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:30.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:31.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:31.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:31.208 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.255 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:31.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:31.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:31.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[29666]: pgmap v451: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.54k op/s; 35181/209316 objects degraded (16.808%) 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2020821454' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[29666]: osdmap e233: 8 total, 7 up, 8 in 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[37920]: pgmap v451: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.54k op/s; 35181/209316 objects degraded (16.808%) 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2020821454' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:09:31.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:31 smithi066 ceph-mon[37920]: osdmap e233: 8 total, 7 up, 8 in 2024-03-26T03:09:31.401 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:31.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:31.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:31.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:31.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:31 smithi192 ceph-mon[33110]: pgmap v451: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.54k op/s; 35181/209316 objects degraded (16.808%) 2024-03-26T03:09:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:31 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:31 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2020821454' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:09:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:31 smithi192 ceph-mon[33110]: osdmap e233: 8 total, 7 up, 8 in 2024-03-26T03:09:31.775 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:31.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:31.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:31.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 37 16 74743 74727 126.208 91.0625 0.0105992 0.00791878 2024-03-26T03:09:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:31.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:31.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:32.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:32.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:32.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:32.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:32.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:32.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:32.420 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:32.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:32.559 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:32.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:32.703 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:32.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:32.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:32.927 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 38 16 76151 76135 125.202 88 0.0167206 0.00798235 2024-03-26T03:09:32.944 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:32.985 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:32.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:32.986 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:09:33.017 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:09:33.054 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:09:33.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:33.094 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:09:33.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:33.135 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:09:33.168 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:09:33.212 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:09:33.212 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:09:33.213 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:09:33.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:33.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:33.359 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:33 smithi192 ceph-mon[33110]: pgmap v453: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s; 35632/214839 objects degraded (16.585%) 2024-03-26T03:09:33.359 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:33 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2546706485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:33.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:33.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:33.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:33.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:33 smithi066 ceph-mon[29666]: pgmap v453: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s; 35632/214839 objects degraded (16.585%) 2024-03-26T03:09:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:33 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2546706485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:33 smithi066 ceph-mon[37920]: pgmap v453: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s; 35632/214839 objects degraded (16.585%) 2024-03-26T03:09:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:33 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2546706485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:33.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:33.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:33.771 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:09:33.771 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:09:33.771 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:09:33.771 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:09:33.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:33.821 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:09:33.821 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:09:33.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:33.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 39 16 77499 77483 124.152 84.25 0.0127797 0.00804989 2024-03-26T03:09:33.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:33.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:33.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:34.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:09:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:09:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:09:34.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:34.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:34.238 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:34.338 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:09:34.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:34.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:34.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:34.536 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:34.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 35632/214839 objects degraded (16.585%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:34.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 35632/214839 objects degraded (16.585%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:34.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:34.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:34.731 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 35632/214839 objects degraded (16.585%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:34.774 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:34.776 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:09:34.820 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:34.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:34.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:09:34.928055+0000 min lat: 0.00130673 max lat: 0.137828 avg lat: 0.00811984 2024-03-26T03:09:34.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:09:34.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 40 16 78799 78783 123.079 81.25 0.00421885 0.00811984 2024-03-26T03:09:34.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:34.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:35.060 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:35.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:35.200 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:35.241 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:35.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:35.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:35.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:35.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:35.541 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:35 smithi066 ceph-mon[29666]: pgmap v454: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s; 35632/214839 objects degraded (16.585%) 2024-03-26T03:09:35.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:35 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:09:35.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:35 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:09:35.542 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:35 smithi066 ceph-mon[37920]: pgmap v454: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s; 35632/214839 objects degraded (16.585%) 2024-03-26T03:09:35.542 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:35 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:09:35.542 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:35 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:09:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:09:35.594 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:09:35.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:35.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:35.670 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:35.678 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:35 smithi192 ceph-mon[33110]: pgmap v454: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s; 35632/214839 objects degraded (16.585%) 2024-03-26T03:09:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:35 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:09:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:35 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:09:35.751 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.804 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:35.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:35.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:35.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 41 16 80198 80182 122.21 87.4375 0.00987196 0.00817805 2024-03-26T03:09:35.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:35.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:36.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:36.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:36.187 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.238 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:36.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:36.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:36.361 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:36 smithi066 ceph-mon[29666]: osdmap e234: 8 total, 7 up, 8 in 2024-03-26T03:09:36.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:36 smithi066 ceph-mon[37920]: osdmap e234: 8 total, 7 up, 8 in 2024-03-26T03:09:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.387 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:36.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:36.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.533 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:36.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:36.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:36.678 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:36 smithi192 ceph-mon[33110]: osdmap e234: 8 total, 7 up, 8 in 2024-03-26T03:09:36.780 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:36.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:36.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 42 16 81662 81646 121.479 91.5 0.0148939 0.00822701 2024-03-26T03:09:36.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:36.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:36.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:37.066 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:37.101 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:37.203 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.244 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:37.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:37.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:37.389 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:37.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:37.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:37.594 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:09:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:37 smithi066 ceph-mon[29666]: pgmap v456: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 39128/235818 objects degraded (16.592%) 2024-03-26T03:09:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:37 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:09:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:37 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:09:37.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:37 smithi066 ceph-mon[37920]: pgmap v456: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 39128/235818 objects degraded (16.592%) 2024-03-26T03:09:37.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:37 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:09:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:37 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:09:37.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:37.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:37 smithi192 ceph-mon[33110]: pgmap v456: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 39128/235818 objects degraded (16.592%) 2024-03-26T03:09:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:37 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:09:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:37 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:09:37.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:37.794 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:37.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:37.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:37.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 43 16 83120 83104 120.772 91.125 0.0099255 0.00827517 2024-03-26T03:09:37.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:37.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:38.048 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:38.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:38.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:38.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:38.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:38.382 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:38.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:38.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:38 smithi066 ceph-mon[29666]: osdmap e235: 8 total, 7 up, 8 in 2024-03-26T03:09:38.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:38 smithi066 ceph-mon[37920]: osdmap e235: 8 total, 7 up, 8 in 2024-03-26T03:09:38.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:38.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:38 smithi192 ceph-mon[33110]: osdmap e235: 8 total, 7 up, 8 in 2024-03-26T03:09:38.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:38.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:38.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:38.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 44 16 84574 84558 120.092 90.875 0.00892687 0.00832145 2024-03-26T03:09:38.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:38.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:39.033 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.071 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:39.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:39.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:39.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:39.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:39.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:39.445 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:39.484 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:39.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:39.604 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:09:39.604 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[29666]: pgmap v458: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 70 MiB/s wr, 1.11k op/s; 39128/235818 objects degraded (16.592%) 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[29666]: osdmap e236: 8 total, 7 up, 8 in 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 39128/235818 objects degraded (16.592%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[37920]: pgmap v458: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 70 MiB/s wr, 1.11k op/s; 39128/235818 objects degraded (16.592%) 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[37920]: osdmap e236: 8 total, 7 up, 8 in 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 39128/235818 objects degraded (16.592%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:39.605 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:39.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:39.725 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:39.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:09:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:09:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:09:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-mon[33110]: pgmap v458: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 70 MiB/s wr, 1.11k op/s; 39128/235818 objects degraded (16.592%) 2024-03-26T03:09:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-mon[33110]: osdmap e236: 8 total, 7 up, 8 in 2024-03-26T03:09:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 39128/235818 objects degraded (16.592%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:39.764 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:39.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:39.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:39.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:39.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 45 16 86013 85997 119.422 89.9375 0.00712186 0.00836917 2024-03-26T03:09:39.971 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:09:39.971 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:09:40.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.048 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:40.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:40.149 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.189 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:40.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:40.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:40.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:40.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:40.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:40.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:40.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:40.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:40.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:40.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:40.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:40.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:40.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 46 15 87448 87433 118.777 89.75 0.008755 0.00841492 2024-03-26T03:09:40.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:41.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:41.115 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.161 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:41.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:41.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:41.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:41.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:41.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:41.448 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 236, 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:09:38.287438+0000", 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T03:06:19.254688+0000", 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:09:41.451 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5806080, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "nodeep-scrub", 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "noscrub", 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 9, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:09:41.452 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.453 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 11, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 11, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 11, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 11, 2024-03-26T03:09:41.454 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 11, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "216", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "164", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:09:41.455 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.456 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.457 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.9000000953674316, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.9000000953674316, 2024-03-26T03:09:41.458 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.5499999523162842, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.5499999523162842, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.459 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.460 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.461 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.462 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.463 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.464 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.465 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:09:41.466 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.469 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.470 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.471 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.472 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.473 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.474 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.475 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.476 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.477 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.478 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2400000095367432, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2400000095367432, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:09:41.479 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:09:41.480 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:09:41.481 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.482 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.483 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:09:41.484 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.485 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.485 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.485 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:09:41.485 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:09:41.485 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:09:41.486 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.487 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:09:41.488 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.489 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.490 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.491 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 9, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_2", 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:08:41.897072+0000", 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:09:41.493 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 16, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 16, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 16, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 16, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 16, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:09:41.494 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "228", 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:09:41.495 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:09:41.496 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:09:41.497 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.498 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.499 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:09:41.500 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.501 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.502 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.503 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:09:41.504 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:09:41.506 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.507 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.509 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:09:41.509 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.509 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.509 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.510 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.511 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.512 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 0, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:09:41.513 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 198, 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.514 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.515 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:09:41.516 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "exists" 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.517 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.518 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.519 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:09:41.520 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.521 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:09:41.522 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.523 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.523 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.523 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.523 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.523 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.523 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.524 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.525 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:09:41.526 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.527 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.529 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:09:41.530 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 225, 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:09:41.531 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.532 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:09:41.533 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:08:31.195879+0000", 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:09:37.857818+0000", 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:09:41.535 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:09:34.672633+0000", 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.536 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:05:36.091229+0000", 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 197 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:09:03.061288+0000", 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:09:41.537 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:09:36.328289+0000", 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.538 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:08:11.653072+0000", 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:08:31.353758+0000", 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:09:41.539 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [ 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.f", 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:09:41.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:09:41.542 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [], 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:09:41.543 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:09:41.544 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:09:41.545 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:09:41.546 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:09:41.546 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:09:41.546 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:09:41.546 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:09:41.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:41.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:41.585 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:41 smithi066 ceph-mon[29666]: pgmap v460: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 42273/252057 objects degraded (16.771%) 2024-03-26T03:09:41.585 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:41.586 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:41 smithi066 ceph-mon[37920]: pgmap v460: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 42273/252057 objects degraded (16.771%) 2024-03-26T03:09:41.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:41.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:09:41.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:41.706 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:41.707 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:41 smithi192 ceph-mon[33110]: pgmap v460: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 42273/252057 objects degraded (16.771%) 2024-03-26T03:09:41.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:41.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:41.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:41.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:41.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:41.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 47 16 88853 88837 118.117 87.75 0.00729325 0.00846135 2024-03-26T03:09:41.969 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:41.974 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:42.021 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:42.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:42.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:42.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:42.302 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:42.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:42 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2258481575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:09:42.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:42 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:09:42.378 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:42 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:09:42.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:42.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:42.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:42.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2258481575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:09:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:42 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:09:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:42 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:09:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:42 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2258481575' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:09:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:42 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:09:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:42 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:09:42.683 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:42.718 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:42.765 INFO:tasks.thrashosds.thrasher:j is {'epoch': 236, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:09:38.287438+0000', 'last_up_change': '2024-03-26T03:06:19.254688+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '216', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '164', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9000000953674316, 'score_stable': 2.9000000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.549999952316284, 'raw_score_stable': 2.549999952316284, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-26T03:08:41.897072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 172, 'down_at': 198, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 225, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:08:31.195879+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:09:37.857818+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:09:34.672633+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:05:36.091229+0000', 'dead_epoch': 197}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:09:03.061288+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:09:36.328289+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:08:11.653072+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:08:31.353758+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [4, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:09:42.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:09:42.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:42.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:42.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:42.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 48 16 90083 90067 117.258 76.875 0.011462 0.00852389 2024-03-26T03:09:42.957 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:42.999 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:43.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:43.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:43.241 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:43.275 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:43.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:43.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:43.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:43.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:43.605 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:43 smithi192 ceph-mon[33110]: pgmap v461: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 740 B/s rd, 81 MiB/s wr, 1.30k op/s; 42698/257472 objects degraded (16.584%) 2024-03-26T03:09:43.605 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:43 smithi192 ceph-mon[33110]: osdmap e237: 8 total, 7 up, 8 in 2024-03-26T03:09:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:43 smithi066 ceph-mon[29666]: pgmap v461: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 740 B/s rd, 81 MiB/s wr, 1.30k op/s; 42698/257472 objects degraded (16.584%) 2024-03-26T03:09:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:43 smithi066 ceph-mon[29666]: osdmap e237: 8 total, 7 up, 8 in 2024-03-26T03:09:43.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:43 smithi066 ceph-mon[37920]: pgmap v461: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 740 B/s rd, 81 MiB/s wr, 1.30k op/s; 42698/257472 objects degraded (16.584%) 2024-03-26T03:09:43.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:43 smithi066 ceph-mon[37920]: osdmap e237: 8 total, 7 up, 8 in 2024-03-26T03:09:43.656 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:43.702 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:43.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:43.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:43.924 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:09:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:09:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:09:43.928 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 49 16 91411 91395 116.558 83 0.00884795 0.00857496 2024-03-26T03:09:43.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:43.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:43.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:44.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.132 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:44.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:44.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:44.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:44.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:44.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:44.512 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:44.553 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:44.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 42698/257472 objects degraded (16.584%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:44.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4253874823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:09:44.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:44 smithi066 ceph-mon[29666]: from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:09:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 42698/257472 objects degraded (16.584%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4253874823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:09:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:44 smithi066 ceph-mon[37920]: from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:09:44.654 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:44.687 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.697 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:44.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 42698/257472 objects degraded (16.584%), 30 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4253874823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:09:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:44 smithi192 ceph-mon[33110]: from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:09:44.750 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:44.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:44.843 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:44.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 50 16 92802 92786 115.963 86.9375 0.00846153 0.00861868 2024-03-26T03:09:44.945 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:44.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:44.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:45.059 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:09:45.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:45.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:45.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:45.268 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:45.370 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:45.408 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:45.509 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:45.551 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[29666]: pgmap v463: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 743 B/s rd, 65 MiB/s wr, 1.05k op/s; 42698/257472 objects degraded (16.584%) 2024-03-26T03:09:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[29666]: from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:09:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[29666]: osdmap e238: 8 total, 7 up, 8 in 2024-03-26T03:09:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[37920]: pgmap v463: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 743 B/s rd, 65 MiB/s wr, 1.05k op/s; 42698/257472 objects degraded (16.584%) 2024-03-26T03:09:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[37920]: from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:09:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:45 smithi066 ceph-mon[37920]: osdmap e238: 8 total, 7 up, 8 in 2024-03-26T03:09:45.653 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:45.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:45 smithi192 ceph-mon[33110]: pgmap v463: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 743 B/s rd, 65 MiB/s wr, 1.05k op/s; 42698/257472 objects degraded (16.584%) 2024-03-26T03:09:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:45 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:09:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:45 smithi192 ceph-mon[33110]: from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:09:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:45 smithi192 ceph-mon[33110]: osdmap e238: 8 total, 7 up, 8 in 2024-03-26T03:09:45.792 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:45.822 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:45.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:45.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 51 16 94074 94058 115.248 79.5 0.0156657 0.00867194 2024-03-26T03:09:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:45.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:45.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:46.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:46.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:46.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:46.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:46.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:46.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:46.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:46.501 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:46.603 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:09:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:09:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:09:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:09:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:09:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:09:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:09:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:46 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:09:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:46.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:46 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:09:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:46 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:09:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:46 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:09:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:46 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:09:46.744 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:46.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:46.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:46.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:46.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:46.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 52 16 95426 95410 114.656 84.5 0.0140204 0.00871723 2024-03-26T03:09:47.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:47.058 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:47.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:47.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:47.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:47.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:47.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:47.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:47.469 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:47.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:47 smithi066 ceph-mon[29666]: pgmap v465: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 112 MiB/s wr, 1.78k op/s; 46155/277923 objects degraded (16.607%) 2024-03-26T03:09:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:47 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:09:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:47 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:09:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:47 smithi066 ceph-mon[37920]: pgmap v465: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 112 MiB/s wr, 1.78k op/s; 46155/277923 objects degraded (16.607%) 2024-03-26T03:09:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:47 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:09:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:47 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:09:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:47.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:47.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:47 smithi192 ceph-mon[33110]: pgmap v465: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 112 MiB/s wr, 1.78k op/s; 46155/277923 objects degraded (16.607%) 2024-03-26T03:09:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:47 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:09:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:47 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:09:47.749 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:47.851 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:47.887 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:47.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:47.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 53 16 96848 96832 114.17 88.875 0.00748023 0.0087539 2024-03-26T03:09:47.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:48.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:48.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:48.170 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:48.271 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:48.314 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:48.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:48.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:48.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:48.593 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:09:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:09:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:09:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:09:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:09:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:48 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:09:48.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:48.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:09:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:09:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:09:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:09:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:09:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:09:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:09:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:48 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:09:48.834 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:48.875 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:48.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:48.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 54 16 98285 98269 113.719 89.8125 0.00617543 0.00878906 2024-03-26T03:09:48.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:49.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:49.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:49.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:49.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:49.311 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:49.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:49.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: pgmap v466: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.08k op/s; 46155/277923 objects degraded (16.607%) 2024-03-26T03:09:49.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:09:49.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:09:49.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 46155/277923 objects degraded (16.607%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[29666]: osdmap e239: 8 total, 7 up, 8 in 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: pgmap v466: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.08k op/s; 46155/277923 objects degraded (16.607%) 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:09:49.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:09:49.421 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 46155/277923 objects degraded (16.607%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:49.421 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:49 smithi066 ceph-mon[37920]: osdmap e239: 8 total, 7 up, 8 in 2024-03-26T03:09:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.464 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:49.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:49.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:49.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:49.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:49.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:09:49.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:09:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: pgmap v466: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.08k op/s; 46155/277923 objects degraded (16.607%) 2024-03-26T03:09:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:09:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:09:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:09:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:09:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:09:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:09:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 46155/277923 objects degraded (16.607%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:49 smithi192 ceph-mon[33110]: osdmap e239: 8 total, 7 up, 8 in 2024-03-26T03:09:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:49.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:49.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:49.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:49.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:49.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 55 16 99626 99610 113.175 83.8125 0.0140442 0.0088311 2024-03-26T03:09:49.979 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:50.046 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:50.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:50.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:50.203 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:09:50.242 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:09:50.277 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:09:50.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:50.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:50.317 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:09:50.356 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:09:50.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:09:50.417 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:09:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:09:50.466 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:50.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:50.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:50 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:09:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:50 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:09:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:50 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/166605926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:50 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:09:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:50 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:09:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:50 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/166605926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:50.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:50 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:09:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:50 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:09:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:50 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/166605926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:09:50.752 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:50.854 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:50.898 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:50.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:50.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 56 16 100907 100891 112.583 80.0625 0.00590837 0.00887742 2024-03-26T03:09:51.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:51.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:51.142 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:51.185 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:51.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:09:51.313 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:09:51.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:51.426 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.466 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:51.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:51.469 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:09:51.469 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:09:51.469 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:09:51.469 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:09:51.469 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:09:51.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:51.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: pgmap v468: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.53k op/s; 49191/293448 objects degraded (16.763%) 2024-03-26T03:09:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:09:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:09:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: pgmap v468: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.53k op/s; 49191/293448 objects degraded (16.763%) 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:09:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:09:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:09:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:51 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:09:51.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:09:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: pgmap v468: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.53k op/s; 49191/293448 objects degraded (16.763%) 2024-03-26T03:09:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:09:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:09:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:09:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:09:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:09:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:51 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:09:51.742 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:51.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:51.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:51.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:51.930 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 57 15 102322 102307 112.161 88.5 0.0141498 0.00891158 2024-03-26T03:09:51.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:52.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:52.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:52.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:52.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:52.286 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:52.388 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:52.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:52.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:09:52.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:09:52.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:09:52.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:09:52.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:09:52.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:52 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:09:52.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:52.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:09:52.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:52.709 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:52 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:09:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:52 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:09:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:52 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:09:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:52 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:09:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:52 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:09:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:52 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:09:52.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:52.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:52.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 58 16 103704 103688 111.715 86.3125 0.00670007 0.00894685 2024-03-26T03:09:52.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:52.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:52.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:53.088 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:53.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:53.228 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:53.266 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:53.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:09:53.368 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:53.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:53.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:53.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[37920]: pgmap v469: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 49576/298383 objects degraded (16.615%) 2024-03-26T03:09:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:09:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:09:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:09:53.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:09:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[29666]: pgmap v469: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 49576/298383 objects degraded (16.615%) 2024-03-26T03:09:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:09:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:09:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:09:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:09:53.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:53.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:53.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:53 smithi192 ceph-mon[33110]: pgmap v469: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 49576/298383 objects degraded (16.615%) 2024-03-26T03:09:53.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:53 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:09:53.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:53 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:09:53.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:53 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:09:53.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:53 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:09:53.798 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:53.846 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:53.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 59 16 105073 105057 111.272 85.5625 0.00701872 0.00898263 2024-03-26T03:09:53.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:53.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:53.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:54.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:09:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:09:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:09:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:54.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:54.232 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:54.276 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:54.378 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.405 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:54.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:54.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:54.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:54.577 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:09:54.580 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:09:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:09:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:09:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 49576/298383 objects degraded (16.615%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: osdmap e240: 8 total, 7 up, 8 in 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 49576/298383 objects degraded (16.615%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: osdmap e240: 8 total, 7 up, 8 in 2024-03-26T03:09:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:54.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:54.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:09:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:09:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:09:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:09:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 49576/298383 objects degraded (16.615%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: osdmap e240: 8 total, 7 up, 8 in 2024-03-26T03:09:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:09:54.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:54.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:54.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:54.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:09:54.931303+0000 min lat: 0.00130673 max lat: 0.137828 avg lat: 0.0090278 2024-03-26T03:09:54.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:09:54.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 60 16 106312 106296 110.708 77.4375 0.015541 0.0090278 2024-03-26T03:09:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:54.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:54.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:55.061 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:55.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:55.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:55.233 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:55.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:55.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:55.424 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:55.452 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-03-26T03:09:54.174116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6531246926,"num_objects":99883,"num_object_clones":0,"num_object_copies":299649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99883,"num_whiteouts":0,"num_read":2096,"num_read_kb":1825,"num_write":100584,"num_write_kb":6378121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":161340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83588,"ondisk_log_size":83588,"up":406,"acting":406,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":472,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21297924,"kb_used_data":18298296,"kb_used_omap":12,"kb_used_meta":2999539,"kb_avail":728663292,"statfs":{"total":767960285184,"available":746151211008,"internally_reserved":0,"allocated":18737455104,"data_stored":18734047175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3071528496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":474611712,"num_objects":7242,"num_object_clones":0,"num_object_copies":21726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7242,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7242,"num_write_kb":463488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004677"},"pg_stats":[{"pgid":"9.5","version":"239'6284","reported_seq":6322,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:50.454197+0000","last_change":"2024-03-26T03:08:43.065724+0000","last_active":"2024-03-26T03:09:50.454197+0000","last_peered":"2024-03-26T03:09:50.454197+0000","last_clean":"2024-03-26T03:09:50.454197+0000","last_became_active":"2024-03-26T03:08:43.065248+0000","last_became_peered":"2024-03-26T03:08:43.065248+0000","last_unstale":"2024-03-26T03:09:50.454197+0000","last_undegraded":"2024-03-26T03:09:50.454197+0000","last_fullsized":"2024-03-26T03:09:50.454197+0000","mapping_epoch":225,"log_start":"229'700","ondisk_log_start":"229'700","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:01.566679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411828224,"num_objects":6284,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6284,"num_write_kb":402176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":537,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.058173+0000","last_change":"2024-03-26T03:09:18.705410+0000","last_active":"2024-03-26T03:09:49.058173+0000","last_peered":"2024-03-26T03:09:49.058173+0000","last_clean":"2024-03-26T03:09:49.058173+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:09:49.058173+0000","last_undegraded":"2024-03-26T03:09:49.058173+0000","last_fullsized":"2024-03-26T03:09:49.058173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:09:18.705279+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:09:18.705279+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:19.098363+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":476,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.050783+0000","last_change":"2024-03-26T03:09:06.967619+0000","last_active":"2024-03-26T03:09:49.050783+0000","last_peered":"2024-03-26T03:09:49.050783+0000","last_clean":"2024-03-26T03:09:49.050783+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:09:49.050783+0000","last_undegraded":"2024-03-26T03:09:49.050783+0000","last_fullsized":"2024-03-26T03:09:49.050783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:09:06.966663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:06.966663+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:14.245790+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":392,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.042714+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:09:49.042714+0000","last_peered":"2024-03-26T03:09:49.042714+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:09:49.042714+0000","last_undegraded":"2024-03-26T03:09:49.042714+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":206,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057627+0000","last_change":"2024-03-26T03:09:18.217911+0000","last_active":"2024-03-26T03:09:49.057627+0000","last_peered":"2024-03-26T03:09:49.057627+0000","last_clean":"2024-03-26T03:09:49.057627+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:09:49.057627+0000","last_undegraded":"2024-03-26T03:09:49.057627+0000","last_fullsized":"2024-03-26T03:09:49.057627+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:18.217789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:18.217789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.742511+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"239'6226","reported_seq":6264,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.518615+0000","last_change":"2024-03-26T03:08:43.064329+0000","last_active":"2024-03-26T03:09:49.518615+0000","last_peered":"2024-03-26T03:09:49.518615+0000","last_clean":"2024-03-26T03:09:49.518615+0000","last_became_active":"2024-03-26T03:08:43.063988+0000","last_became_peered":"2024-03-26T03:08:43.063988+0000","last_unstale":"2024-03-26T03:09:49.518615+0000","last_undegraded":"2024-03-26T03:09:49.518615+0000","last_fullsized":"2024-03-26T03:09:49.518615+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:08.755323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408027136,"num_objects":6226,"num_object_clones":0,"num_object_copies":18678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":398464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":443,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.045291+0000","last_change":"2024-03-26T03:09:02.127628+0000","last_active":"2024-03-26T03:09:49.045291+0000","last_peered":"2024-03-26T03:09:49.045291+0000","last_clean":"2024-03-26T03:09:49.045291+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:09:49.045291+0000","last_undegraded":"2024-03-26T03:09:49.045291+0000","last_fullsized":"2024-03-26T03:09:49.045291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:09:02.127494+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_clean_scrub_stamp":"2024-03-26T03:09:02.127494+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:05.648324+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":392,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.049870+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:09:49.049870+0000","last_peered":"2024-03-26T03:09:49.049870+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:09:49.049870+0000","last_undegraded":"2024-03-26T03:09:49.049870+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339270+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":376,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.041132+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:09:49.041132+0000","last_peered":"2024-03-26T03:09:49.041132+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:09:49.041132+0000","last_undegraded":"2024-03-26T03:09:49.041132+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":393,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.045490+0000","last_change":"2024-03-26T03:09:04.206625+0000","last_active":"2024-03-26T03:09:49.045490+0000","last_peered":"2024-03-26T03:09:49.045490+0000","last_clean":"2024-03-26T03:09:49.045490+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:09:49.045490+0000","last_undegraded":"2024-03-26T03:09:49.045490+0000","last_fullsized":"2024-03-26T03:09:49.045490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:04.203542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:09:04.203542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.035575+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"239'6151","reported_seq":6186,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.212414+0000","last_change":"2024-03-26T03:08:54.927812+0000","last_active":"2024-03-26T03:09:49.212414+0000","last_peered":"2024-03-26T03:09:49.212414+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.069288+0000","last_became_peered":"2024-03-26T03:08:43.069288+0000","last_unstale":"2024-03-26T03:09:49.212414+0000","last_undegraded":"2024-03-26T03:08:54.925664+0000","last_fullsized":"2024-03-26T03:08:43.062900+0000","mapping_epoch":225,"log_start":"230'1800","ondisk_log_start":"230'1800","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4351,"log_dups_size":0,"ondisk_log_size":4351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:06.454112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403111936,"num_objects":6151,"num_object_clones":0,"num_object_copies":18453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6151,"num_write_kb":393664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":6151}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":496,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057943+0000","last_change":"2024-03-26T03:09:03.343435+0000","last_active":"2024-03-26T03:09:49.057943+0000","last_peered":"2024-03-26T03:09:49.057943+0000","last_clean":"2024-03-26T03:09:49.057943+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:09:49.057943+0000","last_undegraded":"2024-03-26T03:09:49.057943+0000","last_fullsized":"2024-03-26T03:09:49.057943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:09:03.343287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:03.343287+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:33.205119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":413,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.041918+0000","last_change":"2024-03-26T03:08:58.028786+0000","last_active":"2024-03-26T03:09:49.041918+0000","last_peered":"2024-03-26T03:09:49.041918+0000","last_clean":"2024-03-26T03:09:49.041918+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:09:49.041918+0000","last_undegraded":"2024-03-26T03:09:49.041918+0000","last_fullsized":"2024-03-26T03:09:49.041918+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:58.028621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:08:58.028621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:15.367458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.041604+0000","last_change":"2024-03-26T03:08:02.929471+0000","last_active":"2024-03-26T03:09:49.041604+0000","last_peered":"2024-03-26T03:09:49.041604+0000","last_clean":"2024-03-26T03:09:49.041604+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:09:49.041604+0000","last_undegraded":"2024-03-26T03:09:49.041604+0000","last_fullsized":"2024-03-26T03:09:49.041604+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:02.929349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:08:02.929349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":372,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.054272+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:09:49.054272+0000","last_peered":"2024-03-26T03:09:49.054272+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:09:49.054272+0000","last_undegraded":"2024-03-26T03:09:49.054272+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"239'6199","reported_seq":6237,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.737772+0000","last_change":"2024-03-26T03:08:43.064333+0000","last_active":"2024-03-26T03:09:49.737772+0000","last_peered":"2024-03-26T03:09:49.737772+0000","last_clean":"2024-03-26T03:09:49.737772+0000","last_became_active":"2024-03-26T03:08:43.063932+0000","last_became_peered":"2024-03-26T03:08:43.063932+0000","last_unstale":"2024-03-26T03:09:49.737772+0000","last_undegraded":"2024-03-26T03:09:49.737772+0000","last_fullsized":"2024-03-26T03:09:49.737772+0000","mapping_epoch":225,"log_start":"229'1500","ondisk_log_start":"229'1500","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:58.010971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406257664,"num_objects":6199,"num_object_clones":0,"num_object_copies":18597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6199,"num_write_kb":396736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":459,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.045388+0000","last_change":"2024-03-26T03:09:11.836610+0000","last_active":"2024-03-26T03:09:54.045388+0000","last_peered":"2024-03-26T03:09:54.045388+0000","last_clean":"2024-03-26T03:09:54.045388+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:09:54.045388+0000","last_undegraded":"2024-03-26T03:09:54.045388+0000","last_fullsized":"2024-03-26T03:09:54.045388+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:09:11.836484+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:09:11.836484+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.468304+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":405,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.055366+0000","last_change":"2024-03-26T03:09:47.553143+0000","last_active":"2024-03-26T03:09:49.055366+0000","last_peered":"2024-03-26T03:09:49.055366+0000","last_clean":"2024-03-26T03:09:49.055366+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:09:49.055366+0000","last_undegraded":"2024-03-26T03:09:49.055366+0000","last_fullsized":"2024-03-26T03:09:49.055366+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:47.553022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.553022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:13.878799+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":389,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.047668+0000","last_change":"2024-03-26T03:09:15.447157+0000","last_active":"2024-03-26T03:09:49.047668+0000","last_peered":"2024-03-26T03:09:49.047668+0000","last_clean":"2024-03-26T03:09:49.047668+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:09:49.047668+0000","last_undegraded":"2024-03-26T03:09:49.047668+0000","last_fullsized":"2024-03-26T03:09:49.047668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:15.447025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:09:15.447025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:21.944032+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":380,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.042872+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:09:49.042872+0000","last_peered":"2024-03-26T03:09:49.042872+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:09:49.042872+0000","last_undegraded":"2024-03-26T03:09:49.042872+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":459,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.041084+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:09:49.041084+0000","last_peered":"2024-03-26T03:09:49.041084+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:09:49.041084+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"239'6251","reported_seq":6286,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.225024+0000","last_change":"2024-03-26T03:08:54.926696+0000","last_active":"2024-03-26T03:09:49.225024+0000","last_peered":"2024-03-26T03:09:49.225024+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.066721+0000","last_became_peered":"2024-03-26T03:08:43.066721+0000","last_unstale":"2024-03-26T03:09:49.225024+0000","last_undegraded":"2024-03-26T03:08:54.926086+0000","last_fullsized":"2024-03-26T03:08:43.059549+0000","mapping_epoch":225,"log_start":"229'300","ondisk_log_start":"229'300","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:06.913170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409665536,"num_objects":6251,"num_object_clones":0,"num_object_copies":18753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":6251}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":650,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.057435+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:09:49.057435+0000","last_peered":"2024-03-26T03:09:49.057435+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:09:49.057435+0000","last_undegraded":"2024-03-26T03:09:49.057435+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":419,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:54.046576+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:09:54.046576+0000","last_peered":"2024-03-26T03:09:54.046576+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:09:54.046576+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":385,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.049261+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:09:49.049261+0000","last_peered":"2024-03-26T03:09:49.049261+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:09:49.049261+0000","last_undegraded":"2024-03-26T03:09:49.049261+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339226+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":537,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.042069+0000","last_change":"2024-03-26T03:08:05.957056+0000","last_active":"2024-03-26T03:09:49.042069+0000","last_peered":"2024-03-26T03:09:49.042069+0000","last_clean":"2024-03-26T03:09:49.042069+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:09:49.042069+0000","last_undegraded":"2024-03-26T03:09:49.042069+0000","last_fullsized":"2024-03-26T03:09:49.042069+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:08:05.956667+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":386,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.046223+0000","last_change":"2024-03-26T03:09:17.928221+0000","last_active":"2024-03-26T03:09:54.046223+0000","last_peered":"2024-03-26T03:09:54.046223+0000","last_clean":"2024-03-26T03:09:54.046223+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:09:54.046223+0000","last_undegraded":"2024-03-26T03:09:54.046223+0000","last_fullsized":"2024-03-26T03:09:54.046223+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:17.927617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:17.927617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:47.317421+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"239'6235","reported_seq":6270,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.744915+0000","last_change":"2024-03-26T03:08:54.925459+0000","last_active":"2024-03-26T03:09:49.744915+0000","last_peered":"2024-03-26T03:09:49.744915+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.057035+0000","last_became_peered":"2024-03-26T03:08:43.057035+0000","last_unstale":"2024-03-26T03:09:49.744915+0000","last_undegraded":"2024-03-26T03:08:54.924597+0000","last_fullsized":"2024-03-26T03:08:43.055113+0000","mapping_epoch":225,"log_start":"229'1500","ondisk_log_start":"229'1500","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408616960,"num_objects":6235,"num_object_clones":0,"num_object_copies":18705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":399040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6235}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":481,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.041261+0000","last_change":"2024-03-26T03:09:17.328769+0000","last_active":"2024-03-26T03:09:49.041261+0000","last_peered":"2024-03-26T03:09:49.041261+0000","last_clean":"2024-03-26T03:09:49.041261+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:09:49.041261+0000","last_undegraded":"2024-03-26T03:09:49.041261+0000","last_fullsized":"2024-03-26T03:09:49.041261+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:17.328613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:09:17.328613+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:20.297744+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":431,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.046695+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:09:49.046695+0000","last_peered":"2024-03-26T03:09:49.046695+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:09:49.046695+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339257+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":385,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.046472+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:09:49.046472+0000","last_peered":"2024-03-26T03:09:49.046472+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:09:49.046472+0000","last_undegraded":"2024-03-26T03:09:49.046472+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":538,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.046128+0000","last_change":"2024-03-26T03:09:51.797832+0000","last_active":"2024-03-26T03:09:54.046128+0000","last_peered":"2024-03-26T03:09:54.046128+0000","last_clean":"2024-03-26T03:09:54.046128+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:09:54.046128+0000","last_undegraded":"2024-03-26T03:09:54.046128+0000","last_fullsized":"2024-03-26T03:09:54.046128+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:09:51.797681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:51.797681+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:06.772612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":190,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.049023+0000","last_change":"2024-03-26T03:08:13.507275+0000","last_active":"2024-03-26T03:09:49.049023+0000","last_peered":"2024-03-26T03:09:49.049023+0000","last_clean":"2024-03-26T03:09:49.049023+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:09:49.049023+0000","last_undegraded":"2024-03-26T03:09:49.049023+0000","last_fullsized":"2024-03-26T03:09:49.049023+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:13.507146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:08:13.507146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"239'6141","reported_seq":6176,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.230104+0000","last_change":"2024-03-26T03:08:54.940154+0000","last_active":"2024-03-26T03:09:49.230104+0000","last_peered":"2024-03-26T03:09:49.230104+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.064645+0000","last_became_peered":"2024-03-26T03:08:43.064645+0000","last_unstale":"2024-03-26T03:09:49.230104+0000","last_undegraded":"2024-03-26T03:08:54.937570+0000","last_fullsized":"2024-03-26T03:08:43.060747+0000","mapping_epoch":225,"log_start":"230'1800","ondisk_log_start":"230'1800","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4341,"log_dups_size":0,"ondisk_log_size":4341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:57.508305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402456576,"num_objects":6141,"num_object_clones":0,"num_object_copies":18423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":6141}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"231'25","reported_seq":445,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.045802+0000","last_change":"2024-03-26T03:09:08.904745+0000","last_active":"2024-03-26T03:09:54.045802+0000","last_peered":"2024-03-26T03:09:54.045802+0000","last_clean":"2024-03-26T03:09:54.045802+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:09:54.045802+0000","last_undegraded":"2024-03-26T03:09:54.045802+0000","last_fullsized":"2024-03-26T03:09:54.045802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:09:08.904611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:08.904611+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:30.544328+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.054667+0000","last_change":"2024-03-26T03:08:00.792256+0000","last_active":"2024-03-26T03:09:49.054667+0000","last_peered":"2024-03-26T03:09:49.054667+0000","last_clean":"2024-03-26T03:09:49.054667+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:09:49.054667+0000","last_undegraded":"2024-03-26T03:09:49.054667+0000","last_fullsized":"2024-03-26T03:09:49.054667+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:00.792042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:00.792042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":198,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.044304+0000","last_change":"2024-03-26T03:08:01.699554+0000","last_active":"2024-03-26T03:09:54.044304+0000","last_peered":"2024-03-26T03:09:54.044304+0000","last_clean":"2024-03-26T03:09:54.044304+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:09:54.044304+0000","last_undegraded":"2024-03-26T03:09:54.044304+0000","last_fullsized":"2024-03-26T03:09:54.044304+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:01.699429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:01.699429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.066478+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:09:49.066478+0000","last_peered":"2024-03-26T03:09:49.066478+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:09:49.066478+0000","last_undegraded":"2024-03-26T03:09:49.066478+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.317200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"240'6622","reported_seq":6657,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:54.159504+0000","last_change":"2024-03-26T03:08:54.944412+0000","last_active":"2024-03-26T03:09:54.159504+0000","last_peered":"2024-03-26T03:09:54.159504+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.063578+0000","last_became_peered":"2024-03-26T03:08:43.063578+0000","last_unstale":"2024-03-26T03:09:54.159504+0000","last_undegraded":"2024-03-26T03:08:54.938583+0000","last_fullsized":"2024-03-26T03:08:43.056078+0000","mapping_epoch":225,"log_start":"229'1500","ondisk_log_start":"229'1500","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433979392,"num_objects":6622,"num_object_clones":0,"num_object_copies":19866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":6622}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":222,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.047564+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:09:49.047564+0000","last_peered":"2024-03-26T03:09:49.047564+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:09:49.047564+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":457,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.044684+0000","last_change":"2024-03-26T03:09:10.837559+0000","last_active":"2024-03-26T03:09:54.044684+0000","last_peered":"2024-03-26T03:09:54.044684+0000","last_clean":"2024-03-26T03:09:54.044684+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:09:54.044684+0000","last_undegraded":"2024-03-26T03:09:54.044684+0000","last_fullsized":"2024-03-26T03:09:54.044684+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:09:10.837466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:10.837466+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:15.221750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":406,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.066291+0000","last_change":"2024-03-26T03:08:57.389505+0000","last_active":"2024-03-26T03:09:49.066291+0000","last_peered":"2024-03-26T03:09:49.066291+0000","last_clean":"2024-03-26T03:09:49.066291+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:09:49.066291+0000","last_undegraded":"2024-03-26T03:09:49.066291+0000","last_fullsized":"2024-03-26T03:09:49.066291+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:57.389433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.389433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:27.014231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":386,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.055672+0000","last_change":"2024-03-26T03:08:59.750899+0000","last_active":"2024-03-26T03:09:49.055672+0000","last_peered":"2024-03-26T03:09:49.055672+0000","last_clean":"2024-03-26T03:09:49.055672+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:09:49.055672+0000","last_undegraded":"2024-03-26T03:09:49.055672+0000","last_fullsized":"2024-03-26T03:09:49.055672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:59.750496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:08:59.750496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.239093+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":567,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.056217+0000","last_change":"2024-03-26T03:06:20.276636+0000","last_active":"2024-03-26T03:09:49.056217+0000","last_peered":"2024-03-26T03:09:49.056217+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:06:20.276636+0000","last_became_peered":"2024-03-26T03:06:20.276636+0000","last_unstale":"2024-03-26T03:09:49.056217+0000","last_undegraded":"2024-03-26T03:09:49.056217+0000","last_fullsized":"2024-03-26T03:06:20.268958+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.517404+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"239'6261","reported_seq":6299,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.218904+0000","last_change":"2024-03-26T03:08:43.069162+0000","last_active":"2024-03-26T03:09:49.218904+0000","last_peered":"2024-03-26T03:09:49.218904+0000","last_clean":"2024-03-26T03:09:49.218904+0000","last_became_active":"2024-03-26T03:08:43.068853+0000","last_became_peered":"2024-03-26T03:08:43.068853+0000","last_unstale":"2024-03-26T03:09:49.218904+0000","last_undegraded":"2024-03-26T03:09:49.218904+0000","last_fullsized":"2024-03-26T03:09:49.218904+0000","mapping_epoch":225,"log_start":"230'1900","ondisk_log_start":"230'1900","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.336203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410320896,"num_objects":6261,"num_object_clones":0,"num_object_copies":18783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6261,"num_write_kb":400704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":378,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-26T03:09:54.045543+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:09:54.045543+0000","last_peered":"2024-03-26T03:09:54.045543+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:09:54.045543+0000","last_undegraded":"2024-03-26T03:09:54.045543+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":216,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.054910+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:09:49.054910+0000","last_peered":"2024-03-26T03:09:49.054910+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:09:49.054910+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":400,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.048845+0000","last_change":"2024-03-26T03:08:19.494106+0000","last_active":"2024-03-26T03:09:49.048845+0000","last_peered":"2024-03-26T03:09:49.048845+0000","last_clean":"2024-03-26T03:09:49.048845+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:09:49.048845+0000","last_undegraded":"2024-03-26T03:09:49.048845+0000","last_fullsized":"2024-03-26T03:09:49.048845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:08:19.493978+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:08:19.493978+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":374,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-26T03:09:54.046707+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:09:54.046707+0000","last_peered":"2024-03-26T03:09:54.046707+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:09:54.046707+0000","last_undegraded":"2024-03-26T03:09:54.046707+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":475,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.045218+0000","last_change":"2024-03-26T03:07:54.784105+0000","last_active":"2024-03-26T03:09:54.045218+0000","last_peered":"2024-03-26T03:09:54.045218+0000","last_clean":"2024-03-26T03:09:54.045218+0000","last_became_active":"2024-03-26T03:04:41.967264+0000","last_became_peered":"2024-03-26T03:04:41.967264+0000","last_unstale":"2024-03-26T03:09:54.045218+0000","last_undegraded":"2024-03-26T03:09:54.045218+0000","last_fullsized":"2024-03-26T03:09:54.045218+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:07:54.783902+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:07:54.783902+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"239'6111","reported_seq":6149,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.225435+0000","last_change":"2024-03-26T03:08:43.067656+0000","last_active":"2024-03-26T03:09:49.225435+0000","last_peered":"2024-03-26T03:09:49.225435+0000","last_clean":"2024-03-26T03:09:49.225435+0000","last_became_active":"2024-03-26T03:08:43.067345+0000","last_became_peered":"2024-03-26T03:08:43.067345+0000","last_unstale":"2024-03-26T03:09:49.225435+0000","last_undegraded":"2024-03-26T03:09:49.225435+0000","last_fullsized":"2024-03-26T03:09:49.225435+0000","mapping_epoch":225,"log_start":"230'1800","ondisk_log_start":"230'1800","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.622955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400490496,"num_objects":6111,"num_object_clones":0,"num_object_copies":18333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6111,"num_write_kb":391104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.058131+0000","last_change":"2024-03-26T03:09:00.346125+0000","last_active":"2024-03-26T03:09:49.058131+0000","last_peered":"2024-03-26T03:09:49.058131+0000","last_clean":"2024-03-26T03:09:49.058131+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:09:49.058131+0000","last_undegraded":"2024-03-26T03:09:49.058131+0000","last_fullsized":"2024-03-26T03:09:49.058131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:00.346008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:00.346008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.285027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":473,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.045148+0000","last_change":"2024-03-26T03:09:47.209344+0000","last_active":"2024-03-26T03:09:49.045148+0000","last_peered":"2024-03-26T03:09:49.045148+0000","last_clean":"2024-03-26T03:09:49.045148+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:09:49.045148+0000","last_undegraded":"2024-03-26T03:09:49.045148+0000","last_fullsized":"2024-03-26T03:09:49.045148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:09:47.209142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.209142+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:12.479392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057908+0000","last_change":"2024-03-26T03:09:09.688709+0000","last_active":"2024-03-26T03:09:49.057908+0000","last_peered":"2024-03-26T03:09:49.057908+0000","last_clean":"2024-03-26T03:09:49.057908+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:09:49.057908+0000","last_undegraded":"2024-03-26T03:09:49.057908+0000","last_fullsized":"2024-03-26T03:09:49.057908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:09.688605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:09.688605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:16.840542+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.046670+0000","last_change":"2024-03-26T03:09:46.394734+0000","last_active":"2024-03-26T03:09:49.046670+0000","last_peered":"2024-03-26T03:09:49.046670+0000","last_clean":"2024-03-26T03:09:49.046670+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:09:49.046670+0000","last_undegraded":"2024-03-26T03:09:49.046670+0000","last_fullsized":"2024-03-26T03:09:49.046670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:46.394652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:46.394652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:15.682689+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":466,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.065642+0000","last_change":"2024-03-26T03:06:20.308944+0000","last_active":"2024-03-26T03:09:49.065642+0000","last_peered":"2024-03-26T03:09:49.065642+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:06:20.308944+0000","last_became_peered":"2024-03-26T03:06:20.308944+0000","last_unstale":"2024-03-26T03:09:49.065642+0000","last_undegraded":"2024-03-26T03:09:49.065642+0000","last_fullsized":"2024-03-26T03:06:20.270592+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.317233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"239'6210","reported_seq":6248,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:50.466221+0000","last_change":"2024-03-26T03:08:43.067206+0000","last_active":"2024-03-26T03:09:50.466221+0000","last_peered":"2024-03-26T03:09:50.466221+0000","last_clean":"2024-03-26T03:09:50.466221+0000","last_became_active":"2024-03-26T03:08:43.066837+0000","last_became_peered":"2024-03-26T03:08:43.066837+0000","last_unstale":"2024-03-26T03:09:50.466221+0000","last_undegraded":"2024-03-26T03:09:50.466221+0000","last_fullsized":"2024-03-26T03:09:50.466221+0000","mapping_epoch":225,"log_start":"229'600","ondisk_log_start":"229'600","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:57.074829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406978560,"num_objects":6210,"num_object_clones":0,"num_object_copies":18630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":397440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":404,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.053538+0000","last_change":"2024-03-26T03:09:46.001706+0000","last_active":"2024-03-26T03:09:49.053538+0000","last_peered":"2024-03-26T03:09:49.053538+0000","last_clean":"2024-03-26T03:09:49.053538+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:09:49.053538+0000","last_undegraded":"2024-03-26T03:09:49.053538+0000","last_fullsized":"2024-03-26T03:09:49.053538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:46.001491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:46.001491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:12.883102+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":444,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.046033+0000","last_change":"2024-03-26T03:08:11.142660+0000","last_active":"2024-03-26T03:09:49.046033+0000","last_peered":"2024-03-26T03:09:49.046033+0000","last_clean":"2024-03-26T03:09:49.046033+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:09:49.046033+0000","last_undegraded":"2024-03-26T03:09:49.046033+0000","last_fullsized":"2024-03-26T03:09:49.046033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'18","last_scrub_stamp":"2024-03-26T03:08:11.142496+0000","last_deep_scrub":"219'18","last_deep_scrub_stamp":"2024-03-26T03:08:11.142496+0000","last_clean_scrub_stamp":"2024-03-26T03:08:11.142496+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.054450+0000","last_change":"2024-03-26T03:08:04.832413+0000","last_active":"2024-03-26T03:09:49.054450+0000","last_peered":"2024-03-26T03:09:49.054450+0000","last_clean":"2024-03-26T03:09:49.054450+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:09:49.054450+0000","last_undegraded":"2024-03-26T03:09:49.054450+0000","last_fullsized":"2024-03-26T03:09:49.054450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:04.832307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:04.832307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":400,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.044825+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:09:49.044825+0000","last_peered":"2024-03-26T03:09:49.044825+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:09:49.044825+0000","last_undegraded":"2024-03-26T03:09:49.044825+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203351+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":381,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.047785+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:09:49.047785+0000","last_peered":"2024-03-26T03:09:49.047785+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:09:49.047785+0000","last_undegraded":"2024-03-26T03:09:49.047785+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":473,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.046464+0000","last_change":"2024-03-26T03:08:14.472108+0000","last_active":"2024-03-26T03:09:49.046464+0000","last_peered":"2024-03-26T03:09:49.046464+0000","last_clean":"2024-03-26T03:09:49.046464+0000","last_became_active":"2024-03-26T03:04:41.964807+0000","last_became_peered":"2024-03-26T03:04:41.964807+0000","last_unstale":"2024-03-26T03:09:49.046464+0000","last_undegraded":"2024-03-26T03:09:49.046464+0000","last_fullsized":"2024-03-26T03:09:49.046464+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:08:14.471989+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:08:14.471989+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"239'6156","reported_seq":6191,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:50.441626+0000","last_change":"2024-03-26T03:08:54.931470+0000","last_active":"2024-03-26T03:09:50.441626+0000","last_peered":"2024-03-26T03:09:50.441626+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.065260+0000","last_became_peered":"2024-03-26T03:08:43.065260+0000","last_unstale":"2024-03-26T03:09:50.441626+0000","last_undegraded":"2024-03-26T03:08:54.924879+0000","last_fullsized":"2024-03-26T03:08:43.057979+0000","mapping_epoch":225,"log_start":"229'600","ondisk_log_start":"229'600","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403439616,"num_objects":6156,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":393984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":6156}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":390,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057017+0000","last_change":"2024-03-26T03:08:11.778604+0000","last_active":"2024-03-26T03:09:49.057017+0000","last_peered":"2024-03-26T03:09:49.057017+0000","last_clean":"2024-03-26T03:09:49.057017+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:09:49.057017+0000","last_undegraded":"2024-03-26T03:09:49.057017+0000","last_fullsized":"2024-03-26T03:09:49.057017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:11.778221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:11.778221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"231'28","reported_seq":573,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.065507+0000","last_change":"2024-03-26T03:09:20.236853+0000","last_active":"2024-03-26T03:09:49.065507+0000","last_peered":"2024-03-26T03:09:49.065507+0000","last_clean":"2024-03-26T03:09:49.065507+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:09:49.065507+0000","last_undegraded":"2024-03-26T03:09:49.065507+0000","last_fullsized":"2024-03-26T03:09:49.065507+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:09:20.236696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:20.236696+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:40.931974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":656,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.054538+0000","last_change":"2024-03-26T03:09:13.758950+0000","last_active":"2024-03-26T03:09:49.054538+0000","last_peered":"2024-03-26T03:09:49.054538+0000","last_clean":"2024-03-26T03:09:49.054538+0000","last_became_active":"2024-03-26T03:04:37.115068+0000","last_became_peered":"2024-03-26T03:04:37.115068+0000","last_unstale":"2024-03-26T03:09:49.054538+0000","last_undegraded":"2024-03-26T03:09:49.054538+0000","last_fullsized":"2024-03-26T03:09:49.054538+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:13.758832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:09:13.758832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:23.498458+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":419,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.047602+0000","last_change":"2024-03-26T03:08:18.871456+0000","last_active":"2024-03-26T03:09:49.047602+0000","last_peered":"2024-03-26T03:09:49.047602+0000","last_clean":"2024-03-26T03:09:49.047602+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:09:49.047602+0000","last_undegraded":"2024-03-26T03:09:49.047602+0000","last_fullsized":"2024-03-26T03:09:49.047602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:18.871325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:08:18.871325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":116,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.048446+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:09:49.048446+0000","last_peered":"2024-03-26T03:09:49.048446+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:09:49.048446+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.964935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":379,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.053638+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:09:49.053638+0000","last_peered":"2024-03-26T03:09:49.053638+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:09:49.053638+0000","last_undegraded":"2024-03-26T03:09:49.053638+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.964906+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"239'6130","reported_seq":6168,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.264957+0000","last_change":"2024-03-26T03:08:43.066632+0000","last_active":"2024-03-26T03:09:49.264957+0000","last_peered":"2024-03-26T03:09:49.264957+0000","last_clean":"2024-03-26T03:09:49.264957+0000","last_became_active":"2024-03-26T03:08:43.066310+0000","last_became_peered":"2024-03-26T03:08:43.066310+0000","last_unstale":"2024-03-26T03:09:49.264957+0000","last_undegraded":"2024-03-26T03:09:49.264957+0000","last_fullsized":"2024-03-26T03:09:49.264957+0000","mapping_epoch":225,"log_start":"229'1200","ondisk_log_start":"229'1200","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:03.693301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":392320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.042901+0000","last_change":"2024-03-26T03:08:55.087647+0000","last_active":"2024-03-26T03:09:49.042901+0000","last_peered":"2024-03-26T03:09:49.042901+0000","last_clean":"2024-03-26T03:09:49.042901+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:09:49.042901+0000","last_undegraded":"2024-03-26T03:09:49.042901+0000","last_fullsized":"2024-03-26T03:09:49.042901+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:55.087532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:55.087532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:04.492212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":447,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.282590+0000","last_change":"2024-03-26T03:09:49.282590+0000","last_active":"2024-03-26T03:09:49.282590+0000","last_peered":"2024-03-26T03:09:49.282590+0000","last_clean":"2024-03-26T03:09:49.282590+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:09:49.282590+0000","last_undegraded":"2024-03-26T03:09:49.282590+0000","last_fullsized":"2024-03-26T03:09:49.282590+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:09:49.282256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:49.282256+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:10.364929+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":393,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.040704+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:09:49.040704+0000","last_peered":"2024-03-26T03:09:49.040704+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:09:49.040704+0000","last_undegraded":"2024-03-26T03:09:49.040704+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1059,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.048220+0000","last_change":"2024-03-26T03:09:05.434045+0000","last_active":"2024-03-26T03:09:49.048220+0000","last_peered":"2024-03-26T03:09:49.048220+0000","last_clean":"2024-03-26T03:09:49.048220+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:09:49.048220+0000","last_undegraded":"2024-03-26T03:09:49.048220+0000","last_fullsized":"2024-03-26T03:09:49.048220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:09:05.433908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:05.433908+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.083107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":643,"num_read_kb":638,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":381,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.057718+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:09:49.057718+0000","last_peered":"2024-03-26T03:09:49.057718+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:09:49.057718+0000","last_undegraded":"2024-03-26T03:09:49.057718+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538528+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":663,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.056737+0000","last_change":"2024-03-26T03:09:14.720943+0000","last_active":"2024-03-26T03:09:49.056737+0000","last_peered":"2024-03-26T03:09:49.056737+0000","last_clean":"2024-03-26T03:09:49.056737+0000","last_became_active":"2024-03-26T03:04:41.965091+0000","last_became_peered":"2024-03-26T03:04:41.965091+0000","last_unstale":"2024-03-26T03:09:49.056737+0000","last_undegraded":"2024-03-26T03:09:49.056737+0000","last_fullsized":"2024-03-26T03:09:49.056737+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:14.720791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:09:14.720791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:22.530807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"239'6349","reported_seq":6387,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:50.471748+0000","last_change":"2024-03-26T03:08:43.066301+0000","last_active":"2024-03-26T03:09:50.471748+0000","last_peered":"2024-03-26T03:09:50.471748+0000","last_clean":"2024-03-26T03:09:50.471748+0000","last_became_active":"2024-03-26T03:08:43.065826+0000","last_became_peered":"2024-03-26T03:08:43.065826+0000","last_unstale":"2024-03-26T03:09:50.471748+0000","last_undegraded":"2024-03-26T03:09:50.471748+0000","last_fullsized":"2024-03-26T03:09:50.471748+0000","mapping_epoch":225,"log_start":"229'700","ondisk_log_start":"229'700","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5649,"log_dups_size":0,"ondisk_log_size":5649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:57.945182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416088064,"num_objects":6349,"num_object_clones":0,"num_object_copies":19047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":406336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.049512+0000","last_change":"2024-03-26T03:08:57.142251+0000","last_active":"2024-03-26T03:09:49.049512+0000","last_peered":"2024-03-26T03:09:49.049512+0000","last_clean":"2024-03-26T03:09:49.049512+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:09:49.049512+0000","last_undegraded":"2024-03-26T03:09:49.049512+0000","last_fullsized":"2024-03-26T03:09:49.049512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:57.142119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.142119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.006292+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"231'58","reported_seq":610,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.046630+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:09:49.046630+0000","last_peered":"2024-03-26T03:09:49.046630+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:09:49.046630+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339240+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":190,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.048803+0000","last_change":"2024-03-26T03:08:02.573347+0000","last_active":"2024-03-26T03:09:49.048803+0000","last_peered":"2024-03-26T03:09:49.048803+0000","last_clean":"2024-03-26T03:09:49.048803+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:09:49.048803+0000","last_undegraded":"2024-03-26T03:09:49.048803+0000","last_fullsized":"2024-03-26T03:09:49.048803+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:02.573111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:02.573111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":92,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-26T03:09:54.046317+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:09:54.046317+0000","last_peered":"2024-03-26T03:09:54.046317+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:09:54.046317+0000","last_undegraded":"2024-03-26T03:09:54.046317+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":479,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.044721+0000","last_change":"2024-03-26T03:08:04.113828+0000","last_active":"2024-03-26T03:09:49.044721+0000","last_peered":"2024-03-26T03:09:49.044721+0000","last_clean":"2024-03-26T03:09:49.044721+0000","last_became_active":"2024-03-26T03:04:41.966771+0000","last_became_peered":"2024-03-26T03:04:41.966771+0000","last_unstale":"2024-03-26T03:09:49.044721+0000","last_undegraded":"2024-03-26T03:09:49.044721+0000","last_fullsized":"2024-03-26T03:09:49.044721+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:08:04.112516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:08:04.112516+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"239'6146","reported_seq":6180,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.249486+0000","last_change":"2024-03-26T03:08:54.940176+0000","last_active":"2024-03-26T03:09:49.249486+0000","last_peered":"2024-03-26T03:09:49.249486+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.067789+0000","last_became_peered":"2024-03-26T03:08:43.067789+0000","last_unstale":"2024-03-26T03:09:49.249486+0000","last_undegraded":"2024-03-26T03:08:54.937224+0000","last_fullsized":"2024-03-26T03:08:43.062110+0000","mapping_epoch":225,"log_start":"229'1200","ondisk_log_start":"229'1200","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:08.775893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6146,"num_write_kb":393344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6146}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":540,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.045582+0000","last_change":"2024-03-26T03:08:59.116331+0000","last_active":"2024-03-26T03:09:49.045582+0000","last_peered":"2024-03-26T03:09:49.045582+0000","last_clean":"2024-03-26T03:09:49.045582+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:09:49.045582+0000","last_undegraded":"2024-03-26T03:09:49.045582+0000","last_fullsized":"2024-03-26T03:09:49.045582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:08:59.116008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:59.116008+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:22.124498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":445,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.049332+0000","last_change":"2024-03-26T03:08:57.248514+0000","last_active":"2024-03-26T03:09:49.049332+0000","last_peered":"2024-03-26T03:09:49.049332+0000","last_clean":"2024-03-26T03:09:49.049332+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:09:49.049332+0000","last_undegraded":"2024-03-26T03:09:49.049332+0000","last_fullsized":"2024-03-26T03:09:49.049332+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:08:57.248360+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.248360+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.554280+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.046896+0000","last_change":"2024-03-26T03:08:16.459794+0000","last_active":"2024-03-26T03:09:49.046896+0000","last_peered":"2024-03-26T03:09:49.046896+0000","last_clean":"2024-03-26T03:09:49.046896+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:09:49.046896+0000","last_undegraded":"2024-03-26T03:09:49.046896+0000","last_fullsized":"2024-03-26T03:09:49.046896+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:16.459676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:16.459676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":381,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.044455+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:09:49.044455+0000","last_peered":"2024-03-26T03:09:49.044455+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:09:49.044455+0000","last_undegraded":"2024-03-26T03:09:49.044455+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203305+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":466,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.066023+0000","last_change":"2024-03-26T03:06:20.272249+0000","last_active":"2024-03-26T03:09:49.066023+0000","last_peered":"2024-03-26T03:09:49.066023+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:06:20.272249+0000","last_became_peered":"2024-03-26T03:06:20.272249+0000","last_unstale":"2024-03-26T03:09:49.066023+0000","last_undegraded":"2024-03-26T03:09:49.066023+0000","last_fullsized":"2024-03-26T03:06:20.268693+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.317249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"239'6182","reported_seq":6215,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.759345+0000","last_change":"2024-03-26T03:08:54.931125+0000","last_active":"2024-03-26T03:09:49.759345+0000","last_peered":"2024-03-26T03:09:49.759345+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.058021+0000","last_became_peered":"2024-03-26T03:08:43.058021+0000","last_unstale":"2024-03-26T03:09:49.759345+0000","last_undegraded":"2024-03-26T03:08:54.930155+0000","last_fullsized":"2024-03-26T03:08:43.054400+0000","mapping_epoch":225,"log_start":"229'1400","ondisk_log_start":"229'1400","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:58.450523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6182,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6182}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":405,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.040504+0000","last_change":"2024-03-26T03:09:05.145615+0000","last_active":"2024-03-26T03:09:49.040504+0000","last_peered":"2024-03-26T03:09:49.040504+0000","last_clean":"2024-03-26T03:09:49.040504+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:09:49.040504+0000","last_undegraded":"2024-03-26T03:09:49.040504+0000","last_fullsized":"2024-03-26T03:09:49.040504+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:05.145462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:05.145462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:32.012329+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":460,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:54.045455+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:09:54.045455+0000","last_peered":"2024-03-26T03:09:54.045455+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:09:54.045455+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.044157+0000","last_change":"2024-03-26T03:09:19.933816+0000","last_active":"2024-03-26T03:09:54.044157+0000","last_peered":"2024-03-26T03:09:54.044157+0000","last_clean":"2024-03-26T03:09:54.044157+0000","last_became_active":"2024-03-26T02:58:56.919410+0000","last_became_peered":"2024-03-26T02:58:56.919410+0000","last_unstale":"2024-03-26T03:09:54.044157+0000","last_undegraded":"2024-03-26T03:09:54.044157+0000","last_fullsized":"2024-03-26T03:09:54.044157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:09:19.933700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:19.933700+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.582033+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.055451+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:09:49.055451+0000","last_peered":"2024-03-26T03:09:49.055451+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:09:49.055451+0000","last_undegraded":"2024-03-26T03:09:49.055451+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":437,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.045991+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:09:49.045991+0000","last_peered":"2024-03-26T03:09:49.045991+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:09:49.045991+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203338+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":411,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.040394+0000","last_change":"2024-03-26T03:07:59.943234+0000","last_active":"2024-03-26T03:09:49.040394+0000","last_peered":"2024-03-26T03:09:49.040394+0000","last_clean":"2024-03-26T03:09:49.040394+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:09:49.040394+0000","last_undegraded":"2024-03-26T03:09:49.040394+0000","last_fullsized":"2024-03-26T03:09:49.040394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:07:59.943025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:07:59.943025+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.047516+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:09:49.047516+0000","last_peered":"2024-03-26T03:09:49.047516+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:09:49.047516+0000","last_undegraded":"2024-03-26T03:09:49.047516+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":393,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.042110+0000","last_change":"2024-03-26T03:08:08.941553+0000","last_active":"2024-03-26T03:09:49.042110+0000","last_peered":"2024-03-26T03:09:49.042110+0000","last_clean":"2024-03-26T03:09:49.042110+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:09:49.042110+0000","last_undegraded":"2024-03-26T03:09:49.042110+0000","last_fullsized":"2024-03-26T03:09:49.042110+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:08:08.941442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":139,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.056253+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:09:49.056253+0000","last_peered":"2024-03-26T03:09:49.056253+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:09:49.056253+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":405,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057873+0000","last_change":"2024-03-26T03:09:45.324624+0000","last_active":"2024-03-26T03:09:49.057873+0000","last_peered":"2024-03-26T03:09:49.057873+0000","last_clean":"2024-03-26T03:09:49.057873+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:09:49.057873+0000","last_undegraded":"2024-03-26T03:09:49.057873+0000","last_fullsized":"2024-03-26T03:09:49.057873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:45.324180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:45.324180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:52.941801+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":383,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.051500+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:09:49.051500+0000","last_peered":"2024-03-26T03:09:49.051500+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:09:49.051500+0000","last_undegraded":"2024-03-26T03:09:49.051500+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.964922+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":377,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.056900+0000","last_change":"2024-03-26T03:07:54.885532+0000","last_active":"2024-03-26T03:09:49.056900+0000","last_peered":"2024-03-26T03:09:49.056900+0000","last_clean":"2024-03-26T03:09:49.056900+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:09:49.056900+0000","last_undegraded":"2024-03-26T03:09:49.056900+0000","last_fullsized":"2024-03-26T03:09:49.056900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:07:54.885266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:07:54.885266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":338,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.047059+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:09:49.047059+0000","last_peered":"2024-03-26T03:09:49.047059+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:09:49.047059+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":399,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.048145+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:09:49.048145+0000","last_peered":"2024-03-26T03:09:49.048145+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:09:49.048145+0000","last_undegraded":"2024-03-26T03:09:49.048145+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.964973+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":536,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.050205+0000","last_change":"2024-03-26T03:08:10.800546+0000","last_active":"2024-03-26T03:09:49.050205+0000","last_peered":"2024-03-26T03:09:49.050205+0000","last_clean":"2024-03-26T03:09:49.050205+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:09:49.050205+0000","last_undegraded":"2024-03-26T03:09:49.050205+0000","last_fullsized":"2024-03-26T03:09:49.050205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:08:10.800422+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:08:10.800422+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":380,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.045046+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:09:49.045046+0000","last_peered":"2024-03-26T03:09:49.045046+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:09:49.045046+0000","last_undegraded":"2024-03-26T03:09:49.045046+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203292+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":233,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.060658+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:09:49.060658+0000","last_peered":"2024-03-26T03:09:49.060658+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:09:49.060658+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":384,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.045282+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:09:49.045282+0000","last_peered":"2024-03-26T03:09:49.045282+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:09:49.045282+0000","last_undegraded":"2024-03-26T03:09:49.045282+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203279+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":158,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.057108+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:09:49.057108+0000","last_peered":"2024-03-26T03:09:49.057108+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:09:49.057108+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":387,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.044382+0000","last_change":"2024-03-26T03:09:02.967331+0000","last_active":"2024-03-26T03:09:54.044382+0000","last_peered":"2024-03-26T03:09:54.044382+0000","last_clean":"2024-03-26T03:09:54.044382+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:09:54.044382+0000","last_undegraded":"2024-03-26T03:09:54.044382+0000","last_fullsized":"2024-03-26T03:09:54.044382+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:09:02.967216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:02.967216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:32.040525+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"230'15","reported_seq":418,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.049225+0000","last_change":"2024-03-26T03:08:59.371339+0000","last_active":"2024-03-26T03:09:49.049225+0000","last_peered":"2024-03-26T03:09:49.049225+0000","last_clean":"2024-03-26T03:09:49.049225+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:09:49.049225+0000","last_undegraded":"2024-03-26T03:09:49.049225+0000","last_fullsized":"2024-03-26T03:09:49.049225+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'9","last_scrub_stamp":"2024-03-26T03:08:59.371196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:08:59.371196+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:12.951063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":392,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.066673+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:09:49.066673+0000","last_peered":"2024-03-26T03:09:49.066673+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:09:49.066673+0000","last_undegraded":"2024-03-26T03:09:49.066673+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.317262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.045411+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:09:49.045411+0000","last_peered":"2024-03-26T03:09:49.045411+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:09:49.045411+0000","last_undegraded":"2024-03-26T03:09:49.045411+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":396,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.044404+0000","last_change":"2024-03-26T03:09:08.164518+0000","last_active":"2024-03-26T03:09:49.044404+0000","last_peered":"2024-03-26T03:09:49.044404+0000","last_clean":"2024-03-26T03:09:49.044404+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:09:49.044404+0000","last_undegraded":"2024-03-26T03:09:49.044404+0000","last_fullsized":"2024-03-26T03:09:49.044404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:08.164294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:08.164294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:13.644605+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":433,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.043463+0000","last_change":"2024-03-26T03:09:48.867245+0000","last_active":"2024-03-26T03:09:54.043463+0000","last_peered":"2024-03-26T03:09:54.043463+0000","last_clean":"2024-03-26T03:09:54.043463+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:09:54.043463+0000","last_undegraded":"2024-03-26T03:09:54.043463+0000","last_fullsized":"2024-03-26T03:09:54.043463+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:09:48.867101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:48.867101+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:10.634184+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":405,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.049397+0000","last_change":"2024-03-26T03:09:01.395031+0000","last_active":"2024-03-26T03:09:49.049397+0000","last_peered":"2024-03-26T03:09:49.049397+0000","last_clean":"2024-03-26T03:09:49.049397+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:09:49.049397+0000","last_undegraded":"2024-03-26T03:09:49.049397+0000","last_fullsized":"2024-03-26T03:09:49.049397+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:01.394917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:09:01.394917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.403976+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":376,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.049378+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:09:49.049378+0000","last_peered":"2024-03-26T03:09:49.049378+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:09:49.049378+0000","last_undegraded":"2024-03-26T03:09:49.049378+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.045203+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:09:49.045203+0000","last_peered":"2024-03-26T03:09:49.045203+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:09:49.045203+0000","last_undegraded":"2024-03-26T03:09:49.045203+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":121,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.048270+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:09:49.048270+0000","last_peered":"2024-03-26T03:09:49.048270+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:09:49.048270+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":412,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.051765+0000","last_change":"2024-03-26T03:09:18.868778+0000","last_active":"2024-03-26T03:09:49.051765+0000","last_peered":"2024-03-26T03:09:49.051765+0000","last_clean":"2024-03-26T03:09:49.051765+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:09:49.051765+0000","last_undegraded":"2024-03-26T03:09:49.051765+0000","last_fullsized":"2024-03-26T03:09:49.051765+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:09:18.868661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:40.839893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057769+0000","last_change":"2024-03-26T03:09:04.326478+0000","last_active":"2024-03-26T03:09:49.057769+0000","last_peered":"2024-03-26T03:09:49.057769+0000","last_clean":"2024-03-26T03:09:49.057769+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:09:49.057769+0000","last_undegraded":"2024-03-26T03:09:49.057769+0000","last_fullsized":"2024-03-26T03:09:49.057769+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:04.326057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:07:18.501806+0000","last_clean_scrub_stamp":"2024-03-26T03:09:04.326057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:16.327879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":393,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.056629+0000","last_change":"2024-03-26T03:09:48.566148+0000","last_active":"2024-03-26T03:09:49.056629+0000","last_peered":"2024-03-26T03:09:49.056629+0000","last_clean":"2024-03-26T03:09:49.056629+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:09:49.056629+0000","last_undegraded":"2024-03-26T03:09:49.056629+0000","last_fullsized":"2024-03-26T03:09:49.056629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:48.563341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:48.563341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:56.090974+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":325,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.057901+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:09:49.057901+0000","last_peered":"2024-03-26T03:09:49.057901+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:09:49.057901+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.066164+0000","last_change":"2024-03-26T03:08:09.345591+0000","last_active":"2024-03-26T03:09:49.066164+0000","last_peered":"2024-03-26T03:09:49.066164+0000","last_clean":"2024-03-26T03:09:49.066164+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:09:49.066164+0000","last_undegraded":"2024-03-26T03:09:49.066164+0000","last_fullsized":"2024-03-26T03:09:49.066164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:09.345473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:09.345473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.054828+0000","last_change":"2024-03-26T03:09:04.656708+0000","last_active":"2024-03-26T03:09:49.054828+0000","last_peered":"2024-03-26T03:09:49.054828+0000","last_clean":"2024-03-26T03:09:49.054828+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:09:49.054828+0000","last_undegraded":"2024-03-26T03:09:49.054828+0000","last_fullsized":"2024-03-26T03:09:49.054828+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:04.656576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:09:04.656576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.020435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.044770+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:09:49.044770+0000","last_peered":"2024-03-26T03:09:49.044770+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:09:49.044770+0000","last_undegraded":"2024-03-26T03:09:49.044770+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":431,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.046954+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:09:49.046954+0000","last_peered":"2024-03-26T03:09:49.046954+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:09:49.046954+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203330+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":412,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.050668+0000","last_change":"2024-03-26T03:09:08.928841+0000","last_active":"2024-03-26T03:09:49.050668+0000","last_peered":"2024-03-26T03:09:49.050668+0000","last_clean":"2024-03-26T03:09:49.050668+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:09:49.050668+0000","last_undegraded":"2024-03-26T03:09:49.050668+0000","last_fullsized":"2024-03-26T03:09:49.050668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:08.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:09:08.928693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:31.466475+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":540,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.066266+0000","last_change":"2024-03-26T03:09:14.129891+0000","last_active":"2024-03-26T03:09:49.066266+0000","last_peered":"2024-03-26T03:09:49.066266+0000","last_clean":"2024-03-26T03:09:49.066266+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:09:49.066266+0000","last_undegraded":"2024-03-26T03:09:49.066266+0000","last_fullsized":"2024-03-26T03:09:49.066266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:09:14.129763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:14.129763+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.260599+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057309+0000","last_change":"2024-03-26T03:09:19.677230+0000","last_active":"2024-03-26T03:09:49.057309+0000","last_peered":"2024-03-26T03:09:49.057309+0000","last_clean":"2024-03-26T03:09:49.057309+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:09:49.057309+0000","last_undegraded":"2024-03-26T03:09:49.057309+0000","last_fullsized":"2024-03-26T03:09:49.057309+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:09:19.677094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:19.677094+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:25.321802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":445,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.066336+0000","last_change":"2024-03-26T03:09:47.385319+0000","last_active":"2024-03-26T03:09:49.066336+0000","last_peered":"2024-03-26T03:09:49.066336+0000","last_clean":"2024-03-26T03:09:49.066336+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:09:49.066336+0000","last_undegraded":"2024-03-26T03:09:49.066336+0000","last_fullsized":"2024-03-26T03:09:49.066336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:09:47.384690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.384690+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:08.837736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":98,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.057557+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:09:49.057557+0000","last_peered":"2024-03-26T03:09:49.057557+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:09:49.057557+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.317218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":398,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.047011+0000","last_change":"2024-03-26T03:08:17.137900+0000","last_active":"2024-03-26T03:09:49.047011+0000","last_peered":"2024-03-26T03:09:49.047011+0000","last_clean":"2024-03-26T03:09:49.047011+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:09:49.047011+0000","last_undegraded":"2024-03-26T03:09:49.047011+0000","last_fullsized":"2024-03-26T03:09:49.047011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:17.137655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:13.118140+0000","last_clean_scrub_stamp":"2024-03-26T03:08:17.137655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":385,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.049163+0000","last_change":"2024-03-26T03:09:06.593694+0000","last_active":"2024-03-26T03:09:49.049163+0000","last_peered":"2024-03-26T03:09:49.049163+0000","last_clean":"2024-03-26T03:09:49.049163+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:09:49.049163+0000","last_undegraded":"2024-03-26T03:09:49.049163+0000","last_fullsized":"2024-03-26T03:09:49.049163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:06.593565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:06.593565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:25.868873+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":431,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.042376+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:09:49.042376+0000","last_peered":"2024-03-26T03:09:49.042376+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:09:49.042376+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":405,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.058044+0000","last_change":"2024-03-26T03:08:05.414556+0000","last_active":"2024-03-26T03:09:49.058044+0000","last_peered":"2024-03-26T03:09:49.058044+0000","last_clean":"2024-03-26T03:09:49.058044+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:09:49.058044+0000","last_undegraded":"2024-03-26T03:09:49.058044+0000","last_fullsized":"2024-03-26T03:09:49.058044+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:05.414450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:05.414450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.050107+0000","last_change":"2024-03-26T03:09:14.479290+0000","last_active":"2024-03-26T03:09:49.050107+0000","last_peered":"2024-03-26T03:09:49.050107+0000","last_clean":"2024-03-26T03:09:49.050107+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:09:49.050107+0000","last_undegraded":"2024-03-26T03:09:49.050107+0000","last_fullsized":"2024-03-26T03:09:49.050107+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:09:14.479135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:25.344597+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":395,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.045967+0000","last_change":"2024-03-26T03:09:13.875355+0000","last_active":"2024-03-26T03:09:54.045967+0000","last_peered":"2024-03-26T03:09:54.045967+0000","last_clean":"2024-03-26T03:09:54.045967+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:09:54.045967+0000","last_undegraded":"2024-03-26T03:09:54.045967+0000","last_fullsized":"2024-03-26T03:09:54.045967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:13.875126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:13.875126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:37.516506+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":153,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.049968+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:09:49.049968+0000","last_peered":"2024-03-26T03:09:49.049968+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:09:49.049968+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:51.339148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.055576+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:09:49.055576+0000","last_peered":"2024-03-26T03:09:49.055576+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:09:49.055576+0000","last_undegraded":"2024-03-26T03:09:49.055576+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":377,"reported_epoch":240,"state":"active+undersized","last_fresh":"2024-03-26T03:09:54.046843+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:09:54.046843+0000","last_peered":"2024-03-26T03:09:54.046843+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:09:54.046843+0000","last_undegraded":"2024-03-26T03:09:54.046843+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.066462+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:09:49.066462+0000","last_peered":"2024-03-26T03:09:49.066462+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:09:49.066462+0000","last_undegraded":"2024-03-26T03:09:49.066462+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.317275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":444,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:54.046193+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:09:54.046193+0000","last_peered":"2024-03-26T03:09:54.046193+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:09:54.046193+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.056484+0000","last_change":"2024-03-26T03:08:08.825760+0000","last_active":"2024-03-26T03:09:49.056484+0000","last_peered":"2024-03-26T03:09:49.056484+0000","last_clean":"2024-03-26T03:09:49.056484+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:09:49.056484+0000","last_undegraded":"2024-03-26T03:09:49.056484+0000","last_fullsized":"2024-03-26T03:09:49.056484+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:08:08.822728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":404,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.047502+0000","last_change":"2024-03-26T03:09:47.952586+0000","last_active":"2024-03-26T03:09:49.047502+0000","last_peered":"2024-03-26T03:09:49.047502+0000","last_clean":"2024-03-26T03:09:49.047502+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:09:49.047502+0000","last_undegraded":"2024-03-26T03:09:49.047502+0000","last_fullsized":"2024-03-26T03:09:49.047502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:47.952473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.952473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:15.178094+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":393,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.045480+0000","last_change":"2024-03-26T03:08:16.198358+0000","last_active":"2024-03-26T03:09:49.045480+0000","last_peered":"2024-03-26T03:09:49.045480+0000","last_clean":"2024-03-26T03:09:49.045480+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:09:49.045480+0000","last_undegraded":"2024-03-26T03:09:49.045480+0000","last_fullsized":"2024-03-26T03:09:49.045480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:16.198221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:08:16.198221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":406,"reported_epoch":240,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:54.043639+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:09:54.043639+0000","last_peered":"2024-03-26T03:09:54.043639+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:09:54.043639+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":397,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.057195+0000","last_change":"2024-03-26T03:08:19.752549+0000","last_active":"2024-03-26T03:09:49.057195+0000","last_peered":"2024-03-26T03:09:49.057195+0000","last_clean":"2024-03-26T03:09:49.057195+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:09:49.057195+0000","last_undegraded":"2024-03-26T03:09:49.057195+0000","last_fullsized":"2024-03-26T03:09:49.057195+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:19.752469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:08:19.752469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":389,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.056446+0000","last_change":"2024-03-26T03:08:09.778077+0000","last_active":"2024-03-26T03:09:49.056446+0000","last_peered":"2024-03-26T03:09:49.056446+0000","last_clean":"2024-03-26T03:09:49.056446+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:09:49.056446+0000","last_undegraded":"2024-03-26T03:09:49.056446+0000","last_fullsized":"2024-03-26T03:09:49.056446+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:09.777940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:09.777940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":379,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.047623+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:09:49.047623+0000","last_peered":"2024-03-26T03:09:49.047623+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:09:49.047623+0000","last_undegraded":"2024-03-26T03:09:49.047623+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:50.964953+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":431,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.043003+0000","last_change":"2024-03-26T03:07:59.704675+0000","last_active":"2024-03-26T03:09:54.043003+0000","last_peered":"2024-03-26T03:09:54.043003+0000","last_clean":"2024-03-26T03:09:54.043003+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:09:54.043003+0000","last_undegraded":"2024-03-26T03:09:54.043003+0000","last_fullsized":"2024-03-26T03:09:54.043003+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:07:59.704533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:07:59.704533+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.046090+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:09:49.046090+0000","last_peered":"2024-03-26T03:09:49.046090+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:09:49.046090+0000","last_undegraded":"2024-03-26T03:09:49.046090+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.203314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":389,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.046372+0000","last_change":"2024-03-26T03:09:17.407289+0000","last_active":"2024-03-26T03:09:49.046372+0000","last_peered":"2024-03-26T03:09:49.046372+0000","last_clean":"2024-03-26T03:09:49.046372+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:09:49.046372+0000","last_undegraded":"2024-03-26T03:09:49.046372+0000","last_fullsized":"2024-03-26T03:09:49.046372+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:17.407186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:17.407186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:46.296854+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.046106+0000","last_change":"2024-03-26T03:09:53.849576+0000","last_active":"2024-03-26T03:09:54.046106+0000","last_peered":"2024-03-26T03:09:54.046106+0000","last_clean":"2024-03-26T03:09:54.046106+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:09:54.046106+0000","last_undegraded":"2024-03-26T03:09:54.046106+0000","last_fullsized":"2024-03-26T03:09:54.046106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:53.849440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:53.849440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:16.864580+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":201,"reported_epoch":239,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:09:49.040420+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:09:49.040420+0000","last_peered":"2024-03-26T03:09:49.040420+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:09:49.040420+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":406,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-26T03:09:49.046088+0000","last_change":"2024-03-26T03:08:08.179147+0000","last_active":"2024-03-26T03:09:49.046088+0000","last_peered":"2024-03-26T03:09:49.046088+0000","last_clean":"2024-03-26T03:09:49.046088+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:09:49.046088+0000","last_undegraded":"2024-03-26T03:09:49.046088+0000","last_fullsized":"2024-03-26T03:09:49.046088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:08.179038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:08:08.179038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":398,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-26T03:09:54.043333+0000","last_change":"2024-03-26T03:09:09.855016+0000","last_active":"2024-03-26T03:09:54.043333+0000","last_peered":"2024-03-26T03:09:54.043333+0000","last_clean":"2024-03-26T03:09:54.043333+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:09:54.043333+0000","last_undegraded":"2024-03-26T03:09:54.043333+0000","last_fullsized":"2024-03-26T03:09:54.043333+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:09.854936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:09.854936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.697252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":90,"reported_epoch":239,"state":"active+undersized","last_fresh":"2024-03-26T03:09:49.055331+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:09:49.055331+0000","last_peered":"2024-03-26T03:09:49.055331+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:09:49.055331+0000","last_undegraded":"2024-03-26T03:09:49.055331+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:52.538503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6530924544,"num_objects":99654,"num_object_clones":0,"num_object_copies":298962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99654,"num_write_kb":6377856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16316039168,"data_stored":16316039168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81154,"ondisk_log_size":81154,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":28,"acting":28,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":643,"num_read_kb":638,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1410,"num_read_kb":1135,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":940,"ondisk_log_size":940,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103303,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3953324,"kb_used_data":3529252,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":89791828,"statfs":{"total":95995035648,"available":91946831872,"internally_reserved":0,"allocated":3613954048,"data_stored":3613529727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":434239946},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430347,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1514404,"kb_used_data":1172252,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92230748,"statfs":{"total":95995035648,"available":94444285952,"internally_reserved":0,"allocated":1200386048,"data_stored":1199969301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4002840,"kb_used_data":3552396,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":89742312,"statfs":{"total":95995035648,"available":91896127488,"internally_reserved":0,"allocated":3637653504,"data_stored":3637196393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2031300,"kb_used_data":1594744,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91713852,"statfs":{"total":95995035648,"available":93914984448,"internally_reserved":0,"allocated":1633017856,"data_stored":1632580676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411486,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550392,"kb_used_data":1190188,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92194760,"statfs":{"total":95995035648,"available":94407434240,"internally_reserved":0,"allocated":1218752512,"data_stored":1218327950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1537028,"kb_used_data":1257340,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92208124,"statfs":{"total":95995035648,"available":94421118976,"internally_reserved":0,"allocated":1287516160,"data_stored":1287074140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691985,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3954028,"kb_used_data":3539684,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":89791124,"statfs":{"total":95995035648,"available":91946110976,"internally_reserved":0,"allocated":3624636416,"data_stored":3624212099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1638465536,"data_stored":1638465536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3661299712,"data_stored":3661299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1220739072,"data_stored":1220739072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1292697600,"data_stored":1292697600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3638951936,"data_stored":3638951936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1215037440,"data_stored":1215037440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3648847872,"data_stored":3648847872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:09:55.459 INFO:tasks.thrashosds.thrasher:Setting 2.f to [6, 5, 4, 7, 0, 1] 2024-03-26T03:09:55.459 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.f', '6', '5', '4', '7', '0', '1'] 2024-03-26T03:09:55.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 2.f 6 5 4 7 0 1 2024-03-26T03:09:55.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:55.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:55.602 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:55.653 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:55 smithi192 ceph-mon[33110]: pgmap v471: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 905 op/s; 49998/299649 objects degraded (16.686%) 2024-03-26T03:09:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:55 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:09:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:55 smithi192 ceph-mon[33110]: from='client.25291 v1:172.21.15.66:0/1351676498' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:09:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:55 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:09:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:55 smithi192 ceph-mon[33110]: 1.5 scrub starts 2024-03-26T03:09:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:55 smithi192 ceph-mon[33110]: 1.5 scrub ok 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[29666]: pgmap v471: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 905 op/s; 49998/299649 objects degraded (16.686%) 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[29666]: from='client.25291 v1:172.21.15.66:0/1351676498' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[29666]: 1.5 scrub starts 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[29666]: 1.5 scrub ok 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[37920]: pgmap v471: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 905 op/s; 49998/299649 objects degraded (16.686%) 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:09:55.743 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[37920]: from='client.25291 v1:172.21.15.66:0/1351676498' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:09:55.744 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:09:55.744 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[37920]: 1.5 scrub starts 2024-03-26T03:09:55.744 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:55 smithi066 ceph-mon[37920]: 1.5 scrub ok 2024-03-26T03:09:55.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:55.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:55.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 61 16 107710 107694 110.325 87.375 0.00814751 0.00905982 2024-03-26T03:09:55.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:55.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:55.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:56.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:56.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:56.191 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:56.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:56.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:56.249 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:09:56.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:56.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:56.490 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:56.534 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:56.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:56.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:56.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:56.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:56.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:56.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 62 16 109135 109119 109.982 89.0625 0.00540141 0.00908789 2024-03-26T03:09:56.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:56.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:57.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:57.162 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:57.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:57.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:57.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:57.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:57.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:57.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:57.678 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.716 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:57.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:57 smithi192 ceph-mon[33110]: pgmap v472: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 52983/318750 objects degraded (16.622%) 2024-03-26T03:09:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:57 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:09:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:57 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:09:57.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:57.827 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:57 smithi066 ceph-mon[29666]: pgmap v472: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 52983/318750 objects degraded (16.622%) 2024-03-26T03:09:57.827 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:57 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:09:57.827 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:57 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:09:57.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:57 smithi066 ceph-mon[37920]: pgmap v472: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 52983/318750 objects degraded (16.622%) 2024-03-26T03:09:57.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:57 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:09:57.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:57 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:09:57.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:57.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 63 16 110569 110553 109.659 89.625 0.00906806 0.00911478 2024-03-26T03:09:57.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:57.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:57.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:58.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:58.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:58.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:09:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:58.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:09:58.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:09:58.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:09:58.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:58.564 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:58.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:09:58.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:09:58.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:09:58.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:09:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:58 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:09:58.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:09:58.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:09:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:09:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:09:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:09:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:09:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:09:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:09:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:09:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:58 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:09:58.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:58.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:58.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 64 15 111984 111969 109.328 88.5 0.00833234 0.00914227 2024-03-26T03:09:58.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:58.983 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:58.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:59.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:59.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:59.223 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:59.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:09:59.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:59.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:59.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[29666]: pgmap v473: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 52983/318750 objects degraded (16.622%) 2024-03-26T03:09:59.412 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[29666]: osdmap e241: 8 total, 7 up, 8 in 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[37920]: pgmap v473: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 52983/318750 objects degraded (16.622%) 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:59.413 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:09:59 smithi066 ceph-mon[37920]: osdmap e241: 8 total, 7 up, 8 in 2024-03-26T03:09:59.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:09:59.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:09:59.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:09:59.665 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:09:59.666 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:09:59.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:09:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:09:59.698 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:09:59.736 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:09:59.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:09:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:09:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:59 smithi192 ceph-mon[33110]: pgmap v473: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 52983/318750 objects degraded (16.622%) 2024-03-26T03:09:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:59 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:09:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:59 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:09:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:09:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:09:59 smithi192 ceph-mon[33110]: osdmap e241: 8 total, 7 up, 8 in 2024-03-26T03:09:59.777 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:09:59.782 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:09:59.820 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:09:59.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:09:59.859 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:09:59.860 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:09:59.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:09:59.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 65 16 113353 113337 108.961 85.5 0.0118382 0.00917325 2024-03-26T03:09:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:09:59.975 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:09:59.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:00.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:00.119 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:00.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:00.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:00.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.404 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:00.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:00.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:00.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:00.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/184957828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: osd.3 (root=default,host=smithi066) is down 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: [WRN] PG_DEGRADED: Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 1.4 is stuck undersized for 3m, current state active+undersized, last acting [7,5] 2024-03-26T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 1.6 is stuck undersized for 3m, current state active+undersized, last acting [6,0] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 1.7 is stuck undersized for 3m, current state active+undersized, last acting [6,7] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 1.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 1.9 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.1 is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.9 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.a is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.d is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.e is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.12 is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.13 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 2.14 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 3.1 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 3.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,6] 2024-03-26T03:10:00.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.4 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,2] 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.9 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,5] 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.c is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.14 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,7] 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:10:00.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/184957828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: osd.3 (root=default,host=smithi066) is down 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: [WRN] PG_DEGRADED: Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 1.4 is stuck undersized for 3m, current state active+undersized, last acting [7,5] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 1.6 is stuck undersized for 3m, current state active+undersized, last acting [6,0] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 1.7 is stuck undersized for 3m, current state active+undersized, last acting [6,7] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 1.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 1.9 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.1 is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.9 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.a is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.d is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.e is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-03-26T03:10:00.660 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.12 is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.13 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 2.14 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 3.1 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 3.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,6] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.4 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,2] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.9 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,5] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.c is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.14 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,7] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,5] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.16 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-26T03:10:00.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 4.17 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.1 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.a is stuck undersized for 3m, current state active+undersized, last acting [2,4] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.12 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.13 is stuck undersized for 3m, current state active+undersized, last acting [0,1] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.14 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 5.17 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.0 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.2 is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.5 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-03-26T03:10:00.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.6 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.8 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.b is stuck undersized for 3m, current state active+undersized, last acting [7,1] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.c is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.f is stuck undersized for 3m, current state active+undersized, last acting [2,4] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 6.16 is stuck undersized for 3m, current state active+undersized, last acting [0,7] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.1 is stuck undersized for 75s, current state active+undersized+degraded, last acting [6,7] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.2 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.3 is stuck undersized for 75s, current state active+undersized+degraded, last acting [7,1] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.4 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.7 is stuck undersized for 75s, current state active+undersized+degraded, last acting [7,5] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.8 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,5] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.c is stuck undersized for 75s, current state active+undersized+degraded, last acting [4,7] 2024-03-26T03:10:00.663 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[29666]: pg 9.f is stuck undersized for 75s, current state active+undersized+degraded, last acting [5,1] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,5] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.16 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 4.17 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.1 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.a is stuck undersized for 3m, current state active+undersized, last acting [2,4] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.12 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.13 is stuck undersized for 3m, current state active+undersized, last acting [0,1] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.14 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 5.17 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.0 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-03-26T03:10:00.664 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.2 is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.5 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.6 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.8 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.b is stuck undersized for 3m, current state active+undersized, last acting [7,1] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.c is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.f is stuck undersized for 3m, current state active+undersized, last acting [2,4] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 6.16 is stuck undersized for 3m, current state active+undersized, last acting [0,7] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.1 is stuck undersized for 75s, current state active+undersized+degraded, last acting [6,7] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.2 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.3 is stuck undersized for 75s, current state active+undersized+degraded, last acting [7,1] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.4 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.7 is stuck undersized for 75s, current state active+undersized+degraded, last acting [7,5] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.8 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,5] 2024-03-26T03:10:00.665 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.c is stuck undersized for 75s, current state active+undersized+degraded, last acting [4,7] 2024-03-26T03:10:00.666 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:00 smithi066 ceph-mon[37920]: pg 9.f is stuck undersized for 75s, current state active+undersized+degraded, last acting [5,1] 2024-03-26T03:10:00.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:00.724 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/184957828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized 2024-03-26T03:10:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: [WRN] OSD_DOWN: 1 osds down 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: osd.3 (root=default,host=smithi066) is down 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: [WRN] PG_DEGRADED: Degraded data redundancy: 52983/318750 objects degraded (16.622%), 30 pgs degraded, 68 pgs undersized 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 1.4 is stuck undersized for 3m, current state active+undersized, last acting [7,5] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 1.6 is stuck undersized for 3m, current state active+undersized, last acting [6,0] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 1.7 is stuck undersized for 3m, current state active+undersized, last acting [6,7] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 1.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,5] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 1.9 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.1 is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.9 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.a is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.d is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.e is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.12 is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-03-26T03:10:00.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.13 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 2.14 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 3.1 is stuck undersized for 3m, current state active+undersized, last acting [0,4] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 3.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,6] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,0] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.4 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,2] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.9 is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,5] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.c is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,6] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.14 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,7] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,5] 2024-03-26T03:10:00.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.16 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 4.17 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.0 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.1 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.a is stuck undersized for 3m, current state active+undersized, last acting [2,4] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.12 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.13 is stuck undersized for 3m, current state active+undersized, last acting [0,1] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.14 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 5.17 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.0 is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.2 is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.5 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-03-26T03:10:00.741 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.6 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [5,4] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.8 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.b is stuck undersized for 3m, current state active+undersized, last acting [7,1] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.c is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.f is stuck undersized for 3m, current state active+undersized, last acting [2,4] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 6.16 is stuck undersized for 3m, current state active+undersized, last acting [0,7] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.1 is stuck undersized for 75s, current state active+undersized+degraded, last acting [6,7] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.2 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.3 is stuck undersized for 75s, current state active+undersized+degraded, last acting [7,1] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.4 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,4] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.7 is stuck undersized for 75s, current state active+undersized+degraded, last acting [7,5] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.8 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,5] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.c is stuck undersized for 75s, current state active+undersized+degraded, last acting [4,7] 2024-03-26T03:10:00.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:00 smithi192 ceph-mon[33110]: pg 9.f is stuck undersized for 75s, current state active+undersized+degraded, last acting [5,1] 2024-03-26T03:10:00.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:00.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:00.925 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:00.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 66 16 114743 114727 108.627 86.875 0.00667223 0.00920149 2024-03-26T03:10:00.967 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:00.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:01.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:01.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:10:01.210 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:10:01.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:01.264 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:01.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:01.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:01.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:01.440 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:10:01.502 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:01.544 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:01.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:01.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:01 smithi192 ceph-mon[33110]: pgmap v475: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 56069/334485 objects degraded (16.763%) 2024-03-26T03:10:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:01 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:10:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:01 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:10:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:01 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3809760227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:10:01.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:01.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[29666]: pgmap v475: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 56069/334485 objects degraded (16.763%) 2024-03-26T03:10:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:10:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:10:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3809760227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:10:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[37920]: pgmap v475: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 56069/334485 objects degraded (16.763%) 2024-03-26T03:10:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:10:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:10:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:01 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3809760227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:10:01.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:01.931 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 67 16 116144 116128 108.312 87.5625 0.00698179 0.00922798 2024-03-26T03:10:01.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:01.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:02.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:02.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:02.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:02.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:02.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:02.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:02.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:02.509 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:02.611 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:02.666 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3809760227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[29666]: osdmap e242: 8 total, 7 up, 8 in 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:10:02.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:10:02.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:10:02.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3809760227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:10:02.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:02 smithi066 ceph-mon[37920]: osdmap e242: 8 total, 7 up, 8 in 2024-03-26T03:10:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:02 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:10:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:02 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:10:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:02 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:10:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:02 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3809760227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:10:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:02 smithi192 ceph-mon[33110]: osdmap e242: 8 total, 7 up, 8 in 2024-03-26T03:10:02.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:02.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:02.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:02.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 68 16 117570 117554 108.03 89.125 0.00698172 0.00925238 2024-03-26T03:10:02.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:02.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:03.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:03.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:03.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:03.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:10:03.217 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:03.320 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:03.365 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:03.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:03.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:03.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[29666]: pgmap v477: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 56500/339678 objects degraded (16.633%) 2024-03-26T03:10:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:10:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[37920]: pgmap v477: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 56500/339678 objects degraded (16.633%) 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:10:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:10:03.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:03 smithi192 ceph-mon[33110]: pgmap v477: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 56500/339678 objects degraded (16.633%) 2024-03-26T03:10:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:03 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:10:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:03 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:10:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:03 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:10:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:03 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:10:03.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:03.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:03.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:03.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:03.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:03.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 69 16 118914 118898 107.681 84 0.0185406 0.00928229 2024-03-26T03:10:03.972 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:10:03.972 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:10:03.973 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:10:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:10:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:10:04.011 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:04.054 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:04.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:04.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:04.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:04.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:04.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:04.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:04.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:04.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:04 smithi066 ceph-mon[29666]: 9.0 scrub starts 2024-03-26T03:10:04.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 56500/339678 objects degraded (16.633%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:04.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:04 smithi066 ceph-mon[29666]: osdmap e243: 8 total, 7 up, 8 in 2024-03-26T03:10:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:04 smithi066 ceph-mon[37920]: 9.0 scrub starts 2024-03-26T03:10:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 56500/339678 objects degraded (16.633%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:04 smithi066 ceph-mon[37920]: osdmap e243: 8 total, 7 up, 8 in 2024-03-26T03:10:04.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:04.649 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:10:04.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:04 smithi192 ceph-mon[33110]: 9.0 scrub starts 2024-03-26T03:10:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 56500/339678 objects degraded (16.633%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:04 smithi192 ceph-mon[33110]: osdmap e243: 8 total, 7 up, 8 in 2024-03-26T03:10:04.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:04.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:04.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:04.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:04.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 70 16 120244 120228 107.331 83.125 0.0151093 0.00931239 2024-03-26T03:10:04.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:05.034 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:05.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:05.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:05.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:05.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:05.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:05.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:05.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:05.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:05.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:05 smithi192 ceph-mon[33110]: pgmap v479: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.16k op/s; 56500/339678 objects degraded (16.633%) 2024-03-26T03:10:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:05 smithi192 ceph-mon[33110]: 1.1 deep-scrub starts 2024-03-26T03:10:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:05 smithi192 ceph-mon[33110]: 1.1 deep-scrub ok 2024-03-26T03:10:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:05 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:10:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:05 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[29666]: pgmap v479: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.16k op/s; 56500/339678 objects degraded (16.633%) 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[29666]: 1.1 deep-scrub starts 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[29666]: 1.1 deep-scrub ok 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[37920]: pgmap v479: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.16k op/s; 56500/339678 objects degraded (16.633%) 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[37920]: 1.1 deep-scrub starts 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[37920]: 1.1 deep-scrub ok 2024-03-26T03:10:05.790 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:10:05.791 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:05 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:10:05.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:05.838 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:10:05.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:05.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 71 15 121499 121484 106.924 78.5 0.0105366 0.00934801 2024-03-26T03:10:05.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:05.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:05.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:06.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:06.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:06.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:06.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:06.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:06.401 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:06.502 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:06.513 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:06.519 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.558 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:06.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:06.657 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:06 smithi066 ceph-mon[37920]: 9.0 scrub ok 2024-03-26T03:10:06.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:06 smithi066 ceph-mon[29666]: 9.0 scrub ok 2024-03-26T03:10:06.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:06.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:06 smithi192 ceph-mon[33110]: 9.0 scrub ok 2024-03-26T03:10:06.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:06.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:06.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 72 16 122804 122788 106.571 81.5 0.00688595 0.00937913 2024-03-26T03:10:06.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:06.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:06.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:07.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:07.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:07.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:07.251 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:07.353 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:07.385 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:07.472 INFO:teuthology.orchestra.run.smithi066.stderr:set 2.f pg_upmap_items mapping to [6->5,4->7,0->1] 2024-03-26T03:10:07.487 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:07.521 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:07.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: pgmap v480: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 59855/360336 objects degraded (16.611%) 2024-03-26T03:10:07.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2806492304' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]: dispatch 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[29666]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]: dispatch 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: pgmap v480: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 59855/360336 objects degraded (16.611%) 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2806492304' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]: dispatch 2024-03-26T03:10:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:07 smithi066 ceph-mon[37920]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]: dispatch 2024-03-26T03:10:07.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:07.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: pgmap v480: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 59855/360336 objects degraded (16.611%) 2024-03-26T03:10:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:10:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:10:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:10:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:10:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2806492304' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]: dispatch 2024-03-26T03:10:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:07 smithi192 ceph-mon[33110]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]: dispatch 2024-03-26T03:10:07.763 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:07.798 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:07.900 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:07.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 73 16 124174 124158 106.284 85.625 0.0105857 0.00940442 2024-03-26T03:10:07.938 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:07.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:08.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:08.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:08.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:08.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:08.319 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:08.353 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:08.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:08.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:08.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:08.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:08.728 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:08 smithi192 ceph-mon[33110]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]': finished 2024-03-26T03:10:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:08 smithi192 ceph-mon[33110]: osdmap e244: 8 total, 7 up, 8 in 2024-03-26T03:10:08.765 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:08.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:08 smithi066 ceph-mon[29666]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]': finished 2024-03-26T03:10:08.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:08 smithi066 ceph-mon[29666]: osdmap e244: 8 total, 7 up, 8 in 2024-03-26T03:10:08.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:08 smithi066 ceph-mon[37920]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.f", "id": [6, 5, 4, 7, 0, 1]}]': finished 2024-03-26T03:10:08.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:08 smithi066 ceph-mon[37920]: osdmap e244: 8 total, 7 up, 8 in 2024-03-26T03:10:08.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:08.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:08.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:08.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 74 16 125423 125407 105.903 78.0625 0.00889362 0.00943826 2024-03-26T03:10:09.005 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:09.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:09.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:09.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:09.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:09.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:09.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:09.423 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:09.462 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:09.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:09.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:09.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:09.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:10:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-mon[33110]: pgmap v482: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 758 B/s rd, 64 MiB/s wr, 1.02k op/s; 59855/360336 objects degraded (16.611%) 2024-03-26T03:10:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-mon[33110]: osdmap e245: 8 total, 7 up, 8 in 2024-03-26T03:10:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 59855/360336 objects degraded (16.611%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-mon[33110]: osdmap e246: 8 total, 7 up, 8 in 2024-03-26T03:10:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:09 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:09.746 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:09.848 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:09.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[29666]: pgmap v482: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 758 B/s rd, 64 MiB/s wr, 1.02k op/s; 59855/360336 objects degraded (16.611%) 2024-03-26T03:10:09.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[29666]: osdmap e245: 8 total, 7 up, 8 in 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 59855/360336 objects degraded (16.611%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[29666]: osdmap e246: 8 total, 7 up, 8 in 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[37920]: pgmap v482: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 758 B/s rd, 64 MiB/s wr, 1.02k op/s; 59855/360336 objects degraded (16.611%) 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[37920]: osdmap e245: 8 total, 7 up, 8 in 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 59855/360336 objects degraded (16.611%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[37920]: osdmap e246: 8 total, 7 up, 8 in 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:09 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:09.882 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:09.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:09.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 75 16 126789 126773 105.629 85.375 0.00338088 0.00946095 2024-03-26T03:10:09.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:10.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:10.122 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:10.167 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:10.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:10.306 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:10.408 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.449 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:10.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:10.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:10.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:10.692 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:10.730 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:10.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:10.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:10.872 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:10.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:10.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 76 16 128193 128177 105.394 87.75 0.0294054 0.00948358 2024-03-26T03:10:10.975 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.017 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:11.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:11.118 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:11.154 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:11.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:11.298 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:11.400 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:11.438 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:11.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:11.568 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:11.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:11.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:11.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:11 smithi192 ceph-mon[33110]: pgmap v485: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 123 MiB/s wr, 1.96k op/s; 62674/375033 objects degraded (16.712%); 87 B/s, 0 objects/s recovering 2024-03-26T03:10:11.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:11 smithi192 ceph-mon[33110]: 9.6 scrub starts 2024-03-26T03:10:11.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:11.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:11.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:11.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:11 smithi066 ceph-mon[29666]: pgmap v485: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 123 MiB/s wr, 1.96k op/s; 62674/375033 objects degraded (16.712%); 87 B/s, 0 objects/s recovering 2024-03-26T03:10:11.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:11 smithi066 ceph-mon[29666]: 9.6 scrub starts 2024-03-26T03:10:11.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:11.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:11 smithi066 ceph-mon[37920]: pgmap v485: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 123 MiB/s wr, 1.96k op/s; 62674/375033 objects degraded (16.712%); 87 B/s, 0 objects/s recovering 2024-03-26T03:10:11.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:11 smithi066 ceph-mon[37920]: 9.6 scrub starts 2024-03-26T03:10:11.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 77 15 129467 129452 105.06 79.6875 0.0162236 0.00951411 2024-03-26T03:10:11.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:11.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:11.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:12.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:12.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:12.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:12.272 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:12.373 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.420 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:12.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:12.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:12.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:12.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:12.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:12 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:10:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:12 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:10:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:12 smithi192 ceph-mon[33110]: 9.a scrub starts 2024-03-26T03:10:12.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:12.826 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:12.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:12 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:10:12.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:12 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:10:12.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:12 smithi066 ceph-mon[29666]: 9.a scrub starts 2024-03-26T03:10:12.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:12 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:10:12.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:12 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:10:12.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:12 smithi066 ceph-mon[37920]: 9.a scrub starts 2024-03-26T03:10:12.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:12.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 78 16 130655 130639 104.664 74.1875 0.00769028 0.00954967 2024-03-26T03:10:12.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:12.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:13.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:13.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:13.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:13.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:13.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:13.388 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:13.490 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:13.530 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:13.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:13.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:13.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:13.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:13.816 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:13.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:13 smithi066 ceph-mon[29666]: pgmap v486: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 63072/379977 objects degraded (16.599%); 87 B/s, 0 objects/s recovering 2024-03-26T03:10:13.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:13 smithi066 ceph-mon[29666]: 9.6 scrub ok 2024-03-26T03:10:13.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:13 smithi066 ceph-mon[37920]: pgmap v486: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 63072/379977 objects degraded (16.599%); 87 B/s, 0 objects/s recovering 2024-03-26T03:10:13.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:13 smithi066 ceph-mon[37920]: 9.6 scrub ok 2024-03-26T03:10:13.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:13.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 79 16 131918 131902 104.338 78.9375 0.00793596 0.00957933 2024-03-26T03:10:13.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:13.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:13 smithi192 ceph-mon[33110]: pgmap v486: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 63072/379977 objects degraded (16.599%); 87 B/s, 0 objects/s recovering 2024-03-26T03:10:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:13 smithi192 ceph-mon[33110]: 9.6 scrub ok 2024-03-26T03:10:14.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:14.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:10:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:10:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:10:14.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:14.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:14.333 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:14.375 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:14.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:14.514 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:14.616 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:14.654 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:14.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:14.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:10:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:10:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 63072/379977 objects degraded (16.599%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[29666]: pgmap v487: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 761 B/s rd, 65 MiB/s wr, 1.04k op/s; 63522/381327 objects degraded (16.658%); 78 B/s, 0 objects/s recovering 2024-03-26T03:10:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:10:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:10:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 63072/379977 objects degraded (16.599%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:14 smithi066 ceph-mon[37920]: pgmap v487: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 761 B/s rd, 65 MiB/s wr, 1.04k op/s; 63522/381327 objects degraded (16.658%); 78 B/s, 0 objects/s recovering 2024-03-26T03:10:14.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:14.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:14.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:14.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:10:14.933042+0000 min lat: 0.00130673 max lat: 0.137828 avg lat: 0.00960547 2024-03-26T03:10:14.932 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:10:14.933 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 80 16 133229 133213 104.058 81.9375 0.0039461 0.00960547 2024-03-26T03:10:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:14 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:10:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:14 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:10:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 63072/379977 objects degraded (16.599%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:14 smithi192 ceph-mon[33110]: pgmap v487: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 7.7 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 761 B/s rd, 65 MiB/s wr, 1.04k op/s; 63522/381327 objects degraded (16.658%); 78 B/s, 0 objects/s recovering 2024-03-26T03:10:15.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:15.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:15.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:15.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:15.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:15.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:15.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:15.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:15.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:15.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:15.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:15.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:15.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[29666]: 9.a scrub ok 2024-03-26T03:10:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:10:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:10:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:10:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:10:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[37920]: 9.a scrub ok 2024-03-26T03:10:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:10:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:10:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:10:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:15 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:10:15.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:15.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:15.933 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 81 16 134619 134603 103.846 86.875 0.0115821 0.00962544 2024-03-26T03:10:15.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:15.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:15 smithi192 ceph-mon[33110]: 9.a scrub ok 2024-03-26T03:10:15.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:15 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:10:15.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:15 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:10:15.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:15 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:10:15.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:15 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:10:16.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:16.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:16.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:16.265 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:16.353 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:16.455 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:16.527 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:16.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:16.676 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:16.738 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:16 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:10:16.738 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:16 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:10:16.738 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:16 smithi066 ceph-mon[29666]: pgmap v488: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 66332/399381 objects degraded (16.609%); 65 B/s, 0 objects/s recovering 2024-03-26T03:10:16.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:16 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:10:16.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:16 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:10:16.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:16 smithi066 ceph-mon[37920]: pgmap v488: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 66332/399381 objects degraded (16.609%); 65 B/s, 0 objects/s recovering 2024-03-26T03:10:16.778 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:16.821 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:16.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:16.933 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 82 16 136003 135987 103.634 86.5 0.0081342 0.00964505 2024-03-26T03:10:16.964 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:16.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:16 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:10:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:16 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:10:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:16 smithi192 ceph-mon[33110]: pgmap v488: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 66332/399381 objects degraded (16.609%); 65 B/s, 0 objects/s recovering 2024-03-26T03:10:17.063 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:10:17.066 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:17.108 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:10:17.109 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:17.143 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:10:17.182 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:10:17.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:17.217 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:10:17.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:17.246 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:10:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:17.287 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:10:17.287 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:10:17.287 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:10:17.288 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:10:17.288 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:10:17.288 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:10:17.288 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:10:17.288 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:10:17.288 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:10:17.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:17.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:17.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:17.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:17.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:17 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:10:17.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:17 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:10:17.593 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1410135139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:17.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:17 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:10:17.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:17 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:10:17.594 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1410135139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:17.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:17.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:17.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:17.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:17.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:17.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:17.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:17.933 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 83 15 137361 137346 103.408 84.9375 0.00847662 0.00966584 2024-03-26T03:10:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:17 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:10:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:17 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:10:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1410135139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:18.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:18.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:18.150 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:10:18.150 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:10:18.150 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-26T03:10:18.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:10:18.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:18.257 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:18.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:18.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:18.460 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:138022 9.e e246/225 9:71bb8a1f:::benchmark_data_smithi066_125102_object138021:head v 246'8596, mlcod=246'8593)", 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:18.452847+0000", 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.003678209, 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0037122840000000002, 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:18.461 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.452847+0000", 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.452847+0000", 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.452849+0000", 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.226e-06 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.462 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.452880+0000", 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.1504999999999997e-05 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.452902+0000", 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.207e-05 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.452910+0000", 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5970000000000003e-06 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.463 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454289+0000", 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001379103 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454309+0000", 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0509999999999998e-05 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.15360.0:138001 9.0 e246/225 ondisk, result = 0)", 2024-03-26T03:10:18.464 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:18.453813+0000", 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.002712796, 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028177229999999998, 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453813+0000", 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:18.465 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453813+0000", 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453814+0000", 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.26e-06 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453819+0000", 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9899999999999997e-06 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.466 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453833+0000", 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3791999999999999e-05 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453840+0000", 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8950000000000001e-06 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453966+0000", 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000126463 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.467 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453974+0000", 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3299999999999999e-06 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.15360.0:138004 9.0 e246/225)", 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:18.453855+0000", 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0026700180000000001, 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0033904590000000002, 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:18.468 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453855+0000", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453855+0000", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453856+0000", 2024-03-26T03:10:18.469 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0400000000000002e-07 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453858+0000", 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5119999999999998e-06 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453863+0000", 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8289999999999998e-06 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:18.470 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.453867+0000", 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2379999999999999e-06 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.457053+0000", 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031858289999999998 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.457061+0000", 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7209999999999994e-06 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.471 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.457246+0000", 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000184926 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:138024 9.f e246/225 9:f79afbe2:::benchmark_data_smithi066_125102_object138023:head v 246'8584, mlcod=246'8582)", 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:18.454034+0000", 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.002491629, 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0035786669999999998, 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:18.472 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454034+0000", 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454034+0000", 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454035+0000", 2024-03-26T03:10:18.473 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1689999999999999e-06 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454101+0000", 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6118999999999995e-05 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454109+0000", 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8809999999999996e-06 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:18.474 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454116+0000", 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7510000000000002e-06 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454170+0000", 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3919999999999999e-05 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.454194+0000", 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3949999999999999e-05 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.475 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:138025 9.e e246/225 9:7825117b:::benchmark_data_smithi066_125102_object138024:head v 246'8597, mlcod=246'8593)", 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:18.456008+0000", 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.00051717099999999997, 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001749298, 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:18.476 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456008+0000", 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456008+0000", 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456009+0000", 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.144e-06 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:18.477 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456046+0000", 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6399999999999997e-05 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456061+0000", 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5124e-05 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456069+0000", 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1329999999999999e-06 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.478 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456636+0000", 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00056702800000000002 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.456659+0000", 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2881e-05 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:138020 9.e e246/225 9:74baf348:::benchmark_data_smithi066_125102_object138019:head v 246'8595, mlcod=246'8593)", 2024-03-26T03:10:18.479 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:18.450916+0000", 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0056094339999999999, 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069052009999999997, 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.450916+0000", 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.450916+0000", 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:18.480 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.450917+0000", 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.415e-06 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.450955+0000", 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7855000000000003e-05 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.450971+0000", 2024-03-26T03:10:18.481 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5688e-05 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.450980+0000", 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5949999999999999e-06 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.451921+0000", 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00094087700000000001 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:18.482 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:18.451940+0000", 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9354000000000001e-05 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 6 2024-03-26T03:10:18.483 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:10:18.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:18.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:18 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:10:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:18 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:10:18.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:18 smithi066 ceph-mon[29666]: pgmap v489: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 947 B/s rd, 84 MiB/s wr, 1.34k op/s; 66332/399381 objects degraded (16.609%); 54 B/s, 0 objects/s recovering 2024-03-26T03:10:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:18 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:10:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:18 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:10:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:18 smithi066 ceph-mon[37920]: pgmap v489: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 947 B/s rd, 84 MiB/s wr, 1.34k op/s; 66332/399381 objects degraded (16.609%); 54 B/s, 0 objects/s recovering 2024-03-26T03:10:18.655 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:18.696 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:18.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:18.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:18.933 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 84 16 138681 138665 103.159 82.4375 0.00526586 0.0096895 2024-03-26T03:10:18.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:18.976 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:18.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:18 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:10:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:18 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:10:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:18 smithi192 ceph-mon[33110]: pgmap v489: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 947 B/s rd, 84 MiB/s wr, 1.34k op/s; 66332/399381 objects degraded (16.609%); 54 B/s, 0 objects/s recovering 2024-03-26T03:10:19.078 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:19.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:19.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:19.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:19.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:19.393 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:19.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:19.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:10:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[29666]: 9.e scrub starts 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 66332/399381 objects degraded (16.609%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[37920]: 9.e scrub starts 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 66332/399381 objects degraded (16.609%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:19 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:10:19.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:19.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:19.789 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:10:19.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:10:19.843 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:19.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 85 16 139829 139813 102.789 71.75 0.00514296 0.009724 2024-03-26T03:10:19.945 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:19.983 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:19.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:19.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:19 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:10:19.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:19 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:10:19.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:19 smithi192 ceph-mon[33110]: 9.e scrub starts 2024-03-26T03:10:19.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 66332/399381 objects degraded (16.609%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:19 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:10:19.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:10:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:10:20.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:20.120 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:20.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:20.263 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:20.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:20.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:20.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:20.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:20.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: 5.11 deep-scrub starts 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: 5.11 deep-scrub ok 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: pgmap v490: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 69286/414705 objects degraded (16.707%); 47 B/s, 0 objects/s recovering 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi192 to 3248M 2024-03-26T03:10:20.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:20.611 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:10:20.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: 5.11 deep-scrub starts 2024-03-26T03:10:20.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: 5.11 deep-scrub ok 2024-03-26T03:10:20.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:20.612 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: pgmap v490: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 69286/414705 objects degraded (16.707%); 47 B/s, 0 objects/s recovering 2024-03-26T03:10:20.613 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.613 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.613 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.613 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.613 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi192 to 3248M 2024-03-26T03:10:20.613 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:20 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:20.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:20.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:20.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.844 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:20.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:20.922 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:20.933 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 86 16 141004 140988 102.448 73.4375 0.0191808 0.00975663 2024-03-26T03:10:20.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:20.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:20.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: 5.11 deep-scrub starts 2024-03-26T03:10:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: 5.11 deep-scrub ok 2024-03-26T03:10:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: pgmap v490: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 69286/414705 objects degraded (16.707%); 47 B/s, 0 objects/s recovering 2024-03-26T03:10:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi192 to 3248M 2024-03-26T03:10:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:20 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:21.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:21.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:21.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:21.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:21.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:21.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:21.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:21.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:21.684 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:21.775 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: 5.1e deep-scrub starts 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: 5.1e deep-scrub ok 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: Adjusting osd_memory_target on smithi066 to 3248M 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3131314589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:10:21.776 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[29666]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: 5.1e deep-scrub starts 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: 5.1e deep-scrub ok 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: Adjusting osd_memory_target on smithi066 to 3248M 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:10:21.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:10:21.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:10:21.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3131314589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:10:21.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:21 smithi066 ceph-mon[37920]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:10:21.778 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:10:21.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:21.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:21.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:10:21.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:21.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 87 16 142339 142323 102.229 83.4375 0.0137327 0.00977765 2024-03-26T03:10:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:21.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:21.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: 5.1e deep-scrub starts 2024-03-26T03:10:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: 5.1e deep-scrub ok 2024-03-26T03:10:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: Adjusting osd_memory_target on smithi066 to 3248M 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3131314589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:10:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:21 smithi192 ceph-mon[33110]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:10:22.059 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:22.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:22.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:22.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:22.354 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:22.397 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:22.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:22.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:22.641 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:22.686 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:22.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:22.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:22.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:10:22.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: 9.e scrub ok 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: osdmap e247: 8 total, 7 up, 8 in 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[29666]: pgmap v492: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 920 B/s rd, 82 MiB/s wr, 1.31k op/s; 69693/419478 objects degraded (16.614%) 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: 9.e scrub ok 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: osdmap e247: 8 total, 7 up, 8 in 2024-03-26T03:10:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:22 smithi066 ceph-mon[37920]: pgmap v492: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 920 B/s rd, 82 MiB/s wr, 1.31k op/s; 69693/419478 objects degraded (16.614%) 2024-03-26T03:10:22.926 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:22.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 88 16 143676 143660 102.017 83.5625 0.00813105 0.00979766 2024-03-26T03:10:22.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:22.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:10:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:10:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: 9.e scrub ok 2024-03-26T03:10:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:10:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: osdmap e247: 8 total, 7 up, 8 in 2024-03-26T03:10:22.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:22 smithi192 ceph-mon[33110]: pgmap v492: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 920 B/s rd, 82 MiB/s wr, 1.31k op/s; 69693/419478 objects degraded (16.614%) 2024-03-26T03:10:23.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:23.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:23.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:23.248 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:23.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:23.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:23.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:23.523 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:23.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:23.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:23.767 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:23.812 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:23.869 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:23.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:23.934 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:23.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 89 16 144924 144908 101.747 78 0.0222335 0.00982372 2024-03-26T03:10:23.966 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:23.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:24.010 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:10:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:10:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:10:24.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.125 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:24.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:24.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:24.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:24.311 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 69693/419478 objects degraded (16.614%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:24.312 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:24 smithi066 ceph-mon[29666]: osdmap e248: 8 total, 7 up, 8 in 2024-03-26T03:10:24.313 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 69693/419478 objects degraded (16.614%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:24.313 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:24 smithi066 ceph-mon[37920]: osdmap e248: 8 total, 7 up, 8 in 2024-03-26T03:10:24.357 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:10:24.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:24.367 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:10:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:24.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 69693/419478 objects degraded (16.614%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:24 smithi192 ceph-mon[33110]: osdmap e248: 8 total, 7 up, 8 in 2024-03-26T03:10:24.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:24.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:24.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:24.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:24.779 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:24.816 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:24.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:24.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total time run: 90.0067 2024-03-26T03:10:24.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total writes made: 146298 2024-03-26T03:10:24.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Write size: 65536 2024-03-26T03:10:24.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object size: 65536 2024-03-26T03:10:24.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Bandwidth (MB/sec): 101.588 2024-03-26T03:10:24.934 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Bandwidth: 34.7961 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max bandwidth (MB/sec): 254.562 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min bandwidth (MB/sec): 71.75 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average IOPS: 1625 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev IOPS: 556.737 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max IOPS: 4073 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min IOPS: 1148 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average Latency(s): 0.00983986 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Latency(s): 0.00704651 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max latency(s): 0.137828 2024-03-26T03:10:24.935 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min latency(s): 0.00130673 2024-03-26T03:10:24.939 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:10:24.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:24.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:25.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:25.086 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:25.188 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:25.200 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:10:25.227 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:25.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[29666]: pgmap v494: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 888 op/s; 70105/420714 objects degraded (16.663%) 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[29666]: from='client.15477 v1:172.21.15.66:0/1419626628' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[37920]: pgmap v494: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 888 op/s; 70105/420714 objects degraded (16.663%) 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:25 smithi066 ceph-mon[37920]: from='client.15477 v1:172.21.15.66:0/1419626628' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:10:25.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:25.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:25.517 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:25.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:25.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:25 smithi192 ceph-mon[33110]: pgmap v494: 158 pgs: 1 active+clean+scrubbing, 30 active+undersized+degraded, 38 active+undersized, 89 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 888 op/s; 70105/420714 objects degraded (16.663%) 2024-03-26T03:10:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:10:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:25 smithi192 ceph-mon[33110]: from='client.15477 v1:172.21.15.66:0/1419626628' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:10:25.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:25.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:25.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:25.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:25.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:26.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:26.066 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:26.126 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-03-26T03:10:24.196293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9175952206,"num_objects":140238,"num_object_clones":0,"num_object_copies":420714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":70105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140238,"num_whiteouts":0,"num_read":2125,"num_read_kb":1854,"num_write":140939,"num_write_kb":8960841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":161868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83917,"ondisk_log_size":83917,"up":406,"acting":406,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":472,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28056568,"kb_used_data":24754292,"kb_used_omap":12,"kb_used_meta":3302195,"kb_avail":721904648,"statfs":{"total":767960285184,"available":739230359552,"internally_reserved":0,"allocated":25348395008,"data_stored":25344704098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3381448240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":466026496,"num_objects":7111,"num_object_clones":0,"num_object_copies":21333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7111,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7111,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007140"},"pg_stats":[{"pgid":"9.5","version":"246'8869","reported_seq":8919,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:20.475062+0000","last_change":"2024-03-26T03:08:43.065724+0000","last_active":"2024-03-26T03:10:20.475062+0000","last_peered":"2024-03-26T03:10:20.475062+0000","last_clean":"2024-03-26T03:10:20.475062+0000","last_became_active":"2024-03-26T03:08:43.065248+0000","last_became_peered":"2024-03-26T03:08:43.065248+0000","last_unstale":"2024-03-26T03:10:20.475062+0000","last_undegraded":"2024-03-26T03:10:20.475062+0000","last_fullsized":"2024-03-26T03:10:20.475062+0000","mapping_epoch":225,"log_start":"231'3200","ondisk_log_start":"231'3200","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5669,"log_dups_size":0,"ondisk_log_size":5669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":581238784,"num_objects":8869,"num_object_clones":0,"num_object_copies":26607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8869,"num_write_kb":567616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":558,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:12.150355+0000","last_change":"2024-03-26T03:09:18.705410+0000","last_active":"2024-03-26T03:10:12.150355+0000","last_peered":"2024-03-26T03:10:12.150355+0000","last_clean":"2024-03-26T03:10:12.150355+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:10:12.150355+0000","last_undegraded":"2024-03-26T03:10:12.150355+0000","last_fullsized":"2024-03-26T03:10:12.150355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:09:18.705279+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:09:18.705279+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:19.098363+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":498,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:14.796347+0000","last_change":"2024-03-26T03:10:14.796412+0000","last_active":"2024-03-26T03:10:14.796347+0000","last_peered":"2024-03-26T03:10:14.796347+0000","last_clean":"2024-03-26T03:10:14.796347+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:10:14.796347+0000","last_undegraded":"2024-03-26T03:10:14.796347+0000","last_fullsized":"2024-03-26T03:10:14.796347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:10:14.796187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.796187+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:32.999963+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":407,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.048260+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:10:09.048260+0000","last_peered":"2024-03-26T03:10:09.048260+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:10:09.048260+0000","last_undegraded":"2024-03-26T03:10:09.048260+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:19.229545+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":220,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049013+0000","last_change":"2024-03-26T03:09:18.217911+0000","last_active":"2024-03-26T03:10:09.049013+0000","last_peered":"2024-03-26T03:10:09.049013+0000","last_clean":"2024-03-26T03:10:09.049013+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:10:09.049013+0000","last_undegraded":"2024-03-26T03:10:09.049013+0000","last_fullsized":"2024-03-26T03:10:09.049013+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:18.217789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:18.217789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.742511+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"246'8665","reported_seq":8806,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:19.505368+0000","last_change":"2024-03-26T03:10:12.982393+0000","last_active":"2024-03-26T03:10:19.505368+0000","last_peered":"2024-03-26T03:10:19.505368+0000","last_clean":"2024-03-26T03:10:19.505368+0000","last_became_active":"2024-03-26T03:08:43.063988+0000","last_became_peered":"2024-03-26T03:08:43.063988+0000","last_unstale":"2024-03-26T03:10:19.505368+0000","last_undegraded":"2024-03-26T03:10:19.505368+0000","last_fullsized":"2024-03-26T03:10:19.505368+0000","mapping_epoch":225,"log_start":"231'2200","ondisk_log_start":"231'2200","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'8126","last_scrub_stamp":"2024-03-26T03:10:12.982181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:12.982181+0000","objects_scrubbed":8027,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:16.326438+0000","scrub_duration":2644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567869440,"num_objects":8665,"num_object_clones":0,"num_object_copies":25995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8665,"num_write_kb":554560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":458,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056223+0000","last_change":"2024-03-26T03:09:02.127628+0000","last_active":"2024-03-26T03:10:09.056223+0000","last_peered":"2024-03-26T03:10:09.056223+0000","last_clean":"2024-03-26T03:10:09.056223+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:10:09.056223+0000","last_undegraded":"2024-03-26T03:10:09.056223+0000","last_fullsized":"2024-03-26T03:10:09.056223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'20","last_scrub_stamp":"2024-03-26T03:09:02.127494+0000","last_deep_scrub":"91'20","last_deep_scrub_stamp":"2024-03-26T03:04:06.968735+0000","last_clean_scrub_stamp":"2024-03-26T03:09:02.127494+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":407,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.050938+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:10:09.050938+0000","last_peered":"2024-03-26T03:10:09.050938+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:10:09.050938+0000","last_undegraded":"2024-03-26T03:10:09.050938+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.169439+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":391,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.049191+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:10:09.049191+0000","last_peered":"2024-03-26T03:10:09.049191+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:10:09.049191+0000","last_undegraded":"2024-03-26T03:10:09.049191+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.366273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":408,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056686+0000","last_change":"2024-03-26T03:09:04.206625+0000","last_active":"2024-03-26T03:10:09.056686+0000","last_peered":"2024-03-26T03:10:09.056686+0000","last_clean":"2024-03-26T03:10:09.056686+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:10:09.056686+0000","last_undegraded":"2024-03-26T03:10:09.056686+0000","last_fullsized":"2024-03-26T03:10:09.056686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:04.203542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:09:04.203542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.035575+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"246'8754","reported_seq":8803,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:19.280705+0000","last_change":"2024-03-26T03:08:54.927812+0000","last_active":"2024-03-26T03:10:19.280705+0000","last_peered":"2024-03-26T03:10:19.280705+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.069288+0000","last_became_peered":"2024-03-26T03:08:43.069288+0000","last_unstale":"2024-03-26T03:10:19.280705+0000","last_undegraded":"2024-03-26T03:08:54.925664+0000","last_fullsized":"2024-03-26T03:08:43.062900+0000","mapping_epoch":225,"log_start":"232'4400","ondisk_log_start":"232'4400","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:19.652670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573702144,"num_objects":8754,"num_object_clones":0,"num_object_copies":26262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":560256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":8754}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":511,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049489+0000","last_change":"2024-03-26T03:09:03.343435+0000","last_active":"2024-03-26T03:10:09.049489+0000","last_peered":"2024-03-26T03:10:09.049489+0000","last_clean":"2024-03-26T03:10:09.049489+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:10:09.049489+0000","last_undegraded":"2024-03-26T03:10:09.049489+0000","last_fullsized":"2024-03-26T03:10:09.049489+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'25","last_scrub_stamp":"2024-03-26T03:09:03.343287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:03.343287+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:33.205119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":436,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:17.265414+0000","last_change":"2024-03-26T03:10:17.265447+0000","last_active":"2024-03-26T03:10:17.265414+0000","last_peered":"2024-03-26T03:10:17.265414+0000","last_clean":"2024-03-26T03:10:17.265414+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:10:17.265414+0000","last_undegraded":"2024-03-26T03:10:17.265414+0000","last_fullsized":"2024-03-26T03:10:17.265414+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:17.265332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:10:17.265332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:34.761705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.047703+0000","last_change":"2024-03-26T03:09:59.321675+0000","last_active":"2024-03-26T03:10:09.047703+0000","last_peered":"2024-03-26T03:10:09.047703+0000","last_clean":"2024-03-26T03:10:09.047703+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:10:09.047703+0000","last_undegraded":"2024-03-26T03:10:09.047703+0000","last_fullsized":"2024-03-26T03:10:09.047703+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:59.321541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:09:59.321541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:25.067216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":387,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.054387+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:10:09.054387+0000","last_peered":"2024-03-26T03:10:09.054387+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:10:09.054387+0000","last_undegraded":"2024-03-26T03:10:09.054387+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"246'8747","reported_seq":8884,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:19.767102+0000","last_change":"2024-03-26T03:10:05.447645+0000","last_active":"2024-03-26T03:10:19.767102+0000","last_peered":"2024-03-26T03:10:19.767102+0000","last_clean":"2024-03-26T03:10:19.767102+0000","last_became_active":"2024-03-26T03:08:43.063932+0000","last_became_peered":"2024-03-26T03:08:43.063932+0000","last_unstale":"2024-03-26T03:10:19.767102+0000","last_undegraded":"2024-03-26T03:10:19.767102+0000","last_fullsized":"2024-03-26T03:10:19.767102+0000","mapping_epoch":225,"log_start":"232'4100","ondisk_log_start":"232'4100","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'7536","last_scrub_stamp":"2024-03-26T03:10:05.447508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:05.447508+0000","objects_scrubbed":7456,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:29.597013+0000","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573243392,"num_objects":8747,"num_object_clones":0,"num_object_copies":26241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8747,"num_write_kb":559808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":476,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.056706+0000","last_change":"2024-03-26T03:09:11.836610+0000","last_active":"2024-03-26T03:10:24.056706+0000","last_peered":"2024-03-26T03:10:24.056706+0000","last_clean":"2024-03-26T03:10:24.056706+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:10:24.056706+0000","last_undegraded":"2024-03-26T03:10:24.056706+0000","last_fullsized":"2024-03-26T03:10:24.056706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:09:11.836484+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:09:11.836484+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:26.468304+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.057037+0000","last_change":"2024-03-26T03:09:47.553143+0000","last_active":"2024-03-26T03:10:09.057037+0000","last_peered":"2024-03-26T03:10:09.057037+0000","last_clean":"2024-03-26T03:10:09.057037+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:10:09.057037+0000","last_undegraded":"2024-03-26T03:10:09.057037+0000","last_fullsized":"2024-03-26T03:10:09.057037+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:47.553022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.553022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:13.878799+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":404,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.054548+0000","last_change":"2024-03-26T03:09:15.447157+0000","last_active":"2024-03-26T03:10:09.054548+0000","last_peered":"2024-03-26T03:10:09.054548+0000","last_clean":"2024-03-26T03:10:09.054548+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:10:09.054548+0000","last_undegraded":"2024-03-26T03:10:09.054548+0000","last_fullsized":"2024-03-26T03:10:09.054548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:15.447025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:09:15.447025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:21.944032+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":395,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.048197+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:10:09.048197+0000","last_peered":"2024-03-26T03:10:09.048197+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:10:09.048197+0000","last_undegraded":"2024-03-26T03:10:09.048197+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":473,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.047226+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:10:09.047226+0000","last_peered":"2024-03-26T03:10:09.047226+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:10:09.047226+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.366314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"246'8802","reported_seq":8851,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:19.291770+0000","last_change":"2024-03-26T03:08:54.926696+0000","last_active":"2024-03-26T03:10:19.291770+0000","last_peered":"2024-03-26T03:10:19.291770+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.066721+0000","last_became_peered":"2024-03-26T03:08:43.066721+0000","last_unstale":"2024-03-26T03:10:19.291770+0000","last_undegraded":"2024-03-26T03:08:54.926086+0000","last_fullsized":"2024-03-26T03:08:43.059549+0000","mapping_epoch":225,"log_start":"231'2900","ondisk_log_start":"231'2900","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.209915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":576847872,"num_objects":8802,"num_object_clones":0,"num_object_copies":26406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8802,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8802,"num_write_kb":563328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":8802}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":664,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.057923+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:10:09.057923+0000","last_peered":"2024-03-26T03:10:09.057923+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:10:09.057923+0000","last_undegraded":"2024-03-26T03:10:09.057923+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":436,"reported_epoch":248,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:24.061509+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:10:24.061509+0000","last_peered":"2024-03-26T03:10:24.061509+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:10:24.061509+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.050025+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:10:09.050025+0000","last_peered":"2024-03-26T03:10:09.050025+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:10:09.050025+0000","last_undegraded":"2024-03-26T03:10:09.050025+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":566,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:17.147303+0000","last_change":"2024-03-26T03:09:52.347920+0000","last_active":"2024-03-26T03:10:17.147303+0000","last_peered":"2024-03-26T03:10:17.147303+0000","last_clean":"2024-03-26T03:10:17.147303+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:10:17.147303+0000","last_undegraded":"2024-03-26T03:10:17.147303+0000","last_fullsized":"2024-03-26T03:10:17.147303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:09:52.347788+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:09:52.347788+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:53.918690+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":403,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.057994+0000","last_change":"2024-03-26T03:09:17.928221+0000","last_active":"2024-03-26T03:10:24.057994+0000","last_peered":"2024-03-26T03:10:24.057994+0000","last_clean":"2024-03-26T03:10:24.057994+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:10:24.057994+0000","last_undegraded":"2024-03-26T03:10:24.057994+0000","last_fullsized":"2024-03-26T03:10:24.057994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:17.927617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:17.927617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:47.317421+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"246'8751","reported_seq":8800,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:19.760241+0000","last_change":"2024-03-26T03:08:54.925459+0000","last_active":"2024-03-26T03:10:19.760241+0000","last_peered":"2024-03-26T03:10:19.760241+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.057035+0000","last_became_peered":"2024-03-26T03:08:43.057035+0000","last_unstale":"2024-03-26T03:10:19.760241+0000","last_undegraded":"2024-03-26T03:08:54.924597+0000","last_fullsized":"2024-03-26T03:08:43.055113+0000","mapping_epoch":225,"log_start":"232'4126","ondisk_log_start":"232'4126","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":573505536,"num_objects":8751,"num_object_clones":0,"num_object_copies":26253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":560064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":8751}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":495,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049079+0000","last_change":"2024-03-26T03:09:17.328769+0000","last_active":"2024-03-26T03:10:09.049079+0000","last_peered":"2024-03-26T03:10:09.049079+0000","last_clean":"2024-03-26T03:10:09.049079+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:10:09.049079+0000","last_undegraded":"2024-03-26T03:10:09.049079+0000","last_fullsized":"2024-03-26T03:10:09.049079+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:17.328613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:09:17.328613+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:20.297744+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":446,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.050731+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:10:09.050731+0000","last_peered":"2024-03-26T03:10:09.050731+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:10:09.050731+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.169424+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.049793+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:10:09.049793+0000","last_peered":"2024-03-26T03:10:09.049793+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:10:09.049793+0000","last_undegraded":"2024-03-26T03:10:09.049793+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238751+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":561,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.057848+0000","last_change":"2024-03-26T03:09:51.797832+0000","last_active":"2024-03-26T03:10:24.057848+0000","last_peered":"2024-03-26T03:10:24.057848+0000","last_clean":"2024-03-26T03:10:24.057848+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:10:24.057848+0000","last_undegraded":"2024-03-26T03:10:24.057848+0000","last_fullsized":"2024-03-26T03:10:24.057848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:09:51.797681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:51.797681+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:06.772612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":212,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.050115+0000","last_change":"2024-03-26T03:09:54.414700+0000","last_active":"2024-03-26T03:10:09.050115+0000","last_peered":"2024-03-26T03:10:09.050115+0000","last_clean":"2024-03-26T03:10:09.050115+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:10:09.050115+0000","last_undegraded":"2024-03-26T03:10:09.050115+0000","last_fullsized":"2024-03-26T03:10:09.050115+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:54.414398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:09:54.414398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:55.540353+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"246'8631","reported_seq":8680,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:19.235023+0000","last_change":"2024-03-26T03:08:54.940154+0000","last_active":"2024-03-26T03:10:19.235023+0000","last_peered":"2024-03-26T03:10:19.235023+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.064645+0000","last_became_peered":"2024-03-26T03:08:43.064645+0000","last_unstale":"2024-03-26T03:10:19.235023+0000","last_undegraded":"2024-03-26T03:08:54.937570+0000","last_fullsized":"2024-03-26T03:08:43.060747+0000","mapping_epoch":225,"log_start":"232'4300","ondisk_log_start":"232'4300","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:19.652646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":565641216,"num_objects":8631,"num_object_clones":0,"num_object_copies":25893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8631,"num_write_kb":552384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":8631}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"231'25","reported_seq":462,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.056780+0000","last_change":"2024-03-26T03:09:08.904745+0000","last_active":"2024-03-26T03:10:24.056780+0000","last_peered":"2024-03-26T03:10:24.056780+0000","last_clean":"2024-03-26T03:10:24.056780+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:10:24.056780+0000","last_undegraded":"2024-03-26T03:10:24.056780+0000","last_fullsized":"2024-03-26T03:10:24.056780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:09:08.904611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:08.904611+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:30.544328+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.054733+0000","last_change":"2024-03-26T03:09:51.654489+0000","last_active":"2024-03-26T03:10:09.054733+0000","last_peered":"2024-03-26T03:10:09.054733+0000","last_clean":"2024-03-26T03:10:09.054733+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:10:09.054733+0000","last_undegraded":"2024-03-26T03:10:09.054733+0000","last_fullsized":"2024-03-26T03:10:09.054733+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:51.654364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:51.654364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:51.917313+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.057533+0000","last_change":"2024-03-26T03:09:56.868132+0000","last_active":"2024-03-26T03:10:24.057533+0000","last_peered":"2024-03-26T03:10:24.057533+0000","last_clean":"2024-03-26T03:10:24.057533+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:10:24.057533+0000","last_undegraded":"2024-03-26T03:10:24.057533+0000","last_fullsized":"2024-03-26T03:10:24.057533+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:56.867463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:56.867463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:19.527285+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.049370+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:10:09.049370+0000","last_peered":"2024-03-26T03:10:09.049370+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:10:09.049370+0000","last_undegraded":"2024-03-26T03:10:09.049370+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"248'9087","reported_seq":9140,"reported_epoch":248,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:24.186193+0000","last_change":"2024-03-26T03:08:54.944412+0000","last_active":"2024-03-26T03:10:24.186193+0000","last_peered":"2024-03-26T03:10:24.186193+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.063578+0000","last_became_peered":"2024-03-26T03:08:43.063578+0000","last_unstale":"2024-03-26T03:10:24.186193+0000","last_undegraded":"2024-03-26T03:08:54.938583+0000","last_fullsized":"2024-03-26T03:08:43.056078+0000","mapping_epoch":225,"log_start":"232'4000","ondisk_log_start":"232'4000","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":595525632,"num_objects":9087,"num_object_clones":0,"num_object_copies":27261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":581568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":9087}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":243,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:17.147559+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:10:17.147559+0000","last_peered":"2024-03-26T03:10:17.147559+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:10:17.147559+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":481,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.055724+0000","last_change":"2024-03-26T03:10:15.759744+0000","last_active":"2024-03-26T03:10:24.055724+0000","last_peered":"2024-03-26T03:10:24.055724+0000","last_clean":"2024-03-26T03:10:24.055724+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:10:24.055724+0000","last_undegraded":"2024-03-26T03:10:24.055724+0000","last_fullsized":"2024-03-26T03:10:24.055724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:10:15.759584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:15.759584+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:41.731999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051284+0000","last_change":"2024-03-26T03:08:57.389505+0000","last_active":"2024-03-26T03:10:09.051284+0000","last_peered":"2024-03-26T03:10:09.051284+0000","last_clean":"2024-03-26T03:10:09.051284+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:10:09.051284+0000","last_undegraded":"2024-03-26T03:10:09.051284+0000","last_fullsized":"2024-03-26T03:10:09.051284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:57.389433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.389433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:27.014231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":400,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.057611+0000","last_change":"2024-03-26T03:08:59.750899+0000","last_active":"2024-03-26T03:10:09.057611+0000","last_peered":"2024-03-26T03:10:09.057611+0000","last_clean":"2024-03-26T03:10:09.057611+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:10:09.057611+0000","last_undegraded":"2024-03-26T03:10:09.057611+0000","last_fullsized":"2024-03-26T03:10:09.057611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:59.750496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:08:59.750496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.239093+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":581,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.054660+0000","last_change":"2024-03-26T03:06:20.276636+0000","last_active":"2024-03-26T03:10:09.054660+0000","last_peered":"2024-03-26T03:10:09.054660+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:06:20.276636+0000","last_became_peered":"2024-03-26T03:06:20.276636+0000","last_unstale":"2024-03-26T03:10:09.054660+0000","last_undegraded":"2024-03-26T03:10:09.054660+0000","last_fullsized":"2024-03-26T03:06:20.268958+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.685268+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"246'8759","reported_seq":8811,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:19.272742+0000","last_change":"2024-03-26T03:08:43.069162+0000","last_active":"2024-03-26T03:10:19.272742+0000","last_peered":"2024-03-26T03:10:19.272742+0000","last_clean":"2024-03-26T03:10:19.272742+0000","last_became_active":"2024-03-26T03:08:43.068853+0000","last_became_peered":"2024-03-26T03:08:43.068853+0000","last_unstale":"2024-03-26T03:10:19.272742+0000","last_undegraded":"2024-03-26T03:10:19.272742+0000","last_fullsized":"2024-03-26T03:10:19.272742+0000","mapping_epoch":225,"log_start":"232'4400","ondisk_log_start":"232'4400","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4359,"log_dups_size":0,"ondisk_log_size":4359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574029824,"num_objects":8759,"num_object_clones":0,"num_object_copies":26277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8759,"num_write_kb":560576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":394,"reported_epoch":248,"state":"active+undersized","last_fresh":"2024-03-26T03:10:24.056060+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:10:24.056060+0000","last_peered":"2024-03-26T03:10:24.056060+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:10:24.056060+0000","last_undegraded":"2024-03-26T03:10:24.056060+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":230,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.054993+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:10:09.054993+0000","last_peered":"2024-03-26T03:10:09.054993+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:10:09.054993+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":422,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.050698+0000","last_change":"2024-03-26T03:09:50.291684+0000","last_active":"2024-03-26T03:10:09.050698+0000","last_peered":"2024-03-26T03:10:09.050698+0000","last_clean":"2024-03-26T03:10:09.050698+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:10:09.050698+0000","last_undegraded":"2024-03-26T03:10:09.050698+0000","last_fullsized":"2024-03-26T03:10:09.050698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:09:50.291459+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:09:50.291459+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:16.885416+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":390,"reported_epoch":248,"state":"active+undersized","last_fresh":"2024-03-26T03:10:24.056489+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:10:24.056489+0000","last_peered":"2024-03-26T03:10:24.056489+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:10:24.056489+0000","last_undegraded":"2024-03-26T03:10:24.056489+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566335+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":499,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.056224+0000","last_change":"2024-03-26T03:09:54.860295+0000","last_active":"2024-03-26T03:10:24.056224+0000","last_peered":"2024-03-26T03:10:24.056224+0000","last_clean":"2024-03-26T03:10:24.056224+0000","last_became_active":"2024-03-26T03:04:41.967264+0000","last_became_peered":"2024-03-26T03:04:41.967264+0000","last_unstale":"2024-03-26T03:10:24.056224+0000","last_undegraded":"2024-03-26T03:10:24.056224+0000","last_fullsized":"2024-03-26T03:10:24.056224+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:54.860159+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:09:54.860159+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:12.531236+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"246'8669","reported_seq":8745,"reported_epoch":246,"state":"active+clean+scrubbing","last_fresh":"2024-03-26T03:10:19.266991+0000","last_change":"2024-03-26T03:10:18.622327+0000","last_active":"2024-03-26T03:10:19.266991+0000","last_peered":"2024-03-26T03:10:19.266991+0000","last_clean":"2024-03-26T03:10:19.266991+0000","last_became_active":"2024-03-26T03:08:43.067345+0000","last_became_peered":"2024-03-26T03:08:43.067345+0000","last_unstale":"2024-03-26T03:10:19.266991+0000","last_undegraded":"2024-03-26T03:10:19.266991+0000","last_fullsized":"2024-03-26T03:10:19.266991+0000","mapping_epoch":225,"log_start":"232'4300","ondisk_log_start":"232'4300","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":2099,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":568131584,"num_objects":8669,"num_object_clones":0,"num_object_copies":26007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049676+0000","last_change":"2024-03-26T03:09:00.346125+0000","last_active":"2024-03-26T03:10:09.049676+0000","last_peered":"2024-03-26T03:10:09.049676+0000","last_clean":"2024-03-26T03:10:09.049676+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:10:09.049676+0000","last_undegraded":"2024-03-26T03:10:09.049676+0000","last_fullsized":"2024-03-26T03:10:09.049676+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:00.346008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:00.346008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.285027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":487,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056027+0000","last_change":"2024-03-26T03:09:47.209344+0000","last_active":"2024-03-26T03:10:09.056027+0000","last_peered":"2024-03-26T03:10:09.056027+0000","last_clean":"2024-03-26T03:10:09.056027+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:10:09.056027+0000","last_undegraded":"2024-03-26T03:10:09.056027+0000","last_fullsized":"2024-03-26T03:10:09.056027+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:09:47.209142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.209142+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:12.479392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.059095+0000","last_change":"2024-03-26T03:09:09.688709+0000","last_active":"2024-03-26T03:10:09.059095+0000","last_peered":"2024-03-26T03:10:09.059095+0000","last_clean":"2024-03-26T03:10:09.059095+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:10:09.059095+0000","last_undegraded":"2024-03-26T03:10:09.059095+0000","last_fullsized":"2024-03-26T03:10:09.059095+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:09.688605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:09.688605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:16.840542+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049646+0000","last_change":"2024-03-26T03:09:46.394734+0000","last_active":"2024-03-26T03:10:09.049646+0000","last_peered":"2024-03-26T03:10:09.049646+0000","last_clean":"2024-03-26T03:10:09.049646+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:10:09.049646+0000","last_undegraded":"2024-03-26T03:10:09.049646+0000","last_fullsized":"2024-03-26T03:10:09.049646+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:46.394652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:46.394652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:15.682689+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":480,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.051631+0000","last_change":"2024-03-26T03:06:20.308944+0000","last_active":"2024-03-26T03:10:09.051631+0000","last_peered":"2024-03-26T03:10:09.051631+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:06:20.308944+0000","last_became_peered":"2024-03-26T03:06:20.308944+0000","last_unstale":"2024-03-26T03:10:09.051631+0000","last_undegraded":"2024-03-26T03:10:09.051631+0000","last_fullsized":"2024-03-26T03:06:20.270592+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"246'8807","reported_seq":8857,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:20.446931+0000","last_change":"2024-03-26T03:08:43.067206+0000","last_active":"2024-03-26T03:10:20.446931+0000","last_peered":"2024-03-26T03:10:20.446931+0000","last_clean":"2024-03-26T03:10:20.446931+0000","last_became_active":"2024-03-26T03:08:43.066837+0000","last_became_peered":"2024-03-26T03:08:43.066837+0000","last_unstale":"2024-03-26T03:10:20.446931+0000","last_undegraded":"2024-03-26T03:10:20.446931+0000","last_fullsized":"2024-03-26T03:10:20.446931+0000","mapping_epoch":225,"log_start":"231'3100","ondisk_log_start":"231'3100","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5707,"log_dups_size":0,"ondisk_log_size":5707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577175552,"num_objects":8807,"num_object_clones":0,"num_object_copies":26421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":563648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.055737+0000","last_change":"2024-03-26T03:09:46.001706+0000","last_active":"2024-03-26T03:10:09.055737+0000","last_peered":"2024-03-26T03:10:09.055737+0000","last_clean":"2024-03-26T03:10:09.055737+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:10:09.055737+0000","last_undegraded":"2024-03-26T03:10:09.055737+0000","last_fullsized":"2024-03-26T03:10:09.055737+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:46.001491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:46.001491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:12.883102+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":466,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:16.242930+0000","last_change":"2024-03-26T03:10:16.242987+0000","last_active":"2024-03-26T03:10:16.242930+0000","last_peered":"2024-03-26T03:10:16.242930+0000","last_clean":"2024-03-26T03:10:16.242930+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:10:16.242930+0000","last_undegraded":"2024-03-26T03:10:16.242930+0000","last_fullsized":"2024-03-26T03:10:16.242930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:10:16.242828+0000","last_deep_scrub":"219'18","last_deep_scrub_stamp":"2024-03-26T03:08:11.142496+0000","last_clean_scrub_stamp":"2024-03-26T03:10:16.242828+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.466052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.054564+0000","last_change":"2024-03-26T03:09:58.731760+0000","last_active":"2024-03-26T03:10:09.054564+0000","last_peered":"2024-03-26T03:10:09.054564+0000","last_clean":"2024-03-26T03:10:09.054564+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:10:09.054564+0000","last_undegraded":"2024-03-26T03:10:09.054564+0000","last_fullsized":"2024-03-26T03:10:09.054564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:58.731648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:58.731648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:59.837715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":414,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.056350+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:10:09.056350+0000","last_peered":"2024-03-26T03:10:09.056350+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:10:09.056350+0000","last_undegraded":"2024-03-26T03:10:09.056350+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":395,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.054608+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:10:09.054608+0000","last_peered":"2024-03-26T03:10:09.054608+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:10:09.054608+0000","last_undegraded":"2024-03-26T03:10:09.054608+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":495,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051501+0000","last_change":"2024-03-26T03:10:02.228551+0000","last_active":"2024-03-26T03:10:09.051501+0000","last_peered":"2024-03-26T03:10:09.051501+0000","last_clean":"2024-03-26T03:10:09.051501+0000","last_became_active":"2024-03-26T03:04:41.964807+0000","last_became_peered":"2024-03-26T03:04:41.964807+0000","last_unstale":"2024-03-26T03:10:09.051501+0000","last_undegraded":"2024-03-26T03:10:09.051501+0000","last_fullsized":"2024-03-26T03:10:09.051501+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:02.228412+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:02.228412+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:09.734620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"246'8572","reported_seq":8621,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:20.434412+0000","last_change":"2024-03-26T03:08:54.931470+0000","last_active":"2024-03-26T03:10:20.434412+0000","last_peered":"2024-03-26T03:10:20.434412+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.065260+0000","last_became_peered":"2024-03-26T03:08:43.065260+0000","last_unstale":"2024-03-26T03:10:20.434412+0000","last_undegraded":"2024-03-26T03:08:54.924879+0000","last_fullsized":"2024-03-26T03:08:43.057979+0000","mapping_epoch":225,"log_start":"231'2900","ondisk_log_start":"231'2900","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":561774592,"num_objects":8572,"num_object_clones":0,"num_object_copies":25716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8572,"num_write_kb":548608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":8572}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.055140+0000","last_change":"2024-03-26T03:09:52.697598+0000","last_active":"2024-03-26T03:10:09.055140+0000","last_peered":"2024-03-26T03:10:09.055140+0000","last_clean":"2024-03-26T03:10:09.055140+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:10:09.055140+0000","last_undegraded":"2024-03-26T03:10:09.055140+0000","last_fullsized":"2024-03-26T03:10:09.055140+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:52.697426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:52.697426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:18.762123+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"231'28","reported_seq":593,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:12.146331+0000","last_change":"2024-03-26T03:09:20.236853+0000","last_active":"2024-03-26T03:10:12.146331+0000","last_peered":"2024-03-26T03:10:12.146331+0000","last_clean":"2024-03-26T03:10:12.146331+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:10:12.146331+0000","last_undegraded":"2024-03-26T03:10:12.146331+0000","last_fullsized":"2024-03-26T03:10:12.146331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:09:20.236696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:20.236696+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:40.931974+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":670,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.054645+0000","last_change":"2024-03-26T03:09:13.758950+0000","last_active":"2024-03-26T03:10:09.054645+0000","last_peered":"2024-03-26T03:10:09.054645+0000","last_clean":"2024-03-26T03:10:09.054645+0000","last_became_active":"2024-03-26T03:04:37.115068+0000","last_became_peered":"2024-03-26T03:04:37.115068+0000","last_unstale":"2024-03-26T03:10:09.054645+0000","last_undegraded":"2024-03-26T03:10:09.054645+0000","last_fullsized":"2024-03-26T03:10:09.054645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:13.758832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:09:13.758832+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:23.498458+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":441,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.055722+0000","last_change":"2024-03-26T03:09:57.869490+0000","last_active":"2024-03-26T03:10:09.055722+0000","last_peered":"2024-03-26T03:10:09.055722+0000","last_clean":"2024-03-26T03:10:09.055722+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:10:09.055722+0000","last_undegraded":"2024-03-26T03:10:09.055722+0000","last_fullsized":"2024-03-26T03:10:09.055722+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:57.869377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.869377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:13.793495+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":130,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.054883+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:10:09.054883+0000","last_peered":"2024-03-26T03:10:09.054883+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:10:09.054883+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":394,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.059504+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:10:09.059504+0000","last_peered":"2024-03-26T03:10:09.059504+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:10:09.059504+0000","last_undegraded":"2024-03-26T03:10:09.059504+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"246'8690","reported_seq":8832,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:19.297901+0000","last_change":"2024-03-26T03:10:14.319579+0000","last_active":"2024-03-26T03:10:19.297901+0000","last_peered":"2024-03-26T03:10:19.297901+0000","last_clean":"2024-03-26T03:10:19.297901+0000","last_became_active":"2024-03-26T03:08:43.066310+0000","last_became_peered":"2024-03-26T03:08:43.066310+0000","last_unstale":"2024-03-26T03:10:19.297901+0000","last_undegraded":"2024-03-26T03:10:19.297901+0000","last_fullsized":"2024-03-26T03:10:19.297901+0000","mapping_epoch":225,"log_start":"232'3700","ondisk_log_start":"232'3700","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'8262","last_scrub_stamp":"2024-03-26T03:10:14.319349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.319349+0000","objects_scrubbed":8182,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:37.047077+0000","scrub_duration":2505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":569507840,"num_objects":8690,"num_object_clones":0,"num_object_copies":26070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8690,"num_write_kb":556160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.048462+0000","last_change":"2024-03-26T03:10:06.377063+0000","last_active":"2024-03-26T03:10:09.048462+0000","last_peered":"2024-03-26T03:10:09.048462+0000","last_clean":"2024-03-26T03:10:09.048462+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:10:09.048462+0000","last_undegraded":"2024-03-26T03:10:09.048462+0000","last_fullsized":"2024-03-26T03:10:09.048462+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:06.376932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:06.376932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:08.244674+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":461,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051060+0000","last_change":"2024-03-26T03:09:49.282590+0000","last_active":"2024-03-26T03:10:09.051060+0000","last_peered":"2024-03-26T03:10:09.051060+0000","last_clean":"2024-03-26T03:10:09.051060+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:10:09.051060+0000","last_undegraded":"2024-03-26T03:10:09.051060+0000","last_fullsized":"2024-03-26T03:10:09.051060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:09:49.282256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:49.282256+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:10.364929+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":407,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.048220+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:10:09.048220+0000","last_peered":"2024-03-26T03:10:09.048220+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:10:09.048220+0000","last_undegraded":"2024-03-26T03:10:09.048220+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.366358+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1110,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:18.161378+0000","last_change":"2024-03-26T03:10:18.161378+0000","last_active":"2024-03-26T03:10:18.161378+0000","last_peered":"2024-03-26T03:10:18.161378+0000","last_clean":"2024-03-26T03:10:18.161378+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:10:18.161378+0000","last_undegraded":"2024-03-26T03:10:18.161378+0000","last_fullsized":"2024-03-26T03:10:18.161378+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:10:18.161157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:18.161157+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:46.581275+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":672,"num_read_kb":667,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":395,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.058916+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:10:09.058916+0000","last_peered":"2024-03-26T03:10:09.058916+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:10:09.058916+0000","last_undegraded":"2024-03-26T03:10:09.058916+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673834+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":677,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056458+0000","last_change":"2024-03-26T03:09:14.720943+0000","last_active":"2024-03-26T03:10:09.056458+0000","last_peered":"2024-03-26T03:10:09.056458+0000","last_clean":"2024-03-26T03:10:09.056458+0000","last_became_active":"2024-03-26T03:04:41.965091+0000","last_became_peered":"2024-03-26T03:04:41.965091+0000","last_unstale":"2024-03-26T03:10:09.056458+0000","last_undegraded":"2024-03-26T03:10:09.056458+0000","last_fullsized":"2024-03-26T03:10:09.056458+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:09:14.720791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:09:14.720791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:22.530807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"246'8812","reported_seq":8864,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:20.462051+0000","last_change":"2024-03-26T03:08:43.066301+0000","last_active":"2024-03-26T03:10:20.462051+0000","last_peered":"2024-03-26T03:10:20.462051+0000","last_clean":"2024-03-26T03:10:20.462051+0000","last_became_active":"2024-03-26T03:08:43.065826+0000","last_became_peered":"2024-03-26T03:08:43.065826+0000","last_unstale":"2024-03-26T03:10:20.462051+0000","last_undegraded":"2024-03-26T03:10:20.462051+0000","last_fullsized":"2024-03-26T03:10:20.462051+0000","mapping_epoch":225,"log_start":"231'3100","ondisk_log_start":"231'3100","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":577503232,"num_objects":8812,"num_object_clones":0,"num_object_copies":26436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":563968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":412,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.058354+0000","last_change":"2024-03-26T03:08:57.142251+0000","last_active":"2024-03-26T03:10:09.058354+0000","last_peered":"2024-03-26T03:10:09.058354+0000","last_clean":"2024-03-26T03:10:09.058354+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:10:09.058354+0000","last_undegraded":"2024-03-26T03:10:09.058354+0000","last_fullsized":"2024-03-26T03:10:09.058354+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:57.142119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.142119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.006292+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"231'58","reported_seq":630,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:12.146401+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:10:12.146401+0000","last_peered":"2024-03-26T03:10:12.146401+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:10:12.146401+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.169404+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":212,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049833+0000","last_change":"2024-03-26T03:09:59.289089+0000","last_active":"2024-03-26T03:10:09.049833+0000","last_peered":"2024-03-26T03:10:09.049833+0000","last_clean":"2024-03-26T03:10:09.049833+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:10:09.049833+0000","last_undegraded":"2024-03-26T03:10:09.049833+0000","last_fullsized":"2024-03-26T03:10:09.049833+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:59.288971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:59.288971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:01.541185+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":108,"reported_epoch":248,"state":"active+undersized","last_fresh":"2024-03-26T03:10:24.057994+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:10:24.057994+0000","last_peered":"2024-03-26T03:10:24.057994+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:10:24.057994+0000","last_undegraded":"2024-03-26T03:10:24.057994+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":501,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056014+0000","last_change":"2024-03-26T03:10:04.204642+0000","last_active":"2024-03-26T03:10:09.056014+0000","last_peered":"2024-03-26T03:10:09.056014+0000","last_clean":"2024-03-26T03:10:09.056014+0000","last_became_active":"2024-03-26T03:04:41.966771+0000","last_became_peered":"2024-03-26T03:04:41.966771+0000","last_unstale":"2024-03-26T03:10:09.056014+0000","last_undegraded":"2024-03-26T03:10:09.056014+0000","last_fullsized":"2024-03-26T03:10:09.056014+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:10:04.204409+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:16.476699+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"246'8662","reported_seq":8711,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:19.310846+0000","last_change":"2024-03-26T03:08:54.940176+0000","last_active":"2024-03-26T03:10:19.310846+0000","last_peered":"2024-03-26T03:10:19.310846+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.067789+0000","last_became_peered":"2024-03-26T03:08:43.067789+0000","last_unstale":"2024-03-26T03:10:19.310846+0000","last_undegraded":"2024-03-26T03:08:54.937224+0000","last_fullsized":"2024-03-26T03:08:43.062110+0000","mapping_epoch":225,"log_start":"232'3700","ondisk_log_start":"232'3700","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567672832,"num_objects":8662,"num_object_clones":0,"num_object_copies":25986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8662,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":554368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8662}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":560,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:12.146530+0000","last_change":"2024-03-26T03:08:59.116331+0000","last_active":"2024-03-26T03:10:12.146530+0000","last_peered":"2024-03-26T03:10:12.146530+0000","last_clean":"2024-03-26T03:10:12.146530+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:10:12.146530+0000","last_undegraded":"2024-03-26T03:10:12.146530+0000","last_fullsized":"2024-03-26T03:10:12.146530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:08:59.116008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:59.116008+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:22.124498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":459,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.050360+0000","last_change":"2024-03-26T03:08:57.248514+0000","last_active":"2024-03-26T03:10:09.050360+0000","last_peered":"2024-03-26T03:10:09.050360+0000","last_clean":"2024-03-26T03:10:09.050360+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:10:09.050360+0000","last_undegraded":"2024-03-26T03:10:09.050360+0000","last_fullsized":"2024-03-26T03:10:09.050360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-26T03:08:57.248360+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.248360+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.554280+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.054238+0000","last_change":"2024-03-26T03:10:00.280603+0000","last_active":"2024-03-26T03:10:09.054238+0000","last_peered":"2024-03-26T03:10:09.054238+0000","last_clean":"2024-03-26T03:10:09.054238+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:10:09.054238+0000","last_undegraded":"2024-03-26T03:10:09.054238+0000","last_fullsized":"2024-03-26T03:10:09.054238+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:00.280494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:00.280494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:08.462987+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":395,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.055470+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:10:09.055470+0000","last_peered":"2024-03-26T03:10:09.055470+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:10:09.055470+0000","last_undegraded":"2024-03-26T03:10:09.055470+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194377+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":480,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.051138+0000","last_change":"2024-03-26T03:06:20.272249+0000","last_active":"2024-03-26T03:10:09.051138+0000","last_peered":"2024-03-26T03:10:09.051138+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:06:20.272249+0000","last_became_peered":"2024-03-26T03:06:20.272249+0000","last_unstale":"2024-03-26T03:10:09.051138+0000","last_undegraded":"2024-03-26T03:10:09.051138+0000","last_fullsized":"2024-03-26T03:06:20.268693+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"246'8732","reported_seq":8781,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:19.804000+0000","last_change":"2024-03-26T03:08:54.931125+0000","last_active":"2024-03-26T03:10:19.804000+0000","last_peered":"2024-03-26T03:10:19.804000+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.058021+0000","last_became_peered":"2024-03-26T03:08:43.058021+0000","last_unstale":"2024-03-26T03:10:19.804000+0000","last_undegraded":"2024-03-26T03:08:54.930155+0000","last_fullsized":"2024-03-26T03:08:43.054400+0000","mapping_epoch":225,"log_start":"232'4100","ondisk_log_start":"232'4100","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":572260352,"num_objects":8732,"num_object_clones":0,"num_object_copies":26196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8732,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8732,"num_write_kb":558848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":8732}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.047661+0000","last_change":"2024-03-26T03:09:05.145615+0000","last_active":"2024-03-26T03:10:09.047661+0000","last_peered":"2024-03-26T03:10:09.047661+0000","last_clean":"2024-03-26T03:10:09.047661+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:10:09.047661+0000","last_undegraded":"2024-03-26T03:10:09.047661+0000","last_fullsized":"2024-03-26T03:10:09.047661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:05.145462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:05.145462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:32.012329+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":476,"reported_epoch":248,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:24.055945+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:10:24.055945+0000","last_peered":"2024-03-26T03:10:24.055945+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:10:24.055945+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566264+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":455,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.057178+0000","last_change":"2024-03-26T03:10:08.567751+0000","last_active":"2024-03-26T03:10:24.057178+0000","last_peered":"2024-03-26T03:10:24.057178+0000","last_clean":"2024-03-26T03:10:24.057178+0000","last_became_active":"2024-03-26T03:10:08.555235+0000","last_became_peered":"2024-03-26T03:10:08.555235+0000","last_unstale":"2024-03-26T03:10:24.057178+0000","last_undegraded":"2024-03-26T03:10:24.057178+0000","last_fullsized":"2024-03-26T03:10:24.057178+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:09:19.933700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:19.933700+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:29.145968+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.057086+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:10:09.057086+0000","last_peered":"2024-03-26T03:10:09.057086+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:10:09.057086+0000","last_undegraded":"2024-03-26T03:10:09.057086+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":451,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.056900+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:10:09.056900+0000","last_peered":"2024-03-26T03:10:09.056900+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:10:09.056900+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194428+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":433,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.047554+0000","last_change":"2024-03-26T03:09:51.324717+0000","last_active":"2024-03-26T03:10:09.047554+0000","last_peered":"2024-03-26T03:10:09.047554+0000","last_clean":"2024-03-26T03:10:09.047554+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:10:09.047554+0000","last_undegraded":"2024-03-26T03:10:09.047554+0000","last_fullsized":"2024-03-26T03:10:09.047554+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:09:51.324431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:51.324431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:02.322588+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.050844+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:10:09.050844+0000","last_peered":"2024-03-26T03:10:09.050844+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:10:09.050844+0000","last_undegraded":"2024-03-26T03:10:09.050844+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":415,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.047454+0000","last_change":"2024-03-26T03:09:53.347817+0000","last_active":"2024-03-26T03:10:09.047454+0000","last_peered":"2024-03-26T03:10:09.047454+0000","last_clean":"2024-03-26T03:10:09.047454+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:10:09.047454+0000","last_undegraded":"2024-03-26T03:10:09.047454+0000","last_fullsized":"2024-03-26T03:10:09.047454+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:53.347679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:09:53.347679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:12.249602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":153,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.057340+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:10:09.057340+0000","last_peered":"2024-03-26T03:10:09.057340+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:10:09.057340+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051254+0000","last_change":"2024-03-26T03:09:45.324624+0000","last_active":"2024-03-26T03:10:09.051254+0000","last_peered":"2024-03-26T03:10:09.051254+0000","last_clean":"2024-03-26T03:10:09.051254+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:10:09.051254+0000","last_undegraded":"2024-03-26T03:10:09.051254+0000","last_fullsized":"2024-03-26T03:10:09.051254+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:45.324180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:45.324180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:52.941801+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":397,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.054919+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:10:09.054919+0000","last_peered":"2024-03-26T03:10:09.054919+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:10:09.054919+0000","last_undegraded":"2024-03-26T03:10:09.054919+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056846+0000","last_change":"2024-03-26T03:09:57.777835+0000","last_active":"2024-03-26T03:10:09.056846+0000","last_peered":"2024-03-26T03:10:09.056846+0000","last_clean":"2024-03-26T03:10:09.056846+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:10:09.056846+0000","last_undegraded":"2024-03-26T03:10:09.056846+0000","last_fullsized":"2024-03-26T03:10:09.056846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:57.777717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.777717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:17.013769+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":352,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.054528+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:10:09.054528+0000","last_peered":"2024-03-26T03:10:09.054528+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:10:09.054528+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":413,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.054464+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:10:09.054464+0000","last_peered":"2024-03-26T03:10:09.054464+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:10:09.054464+0000","last_undegraded":"2024-03-26T03:10:09.054464+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":564,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:12.146282+0000","last_change":"2024-03-26T03:09:50.922297+0000","last_active":"2024-03-26T03:10:12.146282+0000","last_peered":"2024-03-26T03:10:12.146282+0000","last_clean":"2024-03-26T03:10:12.146282+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:10:12.146282+0000","last_undegraded":"2024-03-26T03:10:12.146282+0000","last_fullsized":"2024-03-26T03:10:12.146282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:09:50.922184+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:09:50.922184+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:16.204031+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":394,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.056022+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:10:09.056022+0000","last_peered":"2024-03-26T03:10:09.056022+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:10:09.056022+0000","last_undegraded":"2024-03-26T03:10:09.056022+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194365+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":247,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.058718+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:10:09.058718+0000","last_peered":"2024-03-26T03:10:09.058718+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:10:09.058718+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":398,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.056149+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:10:09.056149+0000","last_peered":"2024-03-26T03:10:09.056149+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:10:09.056149+0000","last_undegraded":"2024-03-26T03:10:09.056149+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194353+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":178,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:12.146330+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:10:12.146330+0000","last_peered":"2024-03-26T03:10:12.146330+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:10:12.146330+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:19.650265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":403,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.057622+0000","last_change":"2024-03-26T03:09:02.967331+0000","last_active":"2024-03-26T03:10:24.057622+0000","last_peered":"2024-03-26T03:10:24.057622+0000","last_clean":"2024-03-26T03:10:24.057622+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:10:24.057622+0000","last_undegraded":"2024-03-26T03:10:24.057622+0000","last_fullsized":"2024-03-26T03:10:24.057622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:09:02.967216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:02.967216+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:32.040525+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"230'15","reported_seq":440,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:13.175094+0000","last_change":"2024-03-26T03:10:13.175140+0000","last_active":"2024-03-26T03:10:13.175094+0000","last_peered":"2024-03-26T03:10:13.175094+0000","last_clean":"2024-03-26T03:10:13.175094+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:10:13.175094+0000","last_undegraded":"2024-03-26T03:10:13.175094+0000","last_fullsized":"2024-03-26T03:10:13.175094+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'15","last_scrub_stamp":"2024-03-26T03:10:13.174994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:13.174994+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:35.656575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":406,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.049458+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:10:09.049458+0000","last_peered":"2024-03-26T03:10:09.049458+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:10:09.049458+0000","last_undegraded":"2024-03-26T03:10:09.049458+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.056285+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:10:09.056285+0000","last_peered":"2024-03-26T03:10:09.056285+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:10:09.056285+0000","last_undegraded":"2024-03-26T03:10:09.056285+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":410,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.055422+0000","last_change":"2024-03-26T03:09:08.164518+0000","last_active":"2024-03-26T03:10:09.055422+0000","last_peered":"2024-03-26T03:10:09.055422+0000","last_clean":"2024-03-26T03:10:09.055422+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:10:09.055422+0000","last_undegraded":"2024-03-26T03:10:09.055422+0000","last_fullsized":"2024-03-26T03:10:09.055422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:08.164294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:08.164294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:13.644605+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":449,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.056305+0000","last_change":"2024-03-26T03:09:48.867245+0000","last_active":"2024-03-26T03:10:24.056305+0000","last_peered":"2024-03-26T03:10:24.056305+0000","last_clean":"2024-03-26T03:10:24.056305+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:10:24.056305+0000","last_undegraded":"2024-03-26T03:10:24.056305+0000","last_fullsized":"2024-03-26T03:10:24.056305+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:09:48.867101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:48.867101+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:10.634184+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.050693+0000","last_change":"2024-03-26T03:09:01.395031+0000","last_active":"2024-03-26T03:10:09.050693+0000","last_peered":"2024-03-26T03:10:09.050693+0000","last_clean":"2024-03-26T03:10:09.050693+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:10:09.050693+0000","last_undegraded":"2024-03-26T03:10:09.050693+0000","last_fullsized":"2024-03-26T03:10:09.050693+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:01.394917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:09:01.394917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:24.403976+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":390,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.050463+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:10:09.050463+0000","last_peered":"2024-03-26T03:10:09.050463+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:10:09.050463+0000","last_undegraded":"2024-03-26T03:10:09.050463+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238780+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.056447+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:10:09.056447+0000","last_peered":"2024-03-26T03:10:09.056447+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:10:09.056447+0000","last_undegraded":"2024-03-26T03:10:09.056447+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":135,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.053173+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:10:09.053173+0000","last_peered":"2024-03-26T03:10:09.053173+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:10:09.053173+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":426,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.059957+0000","last_change":"2024-03-26T03:09:18.868778+0000","last_active":"2024-03-26T03:10:09.059957+0000","last_peered":"2024-03-26T03:10:09.059957+0000","last_clean":"2024-03-26T03:10:09.059957+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:10:09.059957+0000","last_undegraded":"2024-03-26T03:10:09.059957+0000","last_fullsized":"2024-03-26T03:10:09.059957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:09:18.868661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:40.839893+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:19.233314+0000","last_change":"2024-03-26T03:10:19.233349+0000","last_active":"2024-03-26T03:10:19.233314+0000","last_peered":"2024-03-26T03:10:19.233314+0000","last_clean":"2024-03-26T03:10:19.233314+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:10:19.233314+0000","last_undegraded":"2024-03-26T03:10:19.233314+0000","last_fullsized":"2024-03-26T03:10:19.233314+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_clean_scrub_stamp":"2024-03-26T03:10:19.233223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:22.070407+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":407,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056555+0000","last_change":"2024-03-26T03:09:48.566148+0000","last_active":"2024-03-26T03:10:09.056555+0000","last_peered":"2024-03-26T03:10:09.056555+0000","last_clean":"2024-03-26T03:10:09.056555+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:10:09.056555+0000","last_undegraded":"2024-03-26T03:10:09.056555+0000","last_fullsized":"2024-03-26T03:10:09.056555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:48.563341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:48.563341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:56.090974+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":339,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.058291+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:10:09.058291+0000","last_peered":"2024-03-26T03:10:09.058291+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:10:09.058291+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051196+0000","last_change":"2024-03-26T03:09:57.334690+0000","last_active":"2024-03-26T03:10:09.051196+0000","last_peered":"2024-03-26T03:10:09.051196+0000","last_clean":"2024-03-26T03:10:09.051196+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:10:09.051196+0000","last_undegraded":"2024-03-26T03:10:09.051196+0000","last_fullsized":"2024-03-26T03:10:09.051196+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:57.334539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.334539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:17.848136+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:14.660933+0000","last_change":"2024-03-26T03:10:14.660979+0000","last_active":"2024-03-26T03:10:14.660933+0000","last_peered":"2024-03-26T03:10:14.660933+0000","last_clean":"2024-03-26T03:10:14.660933+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:10:14.660933+0000","last_undegraded":"2024-03-26T03:10:14.660933+0000","last_fullsized":"2024-03-26T03:10:14.660933+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:14.660845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.660845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:43.846150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.055751+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:10:09.055751+0000","last_peered":"2024-03-26T03:10:09.055751+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:10:09.055751+0000","last_undegraded":"2024-03-26T03:10:09.055751+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":445,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.057117+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:10:09.057117+0000","last_peered":"2024-03-26T03:10:09.057117+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:10:09.057117+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.194415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":426,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.058355+0000","last_change":"2024-03-26T03:09:08.928841+0000","last_active":"2024-03-26T03:10:09.058355+0000","last_peered":"2024-03-26T03:10:09.058355+0000","last_clean":"2024-03-26T03:10:09.058355+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:10:09.058355+0000","last_undegraded":"2024-03-26T03:10:09.058355+0000","last_fullsized":"2024-03-26T03:10:09.058355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:08.928693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:09:08.928693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:31.466475+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":560,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:12.146378+0000","last_change":"2024-03-26T03:09:14.129891+0000","last_active":"2024-03-26T03:10:12.146378+0000","last_peered":"2024-03-26T03:10:12.146378+0000","last_clean":"2024-03-26T03:10:12.146378+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:10:12.146378+0000","last_undegraded":"2024-03-26T03:10:12.146378+0000","last_fullsized":"2024-03-26T03:10:12.146378+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:09:14.129763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:14.129763+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:18.260599+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056390+0000","last_change":"2024-03-26T03:09:19.677230+0000","last_active":"2024-03-26T03:10:09.056390+0000","last_peered":"2024-03-26T03:10:09.056390+0000","last_clean":"2024-03-26T03:10:09.056390+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:10:09.056390+0000","last_undegraded":"2024-03-26T03:10:09.056390+0000","last_fullsized":"2024-03-26T03:10:09.056390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:09:19.677094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:19.677094+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:25.321802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":459,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051376+0000","last_change":"2024-03-26T03:09:47.385319+0000","last_active":"2024-03-26T03:10:09.051376+0000","last_peered":"2024-03-26T03:10:09.051376+0000","last_clean":"2024-03-26T03:10:09.051376+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:10:09.051376+0000","last_undegraded":"2024-03-26T03:10:09.051376+0000","last_fullsized":"2024-03-26T03:10:09.051376+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:09:47.384690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.384690+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:08.837736+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":112,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.050352+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:10:09.050352+0000","last_peered":"2024-03-26T03:10:09.050352+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:10:09.050352+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":420,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:20.281119+0000","last_change":"2024-03-26T03:10:20.281155+0000","last_active":"2024-03-26T03:10:20.281119+0000","last_peered":"2024-03-26T03:10:20.281119+0000","last_clean":"2024-03-26T03:10:20.281119+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:10:20.281119+0000","last_undegraded":"2024-03-26T03:10:20.281119+0000","last_fullsized":"2024-03-26T03:10:20.281119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:10:20.281044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:33.169884+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":399,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.050488+0000","last_change":"2024-03-26T03:09:06.593694+0000","last_active":"2024-03-26T03:10:09.050488+0000","last_peered":"2024-03-26T03:10:09.050488+0000","last_clean":"2024-03-26T03:10:09.050488+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:10:09.050488+0000","last_undegraded":"2024-03-26T03:10:09.050488+0000","last_fullsized":"2024-03-26T03:10:09.050488+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:06.593565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:06.593565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:25.868873+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":445,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.047784+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:10:09.047784+0000","last_peered":"2024-03-26T03:10:09.047784+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:10:09.047784+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.366294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":427,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049595+0000","last_change":"2024-03-26T03:09:50.345366+0000","last_active":"2024-03-26T03:10:09.049595+0000","last_peered":"2024-03-26T03:10:09.049595+0000","last_clean":"2024-03-26T03:10:09.049595+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:10:09.049595+0000","last_undegraded":"2024-03-26T03:10:09.049595+0000","last_fullsized":"2024-03-26T03:10:09.049595+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:50.345244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:50.345244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:57.587663+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.051242+0000","last_change":"2024-03-26T03:09:14.479290+0000","last_active":"2024-03-26T03:10:09.051242+0000","last_peered":"2024-03-26T03:10:09.051242+0000","last_clean":"2024-03-26T03:10:09.051242+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:10:09.051242+0000","last_undegraded":"2024-03-26T03:10:09.051242+0000","last_fullsized":"2024-03-26T03:10:09.051242+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:09:14.479135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:25.344597+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":411,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.056952+0000","last_change":"2024-03-26T03:09:13.875355+0000","last_active":"2024-03-26T03:10:24.056952+0000","last_peered":"2024-03-26T03:10:24.056952+0000","last_clean":"2024-03-26T03:10:24.056952+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:10:24.056952+0000","last_undegraded":"2024-03-26T03:10:24.056952+0000","last_fullsized":"2024-03-26T03:10:24.056952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:13.875126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:13.875126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:37.516506+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":167,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.051032+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:10:09.051032+0000","last_peered":"2024-03-26T03:10:09.051032+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:10:09.051032+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:10.238703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.057364+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:10:09.057364+0000","last_peered":"2024-03-26T03:10:09.057364+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:10:09.057364+0000","last_undegraded":"2024-03-26T03:10:09.057364+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":393,"reported_epoch":248,"state":"active+undersized","last_fresh":"2024-03-26T03:10:24.061169+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:10:24.061169+0000","last_peered":"2024-03-26T03:10:24.061169+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:10:24.061169+0000","last_undegraded":"2024-03-26T03:10:24.061169+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.051480+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:10:09.051480+0000","last_peered":"2024-03-26T03:10:09.051480+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:10:09.051480+0000","last_undegraded":"2024-03-26T03:10:09.051480+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":460,"reported_epoch":248,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:24.055888+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:10:24.055888+0000","last_peered":"2024-03-26T03:10:24.055888+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:10:24.055888+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566323+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.054920+0000","last_change":"2024-03-26T03:10:04.678447+0000","last_active":"2024-03-26T03:10:09.054920+0000","last_peered":"2024-03-26T03:10:09.054920+0000","last_clean":"2024-03-26T03:10:09.054920+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:10:09.054920+0000","last_undegraded":"2024-03-26T03:10:09.054920+0000","last_fullsized":"2024-03-26T03:10:09.054920+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:04.678325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:10:04.678325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:32.014387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":418,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.055639+0000","last_change":"2024-03-26T03:09:47.952586+0000","last_active":"2024-03-26T03:10:09.055639+0000","last_peered":"2024-03-26T03:10:09.055639+0000","last_clean":"2024-03-26T03:10:09.055639+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:10:09.055639+0000","last_undegraded":"2024-03-26T03:10:09.055639+0000","last_fullsized":"2024-03-26T03:10:09.055639+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:47.952473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.952473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:15.178094+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":415,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.055832+0000","last_change":"2024-03-26T03:09:59.221838+0000","last_active":"2024-03-26T03:10:09.055832+0000","last_peered":"2024-03-26T03:10:09.055832+0000","last_clean":"2024-03-26T03:10:09.055832+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:10:09.055832+0000","last_undegraded":"2024-03-26T03:10:09.055832+0000","last_fullsized":"2024-03-26T03:10:09.055832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:59.221596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:59.221596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:08.789614+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":422,"reported_epoch":248,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:24.056454+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:10:24.056454+0000","last_peered":"2024-03-26T03:10:24.056454+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:10:24.056454+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566296+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":419,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.058625+0000","last_change":"2024-03-26T03:10:06.687931+0000","last_active":"2024-03-26T03:10:09.058625+0000","last_peered":"2024-03-26T03:10:09.058625+0000","last_clean":"2024-03-26T03:10:09.058625+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:10:09.058625+0000","last_undegraded":"2024-03-26T03:10:09.058625+0000","last_fullsized":"2024-03-26T03:10:09.058625+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:06.687821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:10:06.687821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:09.046893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":411,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.056283+0000","last_change":"2024-03-26T03:10:02.681646+0000","last_active":"2024-03-26T03:10:09.056283+0000","last_peered":"2024-03-26T03:10:09.056283+0000","last_clean":"2024-03-26T03:10:09.056283+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:10:09.056283+0000","last_undegraded":"2024-03-26T03:10:09.056283+0000","last_fullsized":"2024-03-26T03:10:09.056283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:02.681454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:02.681454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:24.670985+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":393,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.062016+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:10:09.062016+0000","last_peered":"2024-03-26T03:10:09.062016+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:10:09.062016+0000","last_undegraded":"2024-03-26T03:10:09.062016+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":455,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.055600+0000","last_change":"2024-03-26T03:09:57.820889+0000","last_active":"2024-03-26T03:10:24.055600+0000","last_peered":"2024-03-26T03:10:24.055600+0000","last_clean":"2024-03-26T03:10:24.055600+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:10:24.055600+0000","last_undegraded":"2024-03-26T03:10:24.055600+0000","last_fullsized":"2024-03-26T03:10:24.055600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:09:57.820746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.820746+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:07.131541+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":105,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.057021+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:10:09.057021+0000","last_peered":"2024-03-26T03:10:09.057021+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:10:09.057021+0000","last_undegraded":"2024-03-26T03:10:09.057021+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:21.237260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":403,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:09.049552+0000","last_change":"2024-03-26T03:09:17.407289+0000","last_active":"2024-03-26T03:10:09.049552+0000","last_peered":"2024-03-26T03:10:09.049552+0000","last_clean":"2024-03-26T03:10:09.049552+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:10:09.049552+0000","last_undegraded":"2024-03-26T03:10:09.049552+0000","last_fullsized":"2024-03-26T03:10:09.049552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:17.407186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:17.407186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:46.296854+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":410,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.055826+0000","last_change":"2024-03-26T03:09:53.849576+0000","last_active":"2024-03-26T03:10:24.055826+0000","last_peered":"2024-03-26T03:10:24.055826+0000","last_clean":"2024-03-26T03:10:24.055826+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:10:24.055826+0000","last_undegraded":"2024-03-26T03:10:24.055826+0000","last_fullsized":"2024-03-26T03:10:24.055826+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:53.849440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:53.849440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:16.864580+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":215,"reported_epoch":246,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:10:09.047343+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:10:09.047343+0000","last_peered":"2024-03-26T03:10:09.047343+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:10:09.047343+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:11.366374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":428,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-03-26T03:10:11.215214+0000","last_change":"2024-03-26T03:10:11.215249+0000","last_active":"2024-03-26T03:10:11.215214+0000","last_peered":"2024-03-26T03:10:11.215214+0000","last_clean":"2024-03-26T03:10:11.215214+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:10:11.215214+0000","last_undegraded":"2024-03-26T03:10:11.215214+0000","last_fullsized":"2024-03-26T03:10:11.215214+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:11.215134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:11.215134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:33.383839+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":414,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:10:24.055931+0000","last_change":"2024-03-26T03:09:09.855016+0000","last_active":"2024-03-26T03:10:24.055931+0000","last_peered":"2024-03-26T03:10:24.055931+0000","last_clean":"2024-03-26T03:10:24.055931+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:10:24.055931+0000","last_undegraded":"2024-03-26T03:10:24.055931+0000","last_fullsized":"2024-03-26T03:10:24.055931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:09.854936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:09.854936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:28.566406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":104,"reported_epoch":246,"state":"active+undersized","last_fresh":"2024-03-26T03:10:09.056800+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:10:09.056800+0000","last_peered":"2024-03-26T03:10:09.056800+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:10:09.056800+0000","last_undegraded":"2024-03-26T03:10:09.056800+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:10:09.673811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":9175629824,"num_objects":140009,"num_object_clones":0,"num_object_copies":420027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":69991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":140009,"num_write_kb":8960576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22922657792,"data_stored":22922657792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81483,"ondisk_log_size":81483,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":316358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":28,"acting":28,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":672,"num_read_kb":667,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1410,"num_read_kb":1135,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":940,"ondisk_log_size":940,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103309,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5481996,"kb_used_data":4995460,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":88263156,"statfs":{"total":95995035648,"available":90381471744,"internally_reserved":0,"allocated":5115351040,"data_stored":5114884015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430353,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2033540,"kb_used_data":1677500,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91711612,"statfs":{"total":95995035648,"available":93912690688,"internally_reserved":0,"allocated":1717760000,"data_stored":1717300549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5539508,"kb_used_data":4998828,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":88205644,"statfs":{"total":95995035648,"available":90322579456,"internally_reserved":0,"allocated":5118799872,"data_stored":5118293617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2682440,"kb_used_data":2240320,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91062712,"statfs":{"total":95995035648,"available":93248217088,"internally_reserved":0,"allocated":2294087680,"data_stored":2293624876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043272,"kb_used_data":1669184,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91701880,"statfs":{"total":95995035648,"available":93902725120,"internally_reserved":0,"allocated":1709244416,"data_stored":1708782470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986332,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042152,"kb_used_data":1731932,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91703000,"statfs":{"total":95995035648,"available":93903872000,"internally_reserved":0,"allocated":1773498368,"data_stored":1773013631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691991,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5479052,"kb_used_data":4978628,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":88266100,"statfs":{"total":95995035648,"available":90384486400,"internally_reserved":0,"allocated":5098115072,"data_stored":5097648051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2297757696,"data_stored":2297757696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5143592960,"data_stored":5143592960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1711931392,"data_stored":1711931392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1781661696,"data_stored":1781661696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5126160384,"data_stored":5126160384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1717567488,"data_stored":1717567488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5143986176,"data_stored":5143986176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:10:26.133 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T03:10:26.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-26T03:10:26.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:26.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:26.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:26.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:26.489 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.542 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:26.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:26.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:26.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:26.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:26.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:26.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:26.958 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:26.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:27.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:27.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:27.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:27.233 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:27.335 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:27.382 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:27.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:27.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:27.592 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:27 smithi066 ceph-mon[29666]: pgmap v495: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.64k op/s; 72746/437442 objects degraded (16.630%) 2024-03-26T03:10:27.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:27 smithi066 ceph-mon[37920]: pgmap v495: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.64k op/s; 72746/437442 objects degraded (16.630%) 2024-03-26T03:10:27.609 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:27.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:27.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:27.701 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:10:27.701 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:10:27.701 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:10:27.701 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:10:27.701 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:10:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:27 smithi192 ceph-mon[33110]: pgmap v495: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.64k op/s; 72746/437442 objects degraded (16.630%) 2024-03-26T03:10:27.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:27.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:27.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:27.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:27.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:28.027 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:28.066 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:28.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.210 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:28.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:28.311 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:28.347 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:28.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:28.485 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:28.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:28.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:28.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:28.750 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:10:28.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:28.785 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:10:28.819 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:10:28.851 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:28.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:28.860 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:10:28.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:28.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:28.899 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:10:28.942 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:10:28.982 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:10:28.983 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:10:28.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:29.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:29.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:29.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:29.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:29.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:29 smithi066 ceph-mon[29666]: pgmap v496: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 1k op/s; 72746/437442 objects degraded (16.630%) 2024-03-26T03:10:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:29 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1952636768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 72746/437442 objects degraded (16.630%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:29.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:29 smithi066 ceph-mon[37920]: pgmap v496: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 1k op/s; 72746/437442 objects degraded (16.630%) 2024-03-26T03:10:29.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:29 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1952636768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:29.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 72746/437442 objects degraded (16.630%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:29.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:29.444 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:29.531 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:10:29.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:29.588 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:29 smithi192 ceph-mon[33110]: pgmap v496: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.9 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 1k op/s; 72746/437442 objects degraded (16.630%) 2024-03-26T03:10:29.588 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:29 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1952636768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:29.589 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 72746/437442 objects degraded (16.630%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.619 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:29.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:29.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:29.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:29.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:29.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:29.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:29.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:10:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:10:30.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:30.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:30.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:30.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:30.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:30.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:30.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:30.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:30.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:30.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:30.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:30.970 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:31.007 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:31.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:31.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:31.245 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:31.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:31.386 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:31.424 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:31.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:31.575 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[29666]: pgmap v497: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 1.42k op/s; 70159/423804 objects degraded (16.555%) 2024-03-26T03:10:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:10:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:10:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[37920]: pgmap v497: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 1.42k op/s; 70159/423804 objects degraded (16.555%) 2024-03-26T03:10:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:10:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:31 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:10:31.677 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:31.714 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:31.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:31 smithi192 ceph-mon[33110]: pgmap v497: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 50 MiB/s wr, 1.42k op/s; 70159/423804 objects degraded (16.555%) 2024-03-26T03:10:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:31 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:10:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:31 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:10:31.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:31.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:31.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:31.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:10:32.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:10:32.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:32.133 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:32.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:32.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:32.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:32.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:32.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:32.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:32.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:32 smithi066 ceph-mon[29666]: osdmap e249: 8 total, 7 up, 8 in 2024-03-26T03:10:32.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:32 smithi066 ceph-mon[37920]: osdmap e249: 8 total, 7 up, 8 in 2024-03-26T03:10:32.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:32.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:32 smithi192 ceph-mon[33110]: osdmap e249: 8 total, 7 up, 8 in 2024-03-26T03:10:32.784 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:32.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:32.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:32.983 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:32.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:33.005 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:33.085 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:33.133 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:33.235 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.278 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:33.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:33.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:33 smithi066 ceph-mon[29666]: pgmap v499: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 1.69k op/s; 69728/418368 objects degraded (16.667%) 2024-03-26T03:10:33.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:33 smithi066 ceph-mon[37920]: pgmap v499: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 1.69k op/s; 69728/418368 objects degraded (16.667%) 2024-03-26T03:10:33.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:33.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:33.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:33.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:33.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:33.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:33 smithi192 ceph-mon[33110]: pgmap v499: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 1.69k op/s; 69728/418368 objects degraded (16.667%) 2024-03-26T03:10:33.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:33.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:33.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:33.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:33.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:34.035 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:10:34.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:34.070 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:10:34.085 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:10:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:10:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:10:34.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:34.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:34.237 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:10:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=cleanup t=2024-03-26T03:10:33.951732532Z level=info msg="Completed cleanup jobs" duration=2.257484ms 2024-03-26T03:10:34.238 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:10:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=sqlstore.transactions t=2024-03-26T03:10:33.960002351Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T03:10:34.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:34.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:34.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1962828367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 69728/418368 objects degraded (16.667%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1962828367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[29666]: osdmap e250: 8 total, 7 up, 8 in 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1962828367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 69728/418368 objects degraded (16.667%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1962828367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:10:34.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:34 smithi066 ceph-mon[37920]: osdmap e250: 8 total, 7 up, 8 in 2024-03-26T03:10:34.383 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:34.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:34.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:34.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:34.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1962828367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:10:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 69728/418368 objects degraded (16.667%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:34 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1962828367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:10:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:34 smithi192 ceph-mon[33110]: osdmap e250: 8 total, 7 up, 8 in 2024-03-26T03:10:34.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:34.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:34.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:34.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:34.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:35.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.069 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:35.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:35.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:35.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:35.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:35.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:35.446 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:35.483 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:35.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:35 smithi066 ceph-mon[29666]: pgmap v501: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 3.9 MiB/s wr, 977 op/s; 69267/416985 objects degraded (16.611%) 2024-03-26T03:10:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:35 smithi066 ceph-mon[37920]: pgmap v501: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 3.9 MiB/s wr, 977 op/s; 69267/416985 objects degraded (16.611%) 2024-03-26T03:10:35.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:35.723 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:35 smithi192 ceph-mon[33110]: pgmap v501: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 3.9 MiB/s wr, 977 op/s; 69267/416985 objects degraded (16.611%) 2024-03-26T03:10:35.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:35.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:35.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:35.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:36.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:36.296 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:36.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:36.397 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:36.446 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:36.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:36.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:36.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:36.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:36.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:36.724 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:36.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:36.825 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:36.877 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:36.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:36.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:37.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:37.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:37.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:37.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:37.346 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 1 pg_num to 21 2024-03-26T03:10:37.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:37.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:37.525 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:37.570 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:37 smithi066 ceph-mon[29666]: pgmap v502: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.9 MiB/s wr, 1.84k op/s; 66084/396237 objects degraded (16.678%) 2024-03-26T03:10:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3566596718' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-26T03:10:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:37 smithi066 ceph-mon[29666]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-26T03:10:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:37 smithi066 ceph-mon[37920]: pgmap v502: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.9 MiB/s wr, 1.84k op/s; 66084/396237 objects degraded (16.678%) 2024-03-26T03:10:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3566596718' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-26T03:10:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:37 smithi066 ceph-mon[37920]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-26T03:10:37.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:37.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:37 smithi192 ceph-mon[33110]: pgmap v502: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.9 MiB/s wr, 1.84k op/s; 66084/396237 objects degraded (16.678%) 2024-03-26T03:10:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3566596718' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-26T03:10:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:37 smithi192 ceph-mon[33110]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-26T03:10:37.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:37.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:37.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:37.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:37.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:38.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:38.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:38.214 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:38.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:38.281 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:10:38.281 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:10:38.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:38.399 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:38.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:38.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:38 smithi066 ceph-mon[29666]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-26T03:10:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:38 smithi066 ceph-mon[29666]: osdmap e251: 8 total, 7 up, 8 in 2024-03-26T03:10:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:38 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-26T03:10:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:38 smithi066 ceph-mon[37920]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-26T03:10:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:38 smithi066 ceph-mon[37920]: osdmap e251: 8 total, 7 up, 8 in 2024-03-26T03:10:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:38 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-26T03:10:38.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:38.674 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:38 smithi192 ceph-mon[33110]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-26T03:10:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:38 smithi192 ceph-mon[33110]: osdmap e251: 8 total, 7 up, 8 in 2024-03-26T03:10:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:38 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-26T03:10:38.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:38.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:38.912 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:38.921 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:38.958 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:38.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:39.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:39.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:39.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.235 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:39.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:39.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:39.379 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:39.437 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:10:39.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.523 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:39.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:39.624 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:39.672 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:39.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:10:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:10:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:39 smithi192 ceph-mon[33110]: pgmap v504: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.07k op/s; 66084/396237 objects degraded (16.678%) 2024-03-26T03:10:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-26T03:10:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:39 smithi192 ceph-mon[33110]: osdmap e252: 8 total, 7 up, 8 in 2024-03-26T03:10:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 66084/396237 objects degraded (16.678%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:39.774 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:39.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[29666]: pgmap v504: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.07k op/s; 66084/396237 objects degraded (16.678%) 2024-03-26T03:10:39.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-26T03:10:39.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[29666]: osdmap e252: 8 total, 7 up, 8 in 2024-03-26T03:10:39.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 66084/396237 objects degraded (16.678%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:39.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:39.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[37920]: pgmap v504: 158 pgs: 30 active+undersized+degraded, 38 active+undersized, 90 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.07k op/s; 66084/396237 objects degraded (16.678%) 2024-03-26T03:10:39.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-26T03:10:39.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[37920]: osdmap e252: 8 total, 7 up, 8 in 2024-03-26T03:10:39.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 66084/396237 objects degraded (16.678%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:39.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:39.816 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:39.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:10:39.918 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:39.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:39.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:40.078 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:40.120 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:40.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:40.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:40.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:40.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:40.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:40.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:40.590 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:40.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[29666]: osdmap e253: 8 total, 7 up, 8 in 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[37920]: osdmap e253: 8 total, 7 up, 8 in 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:10:40.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-26T03:10:40.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:40.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:40 smithi192 ceph-mon[33110]: osdmap e253: 8 total, 7 up, 8 in 2024-03-26T03:10:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:40 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:10:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:40 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:10:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-26T03:10:40.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:40.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:40.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:40.950 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:40.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:41.051 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:41.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:41.192 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:41.229 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:41.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:41.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:41.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:41.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:41.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:41.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:41 smithi192 ceph-mon[33110]: pgmap v507: 168 pgs: 8 unknown, 6 peering, 30 active+undersized+degraded, 35 active+undersized, 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.07k op/s; 62858/379737 objects degraded (16.553%) 2024-03-26T03:10:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-26T03:10:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:41 smithi192 ceph-mon[33110]: osdmap e254: 8 total, 7 up, 8 in 2024-03-26T03:10:41.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:41.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:41.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:41.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[29666]: pgmap v507: 168 pgs: 8 unknown, 6 peering, 30 active+undersized+degraded, 35 active+undersized, 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.07k op/s; 62858/379737 objects degraded (16.553%) 2024-03-26T03:10:41.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-26T03:10:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[29666]: osdmap e254: 8 total, 7 up, 8 in 2024-03-26T03:10:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[37920]: pgmap v507: 168 pgs: 8 unknown, 6 peering, 30 active+undersized+degraded, 35 active+undersized, 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.07k op/s; 62858/379737 objects degraded (16.553%) 2024-03-26T03:10:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-26T03:10:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:41 smithi066 ceph-mon[37920]: osdmap e254: 8 total, 7 up, 8 in 2024-03-26T03:10:41.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:41.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:41.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:42.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.066 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:42.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:42.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:42.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:42.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:42.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:42.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:42.472 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:42.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:42.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:42.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:42 smithi192 ceph-mon[33110]: osdmap e255: 8 total, 7 up, 8 in 2024-03-26T03:10:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:42 smithi192 ceph-mon[33110]: osdmap e256: 8 total, 7 up, 8 in 2024-03-26T03:10:42.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:42.749 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:42 smithi066 ceph-mon[29666]: osdmap e255: 8 total, 7 up, 8 in 2024-03-26T03:10:42.749 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:42 smithi066 ceph-mon[29666]: osdmap e256: 8 total, 7 up, 8 in 2024-03-26T03:10:42.749 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:42 smithi066 ceph-mon[37920]: osdmap e255: 8 total, 7 up, 8 in 2024-03-26T03:10:42.749 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:42 smithi066 ceph-mon[37920]: osdmap e256: 8 total, 7 up, 8 in 2024-03-26T03:10:42.846 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:42.889 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:42.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:42.960 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:10:42.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:42.995 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:10:43.030 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:10:43.031 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:43.072 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:10:43.113 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:10:43.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:43.153 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:10:43.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:43.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:43.193 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:10:43.193 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:10:43.193 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:10:43.193 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:10:43.193 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:10:43.193 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:10:43.194 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:10:43.194 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:10:43.194 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:10:43.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:43.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:43.405 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.463 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:43.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:43.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:43.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:43.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:43.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:43 smithi192 ceph-mon[33110]: pgmap v510: 168 pgs: 8 unknown, 6 peering, 30 active+undersized+degraded, 35 active+undersized, 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.84k op/s; 62399/374091 objects degraded (16.680%) 2024-03-26T03:10:43.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:43 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4098259108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:43.748 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:10:43.748 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:10:43.748 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-26T03:10:43.748 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:10:43.748 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:10:43.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-26T03:10:43.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:43.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:43 smithi066 ceph-mon[37920]: pgmap v510: 168 pgs: 8 unknown, 6 peering, 30 active+undersized+degraded, 35 active+undersized, 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.84k op/s; 62399/374091 objects degraded (16.680%) 2024-03-26T03:10:43.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:43 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4098259108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:43.777 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:43 smithi066 ceph-mon[29666]: pgmap v510: 168 pgs: 8 unknown, 6 peering, 30 active+undersized+degraded, 35 active+undersized, 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.84k op/s; 62399/374091 objects degraded (16.680%) 2024-03-26T03:10:43.777 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:43 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4098259108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:10:43.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:43.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:43.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:43.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:44.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:44.100 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:10:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:10:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:10:44.113 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:44.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:44.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:10:44.161 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:10:44.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:44.303 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:44.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:44.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:44.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:44.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:44.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:44.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:44 smithi192 ceph-mon[33110]: osdmap e257: 8 total, 7 up, 8 in 2024-03-26T03:10:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 62399/374091 objects degraded (16.680%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:44 smithi192 ceph-mon[33110]: osdmap e258: 8 total, 7 up, 8 in 2024-03-26T03:10:44.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:44.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:44 smithi066 ceph-mon[29666]: osdmap e257: 8 total, 7 up, 8 in 2024-03-26T03:10:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 62399/374091 objects degraded (16.680%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:44 smithi066 ceph-mon[29666]: osdmap e258: 8 total, 7 up, 8 in 2024-03-26T03:10:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:44 smithi066 ceph-mon[37920]: osdmap e257: 8 total, 7 up, 8 in 2024-03-26T03:10:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 62399/374091 objects degraded (16.680%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:44.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:44 smithi066 ceph-mon[37920]: osdmap e258: 8 total, 7 up, 8 in 2024-03-26T03:10:44.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:44.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:44.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:45.096 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:45.132 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:45.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:45.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:45.370 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:45.408 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:45.510 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:45.546 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:45.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:45.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:45.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:45 smithi066 ceph-mon[29666]: pgmap v514: 168 pgs: 1 active+clean+remapped, 7 unknown, 5 peering, 30 active+undersized+degraded, 35 active+undersized, 90 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 9.4 KiB/s wr, 624 op/s; 61894/372576 objects degraded (16.612%); 6.8 KiB/s, 0 objects/s recovering 2024-03-26T03:10:45.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:45 smithi066 ceph-mon[37920]: pgmap v514: 168 pgs: 1 active+clean+remapped, 7 unknown, 5 peering, 30 active+undersized+degraded, 35 active+undersized, 90 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 9.4 KiB/s wr, 624 op/s; 61894/372576 objects degraded (16.612%); 6.8 KiB/s, 0 objects/s recovering 2024-03-26T03:10:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:45 smithi192 ceph-mon[33110]: pgmap v514: 168 pgs: 1 active+clean+remapped, 7 unknown, 5 peering, 30 active+undersized+degraded, 35 active+undersized, 90 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 9.4 KiB/s wr, 624 op/s; 61894/372576 objects degraded (16.612%); 6.8 KiB/s, 0 objects/s recovering 2024-03-26T03:10:45.789 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:45.827 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:45.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:45.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:45.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:46.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:46.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:46.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:10:46.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:46.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:46.344 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:46.384 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:46.458 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:10:46.458 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:10:46.458 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2943754575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:10:46.458 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[29666]: from='client.25345 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:10:46.459 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:10:46.459 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:10:46.459 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2943754575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:10:46.459 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:46 smithi066 ceph-mon[37920]: from='client.25345 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:10:46.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:46.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:46.610 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:10:46.616 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:46.655 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:46 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:10:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:46 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:10:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2943754575' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:10:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:46 smithi192 ceph-mon[33110]: from='client.25345 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:10:46.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:46.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:46.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:46.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:46.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:47.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:47.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:47.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:47.215 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:47.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:47.363 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:47.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:47.508 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[29666]: pgmap v515: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 12 KiB/s rd, 84 KiB/s wr, 1.61k op/s; 58616/350985 objects degraded (16.700%); 60 KiB/s, 0 objects/s recovering 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[29666]: from='client.25345 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[29666]: osdmap e259: 8 total, 7 up, 8 in 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[37920]: pgmap v515: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 12 KiB/s rd, 84 KiB/s wr, 1.61k op/s; 58616/350985 objects degraded (16.700%); 60 KiB/s, 0 objects/s recovering 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[37920]: from='client.25345 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[37920]: osdmap e259: 8 total, 7 up, 8 in 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:10:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:47 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:10:47.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:47.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:47 smithi192 ceph-mon[33110]: pgmap v515: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 12 KiB/s rd, 84 KiB/s wr, 1.61k op/s; 58616/350985 objects degraded (16.700%); 60 KiB/s, 0 objects/s recovering 2024-03-26T03:10:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:47 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:10:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:47 smithi192 ceph-mon[33110]: from='client.25345 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:10:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:47 smithi192 ceph-mon[33110]: osdmap e259: 8 total, 7 up, 8 in 2024-03-26T03:10:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:47 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:10:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:47 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:10:47.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:47.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:47.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:47.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:47.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:48.026 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.067 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:48.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:48.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:48.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:48.308 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.362 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:48.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:48.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:48.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:48.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:10:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:10:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:10:48.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:10:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:10:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:10:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:10:48.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:48 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:10:48.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:10:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:48 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:10:48.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:48.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:48.833 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:48.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:48.919 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:48.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:49.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:49.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:49.157 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:49.210 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:49.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:49.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:49.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:49.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:49.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:49.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:183349 9.2 e260/225 9:4a2de957:::benchmark_data_smithi066_125102_object37048:head v 260'11403, mlcod=260'11400)", 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:10:49.675902+0000", 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.00025765, 2024-03-26T03:10:49.681 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000317013, 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675902+0000", 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675902+0000", 2024-03-26T03:10:49.682 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675903+0000", 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.054e-06 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675909+0000", 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0650000000000004e-06 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675918+0000", 2024-03-26T03:10:49.683 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4710000000000007e-06 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675923+0000", 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5890000000000002e-06 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.675986+0000", 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2142000000000002e-05 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:10:49.684 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:10:49.676006+0000", 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0091000000000001e-05 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 1 2024-03-26T03:10:49.685 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:10:49.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:49.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:10:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: 1.13 scrub starts 2024-03-26T03:10:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: 1.13 scrub ok 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: pgmap v517: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 9.6 KiB/s rd, 69 KiB/s wr, 1.33k op/s; 58616/350985 objects degraded (16.700%); 49 KiB/s, 0 objects/s recovering 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 58616/350985 objects degraded (16.700%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:49 smithi192 ceph-mon[33110]: osdmap e260: 8 total, 7 up, 8 in 2024-03-26T03:10:49.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: 1.13 scrub starts 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: 1.13 scrub ok 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: pgmap v517: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 9.6 KiB/s rd, 69 KiB/s wr, 1.33k op/s; 58616/350985 objects degraded (16.700%); 49 KiB/s, 0 objects/s recovering 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:10:49.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 58616/350985 objects degraded (16.700%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[29666]: osdmap e260: 8 total, 7 up, 8 in 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: 1.13 scrub starts 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: 1.13 scrub ok 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: pgmap v517: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 9.6 KiB/s rd, 69 KiB/s wr, 1.33k op/s; 58616/350985 objects degraded (16.700%); 49 KiB/s, 0 objects/s recovering 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 58616/350985 objects degraded (16.700%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:49.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:49 smithi066 ceph-mon[37920]: osdmap e260: 8 total, 7 up, 8 in 2024-03-26T03:10:49.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:49.901 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:49.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:50.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:50.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:50.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:50.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:50.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:50.317 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:50.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:50.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:50.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:50.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:50.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:50.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:10:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:10:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 1.f scrub starts 2024-03-26T03:10:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 1.f scrub ok 2024-03-26T03:10:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 1.10 scrub starts 2024-03-26T03:10:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 1.10 scrub ok 2024-03-26T03:10:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:10:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:50 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:10:50.822 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:50.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 1.f scrub starts 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 1.f scrub ok 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 1.10 scrub starts 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 1.10 scrub ok 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:10:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 1.f scrub starts 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 1.f scrub ok 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 1.10 scrub starts 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 1.10 scrub ok 2024-03-26T03:10:50.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:10:50.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:50 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:10:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:50.862 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:50.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:50.883 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:10:50.963 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.026 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:51.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:51.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:51.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:51.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:51.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:51.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:51.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:51.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:51.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: pgmap v519: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 9.9 KiB/s rd, 66 KiB/s wr, 2.19k op/s; 55239/333858 objects degraded (16.546%); 47 KiB/s, 0 objects/s recovering 2024-03-26T03:10:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.c scrub starts 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.c scrub ok 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.e scrub starts 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.e scrub ok 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.d scrub starts 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.d scrub ok 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.14 scrub starts 2024-03-26T03:10:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[29666]: 1.14 scrub ok 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: pgmap v519: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 9.9 KiB/s rd, 66 KiB/s wr, 2.19k op/s; 55239/333858 objects degraded (16.546%); 47 KiB/s, 0 objects/s recovering 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.c scrub starts 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.c scrub ok 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.e scrub starts 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.e scrub ok 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.d scrub starts 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.d scrub ok 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.14 scrub starts 2024-03-26T03:10:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:51 smithi066 ceph-mon[37920]: 1.14 scrub ok 2024-03-26T03:10:51.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:51.685 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:10:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:51.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:10:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: pgmap v519: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 9.9 KiB/s rd, 66 KiB/s wr, 2.19k op/s; 55239/333858 objects degraded (16.546%); 47 KiB/s, 0 objects/s recovering 2024-03-26T03:10:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.c scrub starts 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.c scrub ok 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.e scrub starts 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.e scrub ok 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.d scrub starts 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.d scrub ok 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.14 scrub starts 2024-03-26T03:10:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:51 smithi192 ceph-mon[33110]: 1.14 scrub ok 2024-03-26T03:10:51.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.856 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:51.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:51.914 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:51.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:51.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:51.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:52.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:52.137 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:52.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:52.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:52.239 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:52.314 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:52.315 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:52.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:52.322 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:10:52.322 INFO:teuthology.orchestra.run.smithi066.stdout: "success": "" 2024-03-26T03:10:52.322 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:10:52.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:52.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:52.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:52.552 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:52.593 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:52.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:52.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:52 smithi192 ceph-mon[33110]: 1.b scrub starts 2024-03-26T03:10:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:52 smithi192 ceph-mon[33110]: 1.b scrub ok 2024-03-26T03:10:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:52 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:10:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:52 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:10:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:52 smithi192 ceph-mon[33110]: 1.5 scrub starts 2024-03-26T03:10:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:52 smithi192 ceph-mon[33110]: 1.5 scrub ok 2024-03-26T03:10:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:52.740 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:52.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:52.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[29666]: 1.b scrub starts 2024-03-26T03:10:52.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[29666]: 1.b scrub ok 2024-03-26T03:10:52.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:10:52.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:10:52.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[29666]: 1.5 scrub starts 2024-03-26T03:10:52.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[29666]: 1.5 scrub ok 2024-03-26T03:10:52.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[37920]: 1.b scrub starts 2024-03-26T03:10:52.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[37920]: 1.b scrub ok 2024-03-26T03:10:52.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:10:52.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:10:52.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[37920]: 1.5 scrub starts 2024-03-26T03:10:52.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:52 smithi066 ceph-mon[37920]: 1.5 scrub ok 2024-03-26T03:10:52.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:52.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:52.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:52.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:53.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:53.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:53.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:53.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:53.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:53.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:53.427 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:53.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:53.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:53.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:53.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:53 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:10:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:53 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:10:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:53 smithi192 ceph-mon[33110]: pgmap v520: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.85k op/s; 54772/328245 objects degraded (16.686%); 1/328245 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-26T03:10:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:53 smithi192 ceph-mon[33110]: 9.d scrub starts 2024-03-26T03:10:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:53 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:10:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:53 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:10:53.801 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:53.835 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:10:53.835 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:10:53.835 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[29666]: pgmap v520: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.85k op/s; 54772/328245 objects degraded (16.686%); 1/328245 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-26T03:10:53.835 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[29666]: 9.d scrub starts 2024-03-26T03:10:53.835 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[37920]: pgmap v520: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.85k op/s; 54772/328245 objects degraded (16.686%); 1/328245 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[37920]: 9.d scrub starts 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:10:53.836 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:10:53.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:53.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:53.980 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:53.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:54.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:54.095 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:10:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:10:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:10:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.131 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:54.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:54.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:54.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:54.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:54.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:54.505 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:54.544 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:54.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:54.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:54.713 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:10:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 54772/328245 objects degraded (16.686%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:54.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:54.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:54.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 54772/328245 objects degraded (16.686%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:54.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 54772/328245 objects degraded (16.686%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:10:54.918 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:54.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:54.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:54.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:55.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:55.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:55.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:55.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:55.252 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:55.354 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:55.404 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:55.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:55.544 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: pgmap v521: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 54312/326865 objects degraded (16.616%); 1/326865 objects misplaced (0.000%) 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:10:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 9.d scrub ok 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: pgmap v521: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 54312/326865 objects degraded (16.616%); 1/326865 objects misplaced (0.000%) 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:10:55.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:10:55.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:10:55.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:10:55.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 9.d scrub ok 2024-03-26T03:10:55.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:10:55.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:55 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:10:55.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:55.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:10:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: pgmap v521: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 54312/326865 objects degraded (16.616%); 1/326865 objects misplaced (0.000%) 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 9.d scrub ok 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:10:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:55 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:10:55.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.829 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:55.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:55.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:55.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:55.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:56.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:56.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:56.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:56.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:56.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:56.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:56.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:56.516 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:10:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[29666]: 9.b scrub starts 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[37920]: 9.b scrub starts 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:10:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:56 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:10:56.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:56.659 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:56.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:56 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:10:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:56 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:10:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:56 smithi192 ceph-mon[33110]: 9.b scrub starts 2024-03-26T03:10:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:56 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:10:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:56 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:10:56.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:56.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:56.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:56.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:56.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:57.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:57.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:57.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:57.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:57.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:57.357 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:57.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:57.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:57.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:57.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:57.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:57.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:57.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[29666]: pgmap v522: 168 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.53k op/s; 50982/306051 objects degraded (16.658%); 1/306051 objects misplaced (0.000%) 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[37920]: pgmap v522: 168 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.53k op/s; 50982/306051 objects degraded (16.658%); 1/306051 objects misplaced (0.000%) 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:10:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:57 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:10:57.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:10:57.900 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:57.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:10:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:57 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:10:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:57 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:10:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:57 smithi192 ceph-mon[33110]: pgmap v522: 168 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.53k op/s; 50982/306051 objects degraded (16.658%); 1/306051 objects misplaced (0.000%) 2024-03-26T03:10:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:57 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:10:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:57 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:10:58.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:58.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:58.139 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:58.174 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:58.276 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:58.312 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:58.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:10:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:58.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:10:58.548 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:58.585 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:58.686 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:10:58.727 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:10:58.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:10:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[29666]: 9.b scrub ok 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[29666]: pgmap v523: 168 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.49k op/s; 50982/306051 objects degraded (16.658%); 1/306051 objects misplaced (0.000%) 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[37920]: 9.b scrub ok 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:10:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:10:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:58 smithi066 ceph-mon[37920]: pgmap v523: 168 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.49k op/s; 50982/306051 objects degraded (16.658%); 1/306051 objects misplaced (0.000%) 2024-03-26T03:10:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:58.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:58.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:58 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:10:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:58 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:10:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:58 smithi192 ceph-mon[33110]: 9.b scrub ok 2024-03-26T03:10:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:58 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:10:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:58 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:10:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:58 smithi192 ceph-mon[33110]: pgmap v523: 168 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 94 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.49k op/s; 50982/306051 objects degraded (16.658%); 1/306051 objects misplaced (0.000%) 2024-03-26T03:10:58.994 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:58.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:59.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:59.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:59.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:10:59.151 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:10:59.151 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:10:59.151 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:10:59.234 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:10:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:10:59.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:10:59.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:10:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:59.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:10:59.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:59.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:59.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:10:59.688 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:10:59.760 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:59 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:10:59.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:59 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:10:59.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:10:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 50982/306051 objects degraded (16.658%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:59.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:59 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:10:59.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:59 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:10:59.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:10:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 50982/306051 objects degraded (16.658%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:59.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:59.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:59.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:10:59.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:10:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:10:59.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:10:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:59 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:10:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:59 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:10:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:10:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 50982/306051 objects degraded (16.658%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:10:59.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:10:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:11:00.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:00.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:00.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:00.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:00.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:00.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:00.386 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:11:00.429 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:11:00.470 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:11:00.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:00.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:00.511 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:11:00.557 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:11:00.606 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:11:00.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:00.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:00.653 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: pgmap v524: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.82k op/s; 47910/289935 objects degraded (16.524%); 1/289935 objects misplaced (0.000%) 2024-03-26T03:11:00.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3650814880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:11:00.654 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:11:00.655 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: pgmap v524: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.82k op/s; 47910/289935 objects degraded (16.524%); 1/289935 objects misplaced (0.000%) 2024-03-26T03:11:00.655 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3650814880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:00.741 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.786 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:00.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:00.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:00.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:00.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:11:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:11:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:11:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:11:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: pgmap v524: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.82k op/s; 47910/289935 objects degraded (16.524%); 1/289935 objects misplaced (0.000%) 2024-03-26T03:11:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3650814880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:01.021 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:01.063 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:01.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:01.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:01.300 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:01.360 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:01.462 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:01.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.502 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:01.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:01.552 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:11:01.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:01.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:01.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:01.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:11:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:11:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1235845929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[29666]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:11:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:11:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1235845929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:01 smithi066 ceph-mon[37920]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:01.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:01.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:01.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:01 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:11:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:01 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:11:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:01 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1235845929' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:01 smithi192 ceph-mon[33110]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:02.010 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:02.045 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:02.147 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:02.198 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:02.300 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:02.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:02.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:02.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:02.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:02.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: osdmap e261: 8 total, 7 up, 8 in 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[29666]: pgmap v526: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 47468/284658 objects degraded (16.675%); 1/284658 objects misplaced (0.000%) 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: osdmap e261: 8 total, 7 up, 8 in 2024-03-26T03:11:02.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:11:02.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:11:02.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:02 smithi066 ceph-mon[37920]: pgmap v526: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 47468/284658 objects degraded (16.675%); 1/284658 objects misplaced (0.000%) 2024-03-26T03:11:02.698 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:02.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:02.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:02.740 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:11:02.740 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:11:02.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:02.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:02.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:02.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: osdmap e261: 8 total, 7 up, 8 in 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:11:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:02 smithi192 ceph-mon[33110]: pgmap v526: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.45k op/s; 47468/284658 objects degraded (16.675%); 1/284658 objects misplaced (0.000%) 2024-03-26T03:11:03.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:03.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:03.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:03.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:03.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:03.386 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:03.424 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:03.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:03.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:03.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:03.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:11:03.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:11:03.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:03.837 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:03.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:03.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:03.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:04.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:11:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:11:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:11:04.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:04.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:04.217 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:04.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:04.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:04.366 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:11:04.379 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 47468/284658 objects degraded (16.675%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:04.380 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:04 smithi066 ceph-mon[29666]: osdmap e262: 8 total, 7 up, 8 in 2024-03-26T03:11:04.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 47468/284658 objects degraded (16.675%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:04.380 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:04 smithi066 ceph-mon[37920]: osdmap e262: 8 total, 7 up, 8 in 2024-03-26T03:11:04.381 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:11:04.403 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 47468/284658 objects degraded (16.675%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:04.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:04 smithi192 ceph-mon[33110]: osdmap e262: 8 total, 7 up, 8 in 2024-03-26T03:11:04.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:04.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:04.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:04.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:04.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:04.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:04.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:04.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:04.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:05.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:05.114 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:05.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:05.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:05.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[29666]: pgmap v528: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 951 op/s; 46987/283215 objects degraded (16.591%); 1/283215 objects misplaced (0.000%) 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:11:05.305 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:11:05.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:11:05.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:11:05.306 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:05 smithi066 ceph-mon[37920]: pgmap v528: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 951 op/s; 46987/283215 objects degraded (16.591%); 1/283215 objects misplaced (0.000%) 2024-03-26T03:11:05.357 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:05.394 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:05 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:11:05.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:05 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:11:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:05 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:11:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:05 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:11:05.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:05 smithi192 ceph-mon[33110]: pgmap v528: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 951 op/s; 46987/283215 objects degraded (16.591%); 1/283215 objects misplaced (0.000%) 2024-03-26T03:11:05.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:05.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:05.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:05.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:05.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:05.700 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:11:05.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:05.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:05.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:05.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:05.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:06.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:06.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:06.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:06.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:06.321 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:06.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:06 smithi066 ceph-mon[29666]: 9.9 scrub starts 2024-03-26T03:11:06.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:06 smithi066 ceph-mon[37920]: 9.9 scrub starts 2024-03-26T03:11:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.370 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:06.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:06.471 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:06.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:06 smithi192 ceph-mon[33110]: 9.9 scrub starts 2024-03-26T03:11:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:06.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:06.550 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:06.551 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:06.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:06.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:06.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:06.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:06.879 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:06.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:06.946 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:06.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:07.024 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:11:07.024 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:11:07.024 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 263, 2024-03-26T03:11:07.024 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:11:06.121653+0000", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T03:06:19.254688+0000", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5799936, 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:11:07.025 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 9, 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:11:07.026 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 21, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 21, 2024-03-26T03:11:07.027 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 21, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 21, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 21, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "254", 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "252", 2024-03-26T03:11:07.028 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.029 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.030 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.031 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2699999809265137, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2699999809265137, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:07.032 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:07.033 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:07.034 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.035 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.036 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:11:07.037 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.038 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.039 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.040 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.041 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.042 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:11:07.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:11:07.044 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:07.045 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.046 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.047 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.048 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2400000095367432, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2400000095367432, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-26T03:11:07.049 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.050 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:07.051 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:07.052 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.053 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.054 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.055 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.056 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.057 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.058 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.059 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.060 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.061 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 9, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_2", 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:08:41.897072+0000", 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:07.062 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 16, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 16, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 16, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 16, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 16, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:07.063 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "228", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:07.064 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:07.065 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:07.066 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.067 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:11:07.069 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.070 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:11:07.071 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:11:07.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.072 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.074 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:11:07.075 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:11:07.076 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:11:07.077 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.078 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 0, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.080 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 198, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.081 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:07.082 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:11:07.083 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "exists" 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.084 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.085 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.086 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.087 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.088 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.089 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.090 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:07.091 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.092 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.093 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.094 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.095 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:11:07.096 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.097 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.098 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:07.099 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:11:07.100 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:10:01.092783+0000", 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.101 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:10:39.727465+0000", 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:10:46.033387+0000", 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.102 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:05:36.091229+0000", 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 197 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.103 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:10:30.710211+0000", 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:05.177898+0000", 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:09:41.545305+0000", 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:07.105 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:09:48.417782+0000", 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [ 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.f", 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.106 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.f", 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 4, 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 7 2024-03-26T03:11:07.107 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 0, 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 1 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.108 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:11:07.109 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [ 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.c", 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: 6, 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: 5 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.110 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.e", 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: 7, 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: 0 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.11", 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: 5, 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: 2 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.13", 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: 6 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:07.112 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:11:07.113 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:11:07.114 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:11:07.115 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:11:07.115 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:11:07.115 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:07.115 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:11:07.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:07.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:07.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:07.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:11:07.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:11:07.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: osdmap e263: 8 total, 7 up, 8 in 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: pgmap v530: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.56k op/s; 43657/261912 objects degraded (16.669%); 1/261912 objects misplaced (0.000%) 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2261752028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: osdmap e263: 8 total, 7 up, 8 in 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: pgmap v530: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.56k op/s; 43657/261912 objects degraded (16.669%); 1/261912 objects misplaced (0.000%) 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:11:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2261752028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:11:07.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:07.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:11:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:11:07.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: osdmap e263: 8 total, 7 up, 8 in 2024-03-26T03:11:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: pgmap v530: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.56k op/s; 43657/261912 objects degraded (16.669%); 1/261912 objects misplaced (0.000%) 2024-03-26T03:11:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:11:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:11:07.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2261752028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:11:07.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:07.545 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:07.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:07.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:07.788 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:07.827 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:07.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:07.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:07.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:08.067 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:08.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:08.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:08.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:08.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:08.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:08.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:11:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:11:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: 9.9 scrub ok 2024-03-26T03:11:08.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:11:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:11:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:11:08.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:08 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:11:08.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:08.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: 9.9 scrub ok 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:11:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: 9.9 scrub ok 2024-03-26T03:11:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:11:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:11:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:11:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:08 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:11:08.612 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:08.647 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:08.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:08.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:08.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:08.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:08.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:09.023 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:09.055 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:09.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:09.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:09.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:09.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:09.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[29666]: osdmap e264: 8 total, 7 up, 8 in 2024-03-26T03:11:09.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[29666]: pgmap v532: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 43657/261912 objects degraded (16.669%); 1/261912 objects misplaced (0.000%) 2024-03-26T03:11:09.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[29666]: 9.5 deep-scrub starts 2024-03-26T03:11:09.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 43657/261912 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[37920]: osdmap e264: 8 total, 7 up, 8 in 2024-03-26T03:11:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[37920]: pgmap v532: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 43657/261912 objects degraded (16.669%); 1/261912 objects misplaced (0.000%) 2024-03-26T03:11:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[37920]: 9.5 deep-scrub starts 2024-03-26T03:11:09.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 43657/261912 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:09.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:09.469 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:09 smithi192 ceph-mon[33110]: osdmap e264: 8 total, 7 up, 8 in 2024-03-26T03:11:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:09 smithi192 ceph-mon[33110]: pgmap v532: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 43657/261912 objects degraded (16.669%); 1/261912 objects misplaced (0.000%) 2024-03-26T03:11:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:09 smithi192 ceph-mon[33110]: 9.5 deep-scrub starts 2024-03-26T03:11:09.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 43657/261912 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:09.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:09.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:09.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:09.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:09.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:09.866 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:09.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:09.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:11:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:11:09.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:09.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:10.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:10.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:10.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:10.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:10.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:10.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:10.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:10.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:10.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:11:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:11:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:11:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:10 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:11:10.634 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:10.672 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:10 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:11:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:10 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:11:10.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:10.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:10.900 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:10.936 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:10.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:11.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:11.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:11.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:11.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:11.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:11.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:11.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:11.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:11.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:11.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:11:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:11:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[29666]: pgmap v533: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2.07k op/s; 40639/246387 objects degraded (16.494%); 1/246387 objects misplaced (0.000%) 2024-03-26T03:11:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[29666]: osdmap e265: 8 total, 7 up, 8 in 2024-03-26T03:11:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:11:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:11:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[37920]: pgmap v533: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2.07k op/s; 40639/246387 objects degraded (16.494%); 1/246387 objects misplaced (0.000%) 2024-03-26T03:11:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:11 smithi066 ceph-mon[37920]: osdmap e265: 8 total, 7 up, 8 in 2024-03-26T03:11:11.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:11.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:11 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:11:11.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:11 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:11:11.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:11 smithi192 ceph-mon[33110]: pgmap v533: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2.07k op/s; 40639/246387 objects degraded (16.494%); 1/246387 objects misplaced (0.000%) 2024-03-26T03:11:11.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:11 smithi192 ceph-mon[33110]: osdmap e265: 8 total, 7 up, 8 in 2024-03-26T03:11:11.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:11.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:11.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:11.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:11.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:11.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:12.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:12.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:12.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:12.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:12.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:12.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:12.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:11:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:11:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[29666]: 4.b deep-scrub starts 2024-03-26T03:11:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[29666]: 4.b deep-scrub ok 2024-03-26T03:11:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:11:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:11:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[37920]: 4.b deep-scrub starts 2024-03-26T03:11:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:12 smithi066 ceph-mon[37920]: 4.b deep-scrub ok 2024-03-26T03:11:12.671 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:12.704 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:12 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:11:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:12 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:11:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:12 smithi192 ceph-mon[33110]: 4.b deep-scrub starts 2024-03-26T03:11:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:12 smithi192 ceph-mon[33110]: 4.b deep-scrub ok 2024-03-26T03:11:12.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:12.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:12.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:12.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:12.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:13.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:13.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:13.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:13.244 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:13.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:13.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:13.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:13.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-mon[29666]: pgmap v535: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 839 B/s rd, 0 B/s wr, 1.13k op/s; 40224/241302 objects degraded (16.670%); 1/241302 objects misplaced (0.000%) 2024-03-26T03:11:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:11:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:11:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-mon[37920]: pgmap v535: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 839 B/s rd, 0 B/s wr, 1.13k op/s; 40224/241302 objects degraded (16.670%); 1/241302 objects misplaced (0.000%) 2024-03-26T03:11:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:11:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:11:13.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:13.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:13 smithi192 ceph-mon[33110]: pgmap v535: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 839 B/s rd, 0 B/s wr, 1.13k op/s; 40224/241302 objects degraded (16.670%); 1/241302 objects misplaced (0.000%) 2024-03-26T03:11:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:13 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:11:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:13 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:11:13.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:13.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:13.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:13.934 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:13.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:14.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:14.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:11:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:11:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:11:14.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:14.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:14.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:14.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:14.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:14.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:14.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:11:14.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:11:14.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:11:14.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 40224/241302 objects degraded (16.670%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[29666]: osdmap e266: 8 total, 7 up, 8 in 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:11:14.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:11:14.458 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:11:14.458 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:11:14.458 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:11:14.458 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:11:14.458 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 40224/241302 objects degraded (16.670%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:14.458 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:14 smithi066 ceph-mon[37920]: osdmap e266: 8 total, 7 up, 8 in 2024-03-26T03:11:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:14.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:14.499 INFO:tasks.thrashosds.thrasher:j is {'epoch': 263, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:11:06.121653+0000', 'last_up_change': '2024-03-26T03:06:19.254688+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '252', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-26T03:08:41.897072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 172, 'down_at': 198, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:10:01.092783+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:10:39.727465+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:10:46.033387+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:05:36.091229+0000', 'dead_epoch': 197}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:10:30.710211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:05.177898+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:09:41.545305+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:09:48.417782+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [4, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.f', 'mappings': [{'from': 4, 'to': 7}, {'from': 0, 'to': 1}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [0, 6, 5]}, {'pgid': '1.e', 'osds': [4, 7, 0]}, {'pgid': '1.11', 'osds': [0, 5, 2]}, {'pgid': '1.13', 'osds': [1, 4, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:11:14.499 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:11:14.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:14.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:14.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:11:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:11:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 40224/241302 objects degraded (16.670%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:14 smithi192 ceph-mon[33110]: osdmap e266: 8 total, 7 up, 8 in 2024-03-26T03:11:14.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:14.848 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:14.884 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:14.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:14.986 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:15.029 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:15.131 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:15.173 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:15.275 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:15.309 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:15.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:15.445 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: pgmap v537: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 1.19k op/s; 39868/240234 objects degraded (16.595%); 1/240234 objects misplaced (0.000%) 2024-03-26T03:11:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:11:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:11:15.542 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: pgmap v537: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 1.19k op/s; 39868/240234 objects degraded (16.595%); 1/240234 objects misplaced (0.000%) 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:11:15.543 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:15 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:11:15.548 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:15.606 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:15.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: pgmap v537: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 1.19k op/s; 39868/240234 objects degraded (16.595%); 1/240234 objects misplaced (0.000%) 2024-03-26T03:11:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:11:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:11:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:11:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:11:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:11:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:15 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:11:15.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:15.843 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:15.881 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:15.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:15.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:16.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:16.119 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:16.159 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:16.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:16.309 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:16.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:16.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:16 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:11:16.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:16 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:11:16.421 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:16 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:11:16.421 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:16 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:11:16.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:16.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.598 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:16.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:16.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:16 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:11:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:16 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:11:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:16.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:16.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:16.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:16.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:16.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:17.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:17.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:17.137 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:11:17.137 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:11:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:17.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:17.247 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.285 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:17.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:17.381 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:11:17.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 6.1 deep-scrub starts 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 6.1 deep-scrub ok 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: pgmap v538: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 37499/224958 objects degraded (16.669%); 1/224958 objects misplaced (0.000%) 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:11:17.398 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2836224358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 6.1 deep-scrub starts 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 6.1 deep-scrub ok 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: pgmap v538: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 37499/224958 objects degraded (16.669%); 1/224958 objects misplaced (0.000%) 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:11:17.399 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2836224358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:11:17.400 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:11:17.400 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:17 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:11:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:17.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:17.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:17.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:17.656 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:17.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 6.1 deep-scrub starts 2024-03-26T03:11:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 6.1 deep-scrub ok 2024-03-26T03:11:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: pgmap v538: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 37499/224958 objects degraded (16.669%); 1/224958 objects misplaced (0.000%) 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2836224358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:11:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:17 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:11:17.792 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:17.827 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:17.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:17.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:17.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:18.065 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:18.102 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:18.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:18.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:18.329 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:18.363 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:18.431 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:18.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:18.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:18.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:18.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:18.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:18 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:18.709 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:18 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2836224358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:11:18.709 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:18 smithi066 ceph-mon[29666]: osdmap e267: 8 total, 7 up, 8 in 2024-03-26T03:11:18.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:18 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:18.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:18 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2836224358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:11:18.709 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:18 smithi066 ceph-mon[37920]: osdmap e267: 8 total, 7 up, 8 in 2024-03-26T03:11:18.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:18 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:18 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2836224358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:11:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:18 smithi192 ceph-mon[33110]: osdmap e267: 8 total, 7 up, 8 in 2024-03-26T03:11:18.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:18.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:18.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:18.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:19.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:19.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:19.142 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:19.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:19.281 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:19.317 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:19.419 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:19.458 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:19.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:19.594 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:19 smithi066 ceph-mon[29666]: pgmap v540: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 905 op/s; 37499/224958 objects degraded (16.669%); 1/224958 objects misplaced (0.000%) 2024-03-26T03:11:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 37499/224958 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:19 smithi066 ceph-mon[29666]: osdmap e268: 8 total, 7 up, 8 in 2024-03-26T03:11:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:19 smithi066 ceph-mon[37920]: pgmap v540: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 905 op/s; 37499/224958 objects degraded (16.669%); 1/224958 objects misplaced (0.000%) 2024-03-26T03:11:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 37499/224958 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:19 smithi066 ceph-mon[37920]: osdmap e268: 8 total, 7 up, 8 in 2024-03-26T03:11:19.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:19 smithi192 ceph-mon[33110]: pgmap v540: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 905 op/s; 37499/224958 objects degraded (16.669%); 1/224958 objects misplaced (0.000%) 2024-03-26T03:11:19.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 37499/224958 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:19.644 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:19 smithi192 ceph-mon[33110]: osdmap e268: 8 total, 7 up, 8 in 2024-03-26T03:11:19.697 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.735 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:19.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:19.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:19.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:19.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:19.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:19.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:11:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:11:20.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:20.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:20.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:20.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:20.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:20.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:20.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:20.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:20.557 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:20.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:20.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:20.797 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:20.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:20.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:20.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:20.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:21.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:21.103 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:21.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:21.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:21.342 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:21.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:11:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:21.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:21.463 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:21.463 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[29666]: pgmap v542: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.62k op/s; 34878/211443 objects degraded (16.495%); 1/211443 objects misplaced (0.000%) 2024-03-26T03:11:21.463 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:11:21.463 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/737299914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:21.464 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:21.464 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[37920]: pgmap v542: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.62k op/s; 34878/211443 objects degraded (16.495%); 1/211443 objects misplaced (0.000%) 2024-03-26T03:11:21.464 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:11:21.464 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:21 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/737299914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:21.515 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.550 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:21.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:21.651 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:21.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:21.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:21.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:21 smithi192 ceph-mon[33110]: pgmap v542: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.62k op/s; 34878/211443 objects degraded (16.495%); 1/211443 objects misplaced (0.000%) 2024-03-26T03:11:21.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:11:21.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:21 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/737299914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:21.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:21.794 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.837 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:21.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:21.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:21.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:21.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:22.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:22.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:22.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:22.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:22.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:11:22.398 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:11:22.399 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:11:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.401 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:22.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:22.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:22.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:22.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:22.673 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:22.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:22.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:22.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:22.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:22.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:22.962 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:11:23.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:23.088 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:23.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:23.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:11:23.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:11:23.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:23.365 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:23.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:23.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:23.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[29666]: pgmap v543: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 34481/206757 objects degraded (16.677%); 1/206757 objects misplaced (0.000%) 2024-03-26T03:11:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:11:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:11:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[37920]: pgmap v543: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 34481/206757 objects degraded (16.677%); 1/206757 objects misplaced (0.000%) 2024-03-26T03:11:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:11:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:11:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:23.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:23.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:23 smithi192 ceph-mon[33110]: pgmap v543: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 34481/206757 objects degraded (16.677%); 1/206757 objects misplaced (0.000%) 2024-03-26T03:11:23.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:11:23.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:11:23.712 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:23.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:23.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:23.792 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:23.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:23.914 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:11:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:11:23.915 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:23.926 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:11:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:11:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:11:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:23.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:23.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:24.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:24.083 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:24.163 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:11:24.167 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:11:24.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:24.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:24.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:24.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:24.462 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:24.501 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:24.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:24.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 34481/206757 objects degraded (16.677%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:24.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:24.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:24.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 34481/206757 objects degraded (16.677%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:24.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:24.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:24.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 34481/206757 objects degraded (16.677%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:24.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:11:24.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:24.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:24.741 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:24.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:11:24.777 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:24.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:24.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:24.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:25.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:25.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:25.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:25.169 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:25.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:25.239 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:11:25.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:25.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:25.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:25.484 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:25.522 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:25 smithi066 ceph-mon[29666]: from='client.35381 v1:172.21.15.66:0/233103918' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:11:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:25 smithi066 ceph-mon[29666]: pgmap v544: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 801 op/s; 34134/205716 objects degraded (16.593%); 1/205716 objects misplaced (0.000%) 2024-03-26T03:11:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:25 smithi066 ceph-mon[37920]: from='client.35381 v1:172.21.15.66:0/233103918' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:11:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:25 smithi066 ceph-mon[37920]: pgmap v544: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 801 op/s; 34134/205716 objects degraded (16.593%); 1/205716 objects misplaced (0.000%) 2024-03-26T03:11:25.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:25.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:25 smithi192 ceph-mon[33110]: from='client.35381 v1:172.21.15.66:0/233103918' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:11:25.640 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:25 smithi192 ceph-mon[33110]: pgmap v544: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 801 op/s; 34134/205716 objects degraded (16.593%); 1/205716 objects misplaced (0.000%) 2024-03-26T03:11:25.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:25.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:25.806 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:25.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:25.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:25.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:26.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:26.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:26.174 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:26.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:26.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:26.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:26.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:26.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:26.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:26.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:26.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:26.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:26.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:26.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:26.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:27.015 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.056 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:27.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:27.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:27.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:27.290 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:27.330 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:27.421 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2024-03-26T03:11:22.230940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4502151074,"num_objects":68919,"num_object_clones":0,"num_object_copies":206757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34481,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":68919,"num_whiteouts":0,"num_read":2173,"num_read_kb":1896,"num_write":224828,"num_write_kb":9363191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":54528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79983,"ondisk_log_size":79983,"up":432,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":502,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17233424,"kb_used_data":13606204,"kb_used_omap":12,"kb_used_meta":3627059,"kb_avail":732727792,"statfs":{"total":767960285184,"available":750313259008,"internally_reserved":0,"allocated":13932752896,"data_stored":13928284977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3714108976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-731316224,"num_objects":-11159,"num_object_clones":0,"num_object_copies":-33477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11159,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11159,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004329"},"pg_stats":[{"pgid":"1.d","version":"82'128","reported_seq":337,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081948+0000","last_change":"2024-03-26T03:10:50.602175+0000","last_active":"2024-03-26T03:11:19.081948+0000","last_peered":"2024-03-26T03:11:19.081948+0000","last_clean":"2024-03-26T03:11:19.081948+0000","last_became_active":"2024-03-26T03:10:44.090472+0000","last_became_peered":"2024-03-26T03:10:44.090472+0000","last_unstale":"2024-03-26T03:11:19.081948+0000","last_undegraded":"2024-03-26T03:11:19.081948+0000","last_fullsized":"2024-03-26T03:11:19.081948+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.602062+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.602062+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:16.544511+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"268'16300","reported_seq":16462,"reported_epoch":268,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-26T03:11:20.583451+0000","last_change":"2024-03-26T03:11:08.552854+0000","last_active":"2024-03-26T03:11:20.583451+0000","last_peered":"2024-03-26T03:11:20.583451+0000","last_clean":"2024-03-26T03:11:20.583451+0000","last_became_active":"2024-03-26T03:08:43.065248+0000","last_became_peered":"2024-03-26T03:08:43.065248+0000","last_unstale":"2024-03-26T03:11:20.583451+0000","last_undegraded":"2024-03-26T03:11:20.583451+0000","last_fullsized":"2024-03-26T03:11:20.583451+0000","mapping_epoch":225,"log_start":"257'11002","ondisk_log_start":"257'11002","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":2129,"log_size":5298,"log_dups_size":0,"ondisk_log_size":5298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279642112,"num_objects":4267,"num_object_clones":0,"num_object_copies":12801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14171,"num_write_kb":590016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":622,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078791+0000","last_change":"2024-03-26T03:10:55.601026+0000","last_active":"2024-03-26T03:11:19.078791+0000","last_peered":"2024-03-26T03:11:19.078791+0000","last_clean":"2024-03-26T03:11:19.078791+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:11:19.078791+0000","last_undegraded":"2024-03-26T03:11:19.078791+0000","last_fullsized":"2024-03-26T03:11:19.078791+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:10:55.600815+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.600815+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.420458+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":542,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.080525+0000","last_change":"2024-03-26T03:10:14.796412+0000","last_active":"2024-03-26T03:11:19.080525+0000","last_peered":"2024-03-26T03:11:19.080525+0000","last_clean":"2024-03-26T03:11:19.080525+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:11:19.080525+0000","last_undegraded":"2024-03-26T03:11:19.080525+0000","last_fullsized":"2024-03-26T03:11:19.080525+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:10:14.796187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.796187+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:32.999963+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.101933+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:11:19.101933+0000","last_peered":"2024-03-26T03:11:19.101933+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:11:19.101933+0000","last_undegraded":"2024-03-26T03:11:19.101933+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":273,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.075915+0000","last_change":"2024-03-26T03:10:59.286571+0000","last_active":"2024-03-26T03:11:19.075915+0000","last_peered":"2024-03-26T03:11:19.075915+0000","last_clean":"2024-03-26T03:11:19.075915+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:11:19.075915+0000","last_undegraded":"2024-03-26T03:11:19.075915+0000","last_fullsized":"2024-03-26T03:11:19.075915+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:59.286437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:59.286437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.053590+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":334,"reported_epoch":268,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:11:19.083773+0000","last_change":"2024-03-26T03:10:50.577501+0000","last_active":"2024-03-26T03:11:19.083773+0000","last_peered":"2024-03-26T03:11:19.083773+0000","last_clean":"2024-03-26T03:11:19.083773+0000","last_became_active":"2024-03-26T03:10:44.082186+0000","last_became_peered":"2024-03-26T03:10:44.082186+0000","last_unstale":"2024-03-26T03:11:19.083773+0000","last_undegraded":"2024-03-26T03:11:19.083773+0000","last_fullsized":"2024-03-26T03:11:19.083773+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.577205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.577205+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.267018+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"268'13955","reported_seq":14141,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.515109+0000","last_change":"2024-03-26T03:10:12.982393+0000","last_active":"2024-03-26T03:11:19.515109+0000","last_peered":"2024-03-26T03:11:19.515109+0000","last_clean":"2024-03-26T03:11:19.515109+0000","last_became_active":"2024-03-26T03:08:43.063988+0000","last_became_peered":"2024-03-26T03:08:43.063988+0000","last_unstale":"2024-03-26T03:11:19.515109+0000","last_undegraded":"2024-03-26T03:11:19.515109+0000","last_fullsized":"2024-03-26T03:11:19.515109+0000","mapping_epoch":225,"log_start":"245'7753","ondisk_log_start":"245'7753","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'8126","last_scrub_stamp":"2024-03-26T03:10:12.982181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:12.982181+0000","objects_scrubbed":8027,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.033271+0000","scrub_duration":2644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276889600,"num_objects":4225,"num_object_clones":0,"num_object_copies":12675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13955,"num_write_kb":581760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":510,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.103075+0000","last_change":"2024-03-26T03:11:11.587482+0000","last_active":"2024-03-26T03:11:19.103075+0000","last_peered":"2024-03-26T03:11:19.103075+0000","last_clean":"2024-03-26T03:11:19.103075+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:11:19.103075+0000","last_undegraded":"2024-03-26T03:11:19.103075+0000","last_fullsized":"2024-03-26T03:11:19.103075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:11:11.587327+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:27.886569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.073733+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:11:19.073733+0000","last_peered":"2024-03-26T03:11:19.073733+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:11:19.073733+0000","last_undegraded":"2024-03-26T03:11:19.073733+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176160+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":435,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.071822+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:11:19.071822+0000","last_peered":"2024-03-26T03:11:19.071822+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:11:19.071822+0000","last_undegraded":"2024-03-26T03:11:19.071822+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":452,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.101005+0000","last_change":"2024-03-26T03:09:04.206625+0000","last_active":"2024-03-26T03:11:19.101005+0000","last_peered":"2024-03-26T03:11:19.101005+0000","last_clean":"2024-03-26T03:11:19.101005+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:11:19.101005+0000","last_undegraded":"2024-03-26T03:11:19.101005+0000","last_fullsized":"2024-03-26T03:11:19.101005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:04.203542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:09:04.203542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":545,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077757+0000","last_change":"2024-03-26T03:10:49.362862+0000","last_active":"2024-03-26T03:11:19.077757+0000","last_peered":"2024-03-26T03:11:19.077757+0000","last_clean":"2024-03-26T03:11:19.077757+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:11:19.077757+0000","last_undegraded":"2024-03-26T03:11:19.077757+0000","last_fullsized":"2024-03-26T03:11:19.077757+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.362737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.362737+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.713660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"268'13986","reported_seq":14080,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.417154+0000","last_change":"2024-03-26T03:08:54.927812+0000","last_active":"2024-03-26T03:11:19.417154+0000","last_peered":"2024-03-26T03:11:19.417154+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.069288+0000","last_became_peered":"2024-03-26T03:08:43.069288+0000","last_unstale":"2024-03-26T03:11:19.417154+0000","last_undegraded":"2024-03-26T03:08:54.925664+0000","last_fullsized":"2024-03-26T03:08:43.062900+0000","mapping_epoch":225,"log_start":"249'9902","ondisk_log_start":"249'9902","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4084,"log_dups_size":0,"ondisk_log_size":4084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288882688,"num_objects":4408,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13986,"num_write_kb":588608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":4408}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":563,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.076212+0000","last_change":"2024-03-26T03:11:12.308404+0000","last_active":"2024-03-26T03:11:19.076212+0000","last_peered":"2024-03-26T03:11:19.076212+0000","last_clean":"2024-03-26T03:11:19.076212+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:11:19.076212+0000","last_undegraded":"2024-03-26T03:11:19.076212+0000","last_fullsized":"2024-03-26T03:11:19.076212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:11:12.308255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:12.308255+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:29.673061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":480,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.100426+0000","last_change":"2024-03-26T03:10:17.265447+0000","last_active":"2024-03-26T03:11:19.100426+0000","last_peered":"2024-03-26T03:11:19.100426+0000","last_clean":"2024-03-26T03:11:19.100426+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:11:19.100426+0000","last_undegraded":"2024-03-26T03:11:19.100426+0000","last_fullsized":"2024-03-26T03:11:19.100426+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:17.265332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:10:17.265332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:34.761705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.072093+0000","last_change":"2024-03-26T03:09:59.321675+0000","last_active":"2024-03-26T03:11:19.072093+0000","last_peered":"2024-03-26T03:11:19.072093+0000","last_clean":"2024-03-26T03:11:19.072093+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:11:19.072093+0000","last_undegraded":"2024-03-26T03:11:19.072093+0000","last_fullsized":"2024-03-26T03:11:19.072093+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:59.321541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:09:59.321541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:25.067216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":431,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.077318+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:11:19.077318+0000","last_peered":"2024-03-26T03:11:19.077318+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:11:19.077318+0000","last_undegraded":"2024-03-26T03:11:19.077318+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"268'14076","reported_seq":14257,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.873524+0000","last_change":"2024-03-26T03:10:05.447645+0000","last_active":"2024-03-26T03:11:19.873524+0000","last_peered":"2024-03-26T03:11:19.873524+0000","last_clean":"2024-03-26T03:11:19.873524+0000","last_became_active":"2024-03-26T03:08:43.063932+0000","last_became_peered":"2024-03-26T03:08:43.063932+0000","last_unstale":"2024-03-26T03:11:19.873524+0000","last_undegraded":"2024-03-26T03:11:19.873524+0000","last_fullsized":"2024-03-26T03:11:19.873524+0000","mapping_epoch":225,"log_start":"248'9670","ondisk_log_start":"248'9670","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'7536","last_scrub_stamp":"2024-03-26T03:10:05.447508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:05.447508+0000","objects_scrubbed":7456,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:29.597013+0000","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283246592,"num_objects":4322,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14076,"num_write_kb":588736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":524,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084248+0000","last_change":"2024-03-26T03:10:56.581354+0000","last_active":"2024-03-26T03:11:19.084248+0000","last_peered":"2024-03-26T03:11:19.084248+0000","last_clean":"2024-03-26T03:11:19.084248+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:11:19.084248+0000","last_undegraded":"2024-03-26T03:11:19.084248+0000","last_fullsized":"2024-03-26T03:11:19.084248+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:10:56.581216+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:10:56.581216+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:58.311148+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.079277+0000","last_change":"2024-03-26T03:09:47.553143+0000","last_active":"2024-03-26T03:11:19.079277+0000","last_peered":"2024-03-26T03:11:19.079277+0000","last_clean":"2024-03-26T03:11:19.079277+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:11:19.079277+0000","last_undegraded":"2024-03-26T03:11:19.079277+0000","last_fullsized":"2024-03-26T03:11:19.079277+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:47.553022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.553022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.671275+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.074682+0000","last_change":"2024-03-26T03:10:47.065658+0000","last_active":"2024-03-26T03:11:19.074682+0000","last_peered":"2024-03-26T03:11:19.074682+0000","last_clean":"2024-03-26T03:11:19.074682+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:11:19.074682+0000","last_undegraded":"2024-03-26T03:11:19.074682+0000","last_fullsized":"2024-03-26T03:11:19.074682+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:47.065556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.065556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.028651+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.101870+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:11:19.101870+0000","last_peered":"2024-03-26T03:11:19.101870+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:11:19.101870+0000","last_undegraded":"2024-03-26T03:11:19.101870+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":521,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.071439+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:11:19.071439+0000","last_peered":"2024-03-26T03:11:19.071439+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:11:19.071439+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"268'14053","reported_seq":14147,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.281288+0000","last_change":"2024-03-26T03:08:54.926696+0000","last_active":"2024-03-26T03:11:19.281288+0000","last_peered":"2024-03-26T03:11:19.281288+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.066721+0000","last_became_peered":"2024-03-26T03:08:43.066721+0000","last_unstale":"2024-03-26T03:11:19.281288+0000","last_undegraded":"2024-03-26T03:08:54.926086+0000","last_fullsized":"2024-03-26T03:08:43.059549+0000","mapping_epoch":225,"log_start":"246'8694","ondisk_log_start":"246'8694","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14053,"num_write_kb":591936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":4445}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":712,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.078061+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:11:19.078061+0000","last_peered":"2024-03-26T03:11:19.078061+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:11:19.078061+0000","last_undegraded":"2024-03-26T03:11:19.078061+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":476,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.083662+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:11:19.083662+0000","last_peered":"2024-03-26T03:11:19.083662+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:11:19.083662+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792887+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":443,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.073414+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:11:19.073414+0000","last_peered":"2024-03-26T03:11:19.073414+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:11:19.073414+0000","last_undegraded":"2024-03-26T03:11:19.073414+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176119+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":630,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.100227+0000","last_change":"2024-03-26T03:10:55.115703+0000","last_active":"2024-03-26T03:11:19.100227+0000","last_peered":"2024-03-26T03:11:19.100227+0000","last_clean":"2024-03-26T03:11:19.100227+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:11:19.100227+0000","last_undegraded":"2024-03-26T03:11:19.100227+0000","last_fullsized":"2024-03-26T03:11:19.100227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:10:55.115576+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.115576+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:19.781331+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084802+0000","last_change":"2024-03-26T03:10:47.638717+0000","last_active":"2024-03-26T03:11:19.084802+0000","last_peered":"2024-03-26T03:11:19.084802+0000","last_clean":"2024-03-26T03:11:19.084802+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:11:19.084802+0000","last_undegraded":"2024-03-26T03:11:19.084802+0000","last_fullsized":"2024-03-26T03:11:19.084802+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:47.638593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.638593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:52.386653+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"268'13983","reported_seq":14077,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.868492+0000","last_change":"2024-03-26T03:08:54.925459+0000","last_active":"2024-03-26T03:11:19.868492+0000","last_peered":"2024-03-26T03:11:19.868492+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.057035+0000","last_became_peered":"2024-03-26T03:08:43.057035+0000","last_unstale":"2024-03-26T03:11:19.868492+0000","last_undegraded":"2024-03-26T03:08:54.924597+0000","last_fullsized":"2024-03-26T03:08:43.055113+0000","mapping_epoch":225,"log_start":"248'9572","ondisk_log_start":"248'9572","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4411,"log_dups_size":0,"ondisk_log_size":4411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":12819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13983,"num_write_kb":584192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":4273}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":551,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.071776+0000","last_change":"2024-03-26T03:10:21.241202+0000","last_active":"2024-03-26T03:11:19.071776+0000","last_peered":"2024-03-26T03:11:19.071776+0000","last_clean":"2024-03-26T03:11:19.071776+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:11:19.071776+0000","last_undegraded":"2024-03-26T03:11:19.071776+0000","last_fullsized":"2024-03-26T03:11:19.071776+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:21.241052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:21.241052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:49.792807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":490,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.073400+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:11:19.073400+0000","last_peered":"2024-03-26T03:11:19.073400+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:11:19.073400+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176147+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":443,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.072656+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:11:19.072656+0000","last_peered":"2024-03-26T03:11:19.072656+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:11:19.072656+0000","last_undegraded":"2024-03-26T03:11:19.072656+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":612,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084645+0000","last_change":"2024-03-26T03:09:51.797832+0000","last_active":"2024-03-26T03:11:19.084645+0000","last_peered":"2024-03-26T03:11:19.084645+0000","last_clean":"2024-03-26T03:11:19.084645+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:11:19.084645+0000","last_undegraded":"2024-03-26T03:11:19.084645+0000","last_fullsized":"2024-03-26T03:11:19.084645+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:09:51.797681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:51.797681+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":265,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073580+0000","last_change":"2024-03-26T03:11:01.239298+0000","last_active":"2024-03-26T03:11:19.073580+0000","last_peered":"2024-03-26T03:11:19.073580+0000","last_clean":"2024-03-26T03:11:19.073580+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:11:19.073580+0000","last_undegraded":"2024-03-26T03:11:19.073580+0000","last_fullsized":"2024-03-26T03:11:19.073580+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.239012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.239012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:25.557947+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":563,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073657+0000","last_change":"2024-03-26T03:10:51.016455+0000","last_active":"2024-03-26T03:11:19.073657+0000","last_peered":"2024-03-26T03:11:19.073657+0000","last_clean":"2024-03-26T03:11:19.073657+0000","last_became_active":"2024-03-26T03:10:44.079636+0000","last_became_peered":"2024-03-26T03:10:44.079636+0000","last_unstale":"2024-03-26T03:11:19.073657+0000","last_undegraded":"2024-03-26T03:11:19.073657+0000","last_fullsized":"2024-03-26T03:11:19.073657+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.016116+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.016116+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:11.231426+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"268'13881","reported_seq":13975,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.413774+0000","last_change":"2024-03-26T03:08:54.940154+0000","last_active":"2024-03-26T03:11:19.413774+0000","last_peered":"2024-03-26T03:11:19.413774+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.064645+0000","last_became_peered":"2024-03-26T03:08:43.064645+0000","last_unstale":"2024-03-26T03:11:19.413774+0000","last_undegraded":"2024-03-26T03:08:54.937570+0000","last_fullsized":"2024-03-26T03:08:43.060747+0000","mapping_epoch":225,"log_start":"249'9780","ondisk_log_start":"249'9780","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278855680,"num_objects":4255,"num_object_clones":0,"num_object_copies":12765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13881,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":4255}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"231'25","reported_seq":510,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084014+0000","last_change":"2024-03-26T03:11:06.580747+0000","last_active":"2024-03-26T03:11:19.084014+0000","last_peered":"2024-03-26T03:11:19.084014+0000","last_clean":"2024-03-26T03:11:19.084014+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:11:19.084014+0000","last_undegraded":"2024-03-26T03:11:19.084014+0000","last_fullsized":"2024-03-26T03:11:19.084014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:11:06.580543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:06.580543+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.318740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077559+0000","last_change":"2024-03-26T03:11:00.578809+0000","last_active":"2024-03-26T03:11:19.077559+0000","last_peered":"2024-03-26T03:11:19.077559+0000","last_clean":"2024-03-26T03:11:19.077559+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:11:19.077559+0000","last_undegraded":"2024-03-26T03:11:19.077559+0000","last_fullsized":"2024-03-26T03:11:19.077559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:00.578704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:00.578704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.102616+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":263,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084921+0000","last_change":"2024-03-26T03:09:56.868132+0000","last_active":"2024-03-26T03:11:19.084921+0000","last_peered":"2024-03-26T03:11:19.084921+0000","last_clean":"2024-03-26T03:11:19.084921+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:11:19.084921+0000","last_undegraded":"2024-03-26T03:11:19.084921+0000","last_fullsized":"2024-03-26T03:11:19.084921+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:56.867463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:56.867463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:19.527285+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.076553+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:11:19.076553+0000","last_peered":"2024-03-26T03:11:19.076553+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:11:19.076553+0000","last_undegraded":"2024-03-26T03:11:19.076553+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"268'13976","reported_seq":14070,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.198077+0000","last_change":"2024-03-26T03:08:54.944412+0000","last_active":"2024-03-26T03:11:19.198077+0000","last_peered":"2024-03-26T03:11:19.198077+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.063578+0000","last_became_peered":"2024-03-26T03:08:43.063578+0000","last_unstale":"2024-03-26T03:11:19.198077+0000","last_undegraded":"2024-03-26T03:08:54.938583+0000","last_fullsized":"2024-03-26T03:08:43.056078+0000","mapping_epoch":225,"log_start":"248'9239","ondisk_log_start":"248'9239","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283901952,"num_objects":4332,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13976,"num_write_kb":585856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":4332}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":299,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.072486+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:11:19.072486+0000","last_peered":"2024-03-26T03:11:19.072486+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:11:19.072486+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":521,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081780+0000","last_change":"2024-03-26T03:10:15.759744+0000","last_active":"2024-03-26T03:11:19.081780+0000","last_peered":"2024-03-26T03:11:19.081780+0000","last_clean":"2024-03-26T03:11:19.081780+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:11:19.081780+0000","last_undegraded":"2024-03-26T03:11:19.081780+0000","last_fullsized":"2024-03-26T03:11:19.081780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:10:15.759584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:15.759584+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:41.731999+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078147+0000","last_change":"2024-03-26T03:11:07.220709+0000","last_active":"2024-03-26T03:11:19.078147+0000","last_peered":"2024-03-26T03:11:19.078147+0000","last_clean":"2024-03-26T03:11:19.078147+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:11:19.078147+0000","last_undegraded":"2024-03-26T03:11:19.078147+0000","last_fullsized":"2024-03-26T03:11:19.078147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:07.220578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.220578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:08.544006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":452,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.079896+0000","last_change":"2024-03-26T03:10:20.768755+0000","last_active":"2024-03-26T03:11:19.079896+0000","last_peered":"2024-03-26T03:11:19.079896+0000","last_clean":"2024-03-26T03:11:19.079896+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:11:19.079896+0000","last_undegraded":"2024-03-26T03:11:19.079896+0000","last_fullsized":"2024-03-26T03:11:19.079896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:20.768642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:20.768642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:27.477402+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":631,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.078817+0000","last_change":"2024-03-26T03:10:39.370375+0000","last_active":"2024-03-26T03:11:19.078817+0000","last_peered":"2024-03-26T03:11:19.078817+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:10:39.370375+0000","last_became_peered":"2024-03-26T03:10:39.370375+0000","last_unstale":"2024-03-26T03:11:19.078817+0000","last_undegraded":"2024-03-26T03:11:19.078817+0000","last_fullsized":"2024-03-26T03:10:39.363411+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696234+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"268'14121","reported_seq":14292,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.396037+0000","last_change":"2024-03-26T03:10:54.851702+0000","last_active":"2024-03-26T03:11:19.396037+0000","last_peered":"2024-03-26T03:11:19.396037+0000","last_clean":"2024-03-26T03:11:19.396037+0000","last_became_active":"2024-03-26T03:08:43.068853+0000","last_became_peered":"2024-03-26T03:08:43.068853+0000","last_unstale":"2024-03-26T03:11:19.396037+0000","last_undegraded":"2024-03-26T03:11:19.396037+0000","last_fullsized":"2024-03-26T03:11:19.396037+0000","mapping_epoch":225,"log_start":"249'10000","ondisk_log_start":"249'10000","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'12055","last_scrub_stamp":"2024-03-26T03:10:54.851285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.851285+0000","objects_scrubbed":6498,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:06.097942+0000","scrub_duration":2294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283770904,"num_objects":4331,"num_object_clones":0,"num_object_copies":12993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14121,"num_write_kb":590401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":434,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.081891+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:11:19.081891+0000","last_peered":"2024-03-26T03:11:19.081891+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:11:19.081891+0000","last_undegraded":"2024-03-26T03:11:19.081891+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792826+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":275,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.077718+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:11:19.077718+0000","last_peered":"2024-03-26T03:11:19.077718+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:11:19.077718+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":466,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.075318+0000","last_change":"2024-03-26T03:09:50.291684+0000","last_active":"2024-03-26T03:11:19.075318+0000","last_peered":"2024-03-26T03:11:19.075318+0000","last_clean":"2024-03-26T03:11:19.075318+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:11:19.075318+0000","last_undegraded":"2024-03-26T03:11:19.075318+0000","last_fullsized":"2024-03-26T03:11:19.075318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:09:50.291459+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:09:50.291459+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.237881+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":430,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.083197+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:11:19.083197+0000","last_peered":"2024-03-26T03:11:19.083197+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:11:19.083197+0000","last_undegraded":"2024-03-26T03:11:19.083197+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792855+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":556,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.083771+0000","last_change":"2024-03-26T03:10:51.609357+0000","last_active":"2024-03-26T03:11:19.083771+0000","last_peered":"2024-03-26T03:11:19.083771+0000","last_clean":"2024-03-26T03:11:19.083771+0000","last_became_active":"2024-03-26T03:10:39.368150+0000","last_became_peered":"2024-03-26T03:10:39.368150+0000","last_unstale":"2024-03-26T03:11:19.083771+0000","last_undegraded":"2024-03-26T03:11:19.083771+0000","last_fullsized":"2024-03-26T03:11:19.083771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.609183+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.609183+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:14.028320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"268'13905","reported_seq":14097,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.424610+0000","last_change":"2024-03-26T03:10:21.408571+0000","last_active":"2024-03-26T03:11:19.424610+0000","last_peered":"2024-03-26T03:11:19.424610+0000","last_clean":"2024-03-26T03:11:19.424610+0000","last_became_active":"2024-03-26T03:08:43.067345+0000","last_became_peered":"2024-03-26T03:08:43.067345+0000","last_unstale":"2024-03-26T03:11:19.424610+0000","last_undegraded":"2024-03-26T03:11:19.424610+0000","last_fullsized":"2024-03-26T03:11:19.424610+0000","mapping_epoch":225,"log_start":"249'9784","ondisk_log_start":"249'9784","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'8823","last_scrub_stamp":"2024-03-26T03:10:21.407979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:21.407979+0000","objects_scrubbed":8714,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:39.548544+0000","scrub_duration":2784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280952832,"num_objects":4287,"num_object_clones":0,"num_object_copies":12861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":582144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.076396+0000","last_change":"2024-03-26T03:10:54.285604+0000","last_active":"2024-03-26T03:11:19.076396+0000","last_peered":"2024-03-26T03:11:19.076396+0000","last_clean":"2024-03-26T03:11:19.076396+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:11:19.076396+0000","last_undegraded":"2024-03-26T03:11:19.076396+0000","last_fullsized":"2024-03-26T03:11:19.076396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.285471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.285471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:11.594353+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":531,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.101530+0000","last_change":"2024-03-26T03:09:47.209344+0000","last_active":"2024-03-26T03:11:19.101530+0000","last_peered":"2024-03-26T03:11:19.101530+0000","last_clean":"2024-03-26T03:11:19.101530+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:11:19.101530+0000","last_undegraded":"2024-03-26T03:11:19.101530+0000","last_fullsized":"2024-03-26T03:11:19.101530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:09:47.209142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.209142+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.082698+0000","last_change":"2024-03-26T03:10:54.596765+0000","last_active":"2024-03-26T03:11:19.082698+0000","last_peered":"2024-03-26T03:11:19.082698+0000","last_clean":"2024-03-26T03:11:19.082698+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:11:19.082698+0000","last_undegraded":"2024-03-26T03:11:19.082698+0000","last_fullsized":"2024-03-26T03:11:19.082698+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.596513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.596513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.924242+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.072939+0000","last_change":"2024-03-26T03:11:16.222874+0000","last_active":"2024-03-26T03:11:19.072939+0000","last_peered":"2024-03-26T03:11:19.072939+0000","last_clean":"2024-03-26T03:11:19.072939+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:11:19.072939+0000","last_undegraded":"2024-03-26T03:11:19.072939+0000","last_fullsized":"2024-03-26T03:11:19.072939+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.222744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.635224+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":530,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.077298+0000","last_change":"2024-03-26T03:10:39.372465+0000","last_active":"2024-03-26T03:11:19.077298+0000","last_peered":"2024-03-26T03:11:19.077298+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:10:39.372465+0000","last_became_peered":"2024-03-26T03:10:39.372465+0000","last_unstale":"2024-03-26T03:11:19.077298+0000","last_undegraded":"2024-03-26T03:11:19.077298+0000","last_fullsized":"2024-03-26T03:10:39.360974+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:19.321845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"268'14065","reported_seq":14233,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:20.596269+0000","last_change":"2024-03-26T03:10:57.300217+0000","last_active":"2024-03-26T03:11:20.596269+0000","last_peered":"2024-03-26T03:11:20.596269+0000","last_clean":"2024-03-26T03:11:20.596269+0000","last_became_active":"2024-03-26T03:08:43.066837+0000","last_became_peered":"2024-03-26T03:08:43.066837+0000","last_unstale":"2024-03-26T03:11:20.596269+0000","last_undegraded":"2024-03-26T03:11:20.596269+0000","last_fullsized":"2024-03-26T03:11:20.596269+0000","mapping_epoch":225,"log_start":"246'8744","ondisk_log_start":"246'8744","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'12163","last_scrub_stamp":"2024-03-26T03:10:57.299961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.299961+0000","objects_scrubbed":6295,"log_size":5321,"log_dups_size":0,"ondisk_log_size":5321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.920679+0000","scrub_duration":1960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283049984,"num_objects":4319,"num_object_clones":0,"num_object_copies":12957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14065,"num_write_kb":588288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":471,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.083378+0000","last_change":"2024-03-26T03:11:13.563915+0000","last_active":"2024-03-26T03:11:19.083378+0000","last_peered":"2024-03-26T03:11:19.083378+0000","last_clean":"2024-03-26T03:11:19.083378+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:11:19.083378+0000","last_undegraded":"2024-03-26T03:11:19.083378+0000","last_fullsized":"2024-03-26T03:11:19.083378+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.563699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.563699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.093265+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":510,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.101616+0000","last_change":"2024-03-26T03:10:16.242987+0000","last_active":"2024-03-26T03:11:19.101616+0000","last_peered":"2024-03-26T03:11:19.101616+0000","last_clean":"2024-03-26T03:11:19.101616+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:11:19.101616+0000","last_undegraded":"2024-03-26T03:11:19.101616+0000","last_fullsized":"2024-03-26T03:11:19.101616+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:10:16.242828+0000","last_deep_scrub":"219'18","last_deep_scrub_stamp":"2024-03-26T03:08:11.142496+0000","last_clean_scrub_stamp":"2024-03-26T03:10:16.242828+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.466052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":473,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077352+0000","last_change":"2024-03-26T03:11:03.629624+0000","last_active":"2024-03-26T03:11:19.077352+0000","last_peered":"2024-03-26T03:11:19.077352+0000","last_clean":"2024-03-26T03:11:19.077352+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:11:19.077352+0000","last_undegraded":"2024-03-26T03:11:19.077352+0000","last_fullsized":"2024-03-26T03:11:19.077352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:03.629514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:03.629514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.141143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":458,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.100523+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:11:19.100523+0000","last_peered":"2024-03-26T03:11:19.100523+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:11:19.100523+0000","last_undegraded":"2024-03-26T03:11:19.100523+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.074777+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:11:19.074777+0000","last_peered":"2024-03-26T03:11:19.074777+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:11:19.074777+0000","last_undegraded":"2024-03-26T03:11:19.074777+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":556,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.074519+0000","last_change":"2024-03-26T03:10:52.006406+0000","last_active":"2024-03-26T03:11:19.074519+0000","last_peered":"2024-03-26T03:11:19.074519+0000","last_clean":"2024-03-26T03:11:19.074519+0000","last_became_active":"2024-03-26T03:10:39.379213+0000","last_became_peered":"2024-03-26T03:10:39.379213+0000","last_unstale":"2024-03-26T03:11:19.074519+0000","last_undegraded":"2024-03-26T03:11:19.074519+0000","last_fullsized":"2024-03-26T03:11:19.074519+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:52.006290+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:52.006290+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:08.240502+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"268'13826","reported_seq":13920,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:20.541689+0000","last_change":"2024-03-26T03:08:54.931470+0000","last_active":"2024-03-26T03:11:20.541689+0000","last_peered":"2024-03-26T03:11:20.541689+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.065260+0000","last_became_peered":"2024-03-26T03:08:43.065260+0000","last_unstale":"2024-03-26T03:11:20.541689+0000","last_undegraded":"2024-03-26T03:08:54.924879+0000","last_fullsized":"2024-03-26T03:08:43.057979+0000","mapping_epoch":225,"log_start":"246'8528","ondisk_log_start":"246'8528","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5298,"log_dups_size":0,"ondisk_log_size":5298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:25.544529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13826,"num_write_kb":572800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":4074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.079269+0000","last_change":"2024-03-26T03:09:52.697598+0000","last_active":"2024-03-26T03:11:19.079269+0000","last_peered":"2024-03-26T03:11:19.079269+0000","last_clean":"2024-03-26T03:11:19.079269+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:11:19.079269+0000","last_undegraded":"2024-03-26T03:11:19.079269+0000","last_fullsized":"2024-03-26T03:11:19.079269+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:52.697426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:09:52.697426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"231'28","reported_seq":657,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077183+0000","last_change":"2024-03-26T03:10:48.343491+0000","last_active":"2024-03-26T03:11:19.077183+0000","last_peered":"2024-03-26T03:11:19.077183+0000","last_clean":"2024-03-26T03:11:19.077183+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:11:19.077183+0000","last_undegraded":"2024-03-26T03:11:19.077183+0000","last_fullsized":"2024-03-26T03:11:19.077183+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:10:48.343341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:48.343341+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:59.894819+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":731,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077464+0000","last_change":"2024-03-26T03:10:47.568899+0000","last_active":"2024-03-26T03:11:19.077464+0000","last_peered":"2024-03-26T03:11:19.077464+0000","last_clean":"2024-03-26T03:11:19.077464+0000","last_became_active":"2024-03-26T03:10:39.378619+0000","last_became_peered":"2024-03-26T03:10:39.378619+0000","last_unstale":"2024-03-26T03:11:19.077464+0000","last_undegraded":"2024-03-26T03:11:19.077464+0000","last_fullsized":"2024-03-26T03:11:19.077464+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:47.568755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.568755+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.757752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":493,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081585+0000","last_change":"2024-03-26T03:11:14.586398+0000","last_active":"2024-03-26T03:11:19.081585+0000","last_peered":"2024-03-26T03:11:19.081585+0000","last_clean":"2024-03-26T03:11:19.081585+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:11:19.081585+0000","last_undegraded":"2024-03-26T03:11:19.081585+0000","last_fullsized":"2024-03-26T03:11:19.081585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:14.586295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:11:14.586295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:20.767744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":175,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.081897+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:11:19.081897+0000","last_peered":"2024-03-26T03:11:19.081897+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:11:19.081897+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":438,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.083457+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:11:19.083457+0000","last_peered":"2024-03-26T03:11:19.083457+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:11:19.083457+0000","last_undegraded":"2024-03-26T03:11:19.083457+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"268'13893","reported_seq":14080,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.416304+0000","last_change":"2024-03-26T03:10:14.319579+0000","last_active":"2024-03-26T03:11:19.416304+0000","last_peered":"2024-03-26T03:11:19.416304+0000","last_clean":"2024-03-26T03:11:19.416304+0000","last_became_active":"2024-03-26T03:08:43.066310+0000","last_became_peered":"2024-03-26T03:08:43.066310+0000","last_unstale":"2024-03-26T03:11:19.416304+0000","last_undegraded":"2024-03-26T03:11:19.416304+0000","last_fullsized":"2024-03-26T03:11:19.416304+0000","mapping_epoch":225,"log_start":"248'9385","ondisk_log_start":"248'9385","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'8262","last_scrub_stamp":"2024-03-26T03:10:14.319349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.319349+0000","objects_scrubbed":8182,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:37.047077+0000","scrub_duration":2505,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283049984,"num_objects":4319,"num_object_clones":0,"num_object_copies":12957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13893,"num_write_kb":582784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.102221+0000","last_change":"2024-03-26T03:11:15.024122+0000","last_active":"2024-03-26T03:11:19.102221+0000","last_peered":"2024-03-26T03:11:19.102221+0000","last_clean":"2024-03-26T03:11:19.102221+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:11:19.102221+0000","last_undegraded":"2024-03-26T03:11:19.102221+0000","last_fullsized":"2024-03-26T03:11:19.102221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.024002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.024002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:34.521104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":513,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073835+0000","last_change":"2024-03-26T03:11:11.116895+0000","last_active":"2024-03-26T03:11:19.073835+0000","last_peered":"2024-03-26T03:11:19.073835+0000","last_clean":"2024-03-26T03:11:19.073835+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:11:19.073835+0000","last_undegraded":"2024-03-26T03:11:19.073835+0000","last_fullsized":"2024-03-26T03:11:19.073835+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:11:11.116642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:11.116642+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.643762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.071766+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:11:19.071766+0000","last_peered":"2024-03-26T03:11:19.071766+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:11:19.071766+0000","last_undegraded":"2024-03-26T03:11:19.071766+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1215,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.072734+0000","last_change":"2024-03-26T03:10:18.161378+0000","last_active":"2024-03-26T03:11:19.072734+0000","last_peered":"2024-03-26T03:11:19.072734+0000","last_clean":"2024-03-26T03:11:19.072734+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:11:19.072734+0000","last_undegraded":"2024-03-26T03:11:19.072734+0000","last_fullsized":"2024-03-26T03:11:19.072734+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:10:18.161157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:18.161157+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:46.581275+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":732,"num_read_kb":727,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.082544+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:11:19.082544+0000","last_peered":"2024-03-26T03:11:19.082544+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:11:19.082544+0000","last_undegraded":"2024-03-26T03:11:19.082544+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696387+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":738,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078524+0000","last_change":"2024-03-26T03:10:46.593971+0000","last_active":"2024-03-26T03:11:19.078524+0000","last_peered":"2024-03-26T03:11:19.078524+0000","last_clean":"2024-03-26T03:11:19.078524+0000","last_became_active":"2024-03-26T03:10:39.379191+0000","last_became_peered":"2024-03-26T03:10:39.379191+0000","last_unstale":"2024-03-26T03:11:19.078524+0000","last_undegraded":"2024-03-26T03:11:19.078524+0000","last_fullsized":"2024-03-26T03:11:19.078524+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:46.593847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:46.593847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:02.975591+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"268'14128","reported_seq":14286,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:20.623327+0000","last_change":"2024-03-26T03:11:07.255990+0000","last_active":"2024-03-26T03:11:20.623327+0000","last_peered":"2024-03-26T03:11:20.623327+0000","last_clean":"2024-03-26T03:11:20.623327+0000","last_became_active":"2024-03-26T03:08:43.065826+0000","last_became_peered":"2024-03-26T03:08:43.065826+0000","last_unstale":"2024-03-26T03:11:20.623327+0000","last_undegraded":"2024-03-26T03:11:20.623327+0000","last_fullsized":"2024-03-26T03:11:20.623327+0000","mapping_epoch":225,"log_start":"246'8834","ondisk_log_start":"246'8834","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'13144","last_scrub_stamp":"2024-03-26T03:11:07.255669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.255669+0000","objects_scrubbed":5305,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:22.559544+0000","scrub_duration":1702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659072,"num_objects":4252,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14128,"num_write_kb":588160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.111396+0000","last_change":"2024-03-26T03:08:57.142251+0000","last_active":"2024-03-26T03:11:19.111396+0000","last_peered":"2024-03-26T03:11:19.111396+0000","last_clean":"2024-03-26T03:11:19.111396+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:11:19.111396+0000","last_undegraded":"2024-03-26T03:11:19.111396+0000","last_fullsized":"2024-03-26T03:11:19.111396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:57.142119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:08:57.142119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"231'58","reported_seq":686,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.073682+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:11:19.073682+0000","last_peered":"2024-03-26T03:11:19.073682+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:11:19.073682+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176133+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":265,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073256+0000","last_change":"2024-03-26T03:11:04.160279+0000","last_active":"2024-03-26T03:11:19.073256+0000","last_peered":"2024-03-26T03:11:19.073256+0000","last_clean":"2024-03-26T03:11:19.073256+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:11:19.073256+0000","last_undegraded":"2024-03-26T03:11:19.073256+0000","last_fullsized":"2024-03-26T03:11:19.073256+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:04.160195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:04.160195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:10.320752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.084913+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:11:19.084913+0000","last_peered":"2024-03-26T03:11:19.084913+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:11:19.084913+0000","last_undegraded":"2024-03-26T03:11:19.084913+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":562,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.100409+0000","last_change":"2024-03-26T03:10:47.195476+0000","last_active":"2024-03-26T03:11:19.100409+0000","last_peered":"2024-03-26T03:11:19.100409+0000","last_clean":"2024-03-26T03:11:19.100409+0000","last_became_active":"2024-03-26T03:10:39.376563+0000","last_became_peered":"2024-03-26T03:10:39.376563+0000","last_unstale":"2024-03-26T03:11:19.100409+0000","last_undegraded":"2024-03-26T03:11:19.100409+0000","last_fullsized":"2024-03-26T03:11:19.100409+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:47.195197+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.195197+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:50.495199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"268'13915","reported_seq":14009,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.431136+0000","last_change":"2024-03-26T03:08:54.940176+0000","last_active":"2024-03-26T03:11:19.431136+0000","last_peered":"2024-03-26T03:11:19.431136+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.067789+0000","last_became_peered":"2024-03-26T03:08:43.067789+0000","last_unstale":"2024-03-26T03:11:19.431136+0000","last_undegraded":"2024-03-26T03:08:54.937224+0000","last_fullsized":"2024-03-26T03:08:43.062110+0000","mapping_epoch":225,"log_start":"248'9345","ondisk_log_start":"248'9345","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281477120,"num_objects":4295,"num_object_clones":0,"num_object_copies":12885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13915,"num_write_kb":582720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4295}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":624,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.102732+0000","last_change":"2024-03-26T03:11:14.528793+0000","last_active":"2024-03-26T03:11:19.102732+0000","last_peered":"2024-03-26T03:11:19.102732+0000","last_clean":"2024-03-26T03:11:19.102732+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:11:19.102732+0000","last_undegraded":"2024-03-26T03:11:19.102732+0000","last_fullsized":"2024-03-26T03:11:19.102732+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:11:14.528661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:14.528661+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.412124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":511,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073656+0000","last_change":"2024-03-26T03:10:55.068251+0000","last_active":"2024-03-26T03:11:19.073656+0000","last_peered":"2024-03-26T03:11:19.073656+0000","last_clean":"2024-03-26T03:11:19.073656+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:11:19.073656+0000","last_undegraded":"2024-03-26T03:11:19.073656+0000","last_fullsized":"2024-03-26T03:11:19.073656+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:10:55.067970+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.067970+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.607085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.074178+0000","last_change":"2024-03-26T03:11:10.075730+0000","last_active":"2024-03-26T03:11:19.074178+0000","last_peered":"2024-03-26T03:11:19.074178+0000","last_clean":"2024-03-26T03:11:19.074178+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:11:19.074178+0000","last_undegraded":"2024-03-26T03:11:19.074178+0000","last_fullsized":"2024-03-26T03:11:19.074178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:10.075607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:10.075607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:26.228686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":440,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.081248+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:11:19.081248+0000","last_peered":"2024-03-26T03:11:19.081248+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:11:19.081248+0000","last_undegraded":"2024-03-26T03:11:19.081248+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":530,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.077874+0000","last_change":"2024-03-26T03:10:39.371201+0000","last_active":"2024-03-26T03:11:19.077874+0000","last_peered":"2024-03-26T03:11:19.077874+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:10:39.371201+0000","last_became_peered":"2024-03-26T03:10:39.371201+0000","last_unstale":"2024-03-26T03:11:19.077874+0000","last_undegraded":"2024-03-26T03:11:19.077874+0000","last_fullsized":"2024-03-26T03:10:39.361046+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:19.321862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":335,"reported_epoch":268,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:11:19.102230+0000","last_change":"2024-03-26T03:10:50.310997+0000","last_active":"2024-03-26T03:11:19.102230+0000","last_peered":"2024-03-26T03:11:19.102230+0000","last_clean":"2024-03-26T03:11:19.102230+0000","last_became_active":"2024-03-26T03:10:44.081411+0000","last_became_peered":"2024-03-26T03:10:44.081411+0000","last_unstale":"2024-03-26T03:11:19.102230+0000","last_undegraded":"2024-03-26T03:11:19.102230+0000","last_fullsized":"2024-03-26T03:11:19.102230+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.310701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.310701+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.015695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"268'13975","reported_seq":14069,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.866798+0000","last_change":"2024-03-26T03:08:54.931125+0000","last_active":"2024-03-26T03:11:19.866798+0000","last_peered":"2024-03-26T03:11:19.866798+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.058021+0000","last_became_peered":"2024-03-26T03:08:43.058021+0000","last_unstale":"2024-03-26T03:11:19.866798+0000","last_undegraded":"2024-03-26T03:08:54.930155+0000","last_fullsized":"2024-03-26T03:08:43.054400+0000","mapping_epoch":225,"log_start":"248'9537","ondisk_log_start":"248'9537","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.177985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280821760,"num_objects":4285,"num_object_clones":0,"num_object_copies":12855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13975,"num_write_kb":584320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4285}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.071705+0000","last_change":"2024-03-26T03:10:50.022827+0000","last_active":"2024-03-26T03:11:19.071705+0000","last_peered":"2024-03-26T03:11:19.071705+0000","last_clean":"2024-03-26T03:11:19.071705+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:11:19.071705+0000","last_undegraded":"2024-03-26T03:11:19.071705+0000","last_fullsized":"2024-03-26T03:11:19.071705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:50.022721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.022721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:08.998611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":516,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.081881+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:11:19.081881+0000","last_peered":"2024-03-26T03:11:19.081881+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:11:19.081881+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792772+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":503,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084756+0000","last_change":"2024-03-26T03:10:58.571618+0000","last_active":"2024-03-26T03:11:19.084756+0000","last_peered":"2024-03-26T03:11:19.084756+0000","last_clean":"2024-03-26T03:11:19.084756+0000","last_became_active":"2024-03-26T03:10:08.555235+0000","last_became_peered":"2024-03-26T03:10:08.555235+0000","last_unstale":"2024-03-26T03:11:19.084756+0000","last_undegraded":"2024-03-26T03:11:19.084756+0000","last_fullsized":"2024-03-26T03:11:19.084756+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:10:58.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:58.571492+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.423266+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.079379+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:11:19.079379+0000","last_peered":"2024-03-26T03:11:19.079379+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:11:19.079379+0000","last_undegraded":"2024-03-26T03:11:19.079379+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":559,"reported_epoch":268,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:11:19.073077+0000","last_change":"2024-03-26T03:10:48.023253+0000","last_active":"2024-03-26T03:11:19.073077+0000","last_peered":"2024-03-26T03:11:19.073077+0000","last_clean":"2024-03-26T03:11:19.073077+0000","last_became_active":"2024-03-26T03:10:42.404423+0000","last_became_peered":"2024-03-26T03:10:42.404423+0000","last_unstale":"2024-03-26T03:11:19.073077+0000","last_undegraded":"2024-03-26T03:11:19.073077+0000","last_fullsized":"2024-03-26T03:11:19.073077+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:48.023049+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:48.023049+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:56.074769+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":496,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.101623+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:11:19.101623+0000","last_peered":"2024-03-26T03:11:19.101623+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:11:19.101623+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":486,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.071565+0000","last_change":"2024-03-26T03:11:14.000120+0000","last_active":"2024-03-26T03:11:19.071565+0000","last_peered":"2024-03-26T03:11:19.071565+0000","last_clean":"2024-03-26T03:11:19.071565+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:11:19.071565+0000","last_undegraded":"2024-03-26T03:11:19.071565+0000","last_fullsized":"2024-03-26T03:11:19.071565+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:11:13.999986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.999986+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:19.357060+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.073421+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:11:19.073421+0000","last_peered":"2024-03-26T03:11:19.073421+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:11:19.073421+0000","last_undegraded":"2024-03-26T03:11:19.073421+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":460,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.100202+0000","last_change":"2024-03-26T03:09:53.347817+0000","last_active":"2024-03-26T03:11:19.100202+0000","last_peered":"2024-03-26T03:11:19.100202+0000","last_clean":"2024-03-26T03:11:19.100202+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:11:19.100202+0000","last_undegraded":"2024-03-26T03:11:19.100202+0000","last_fullsized":"2024-03-26T03:11:19.100202+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:53.347679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:09:53.347679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.033250+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":335,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081015+0000","last_change":"2024-03-26T03:10:50.548723+0000","last_active":"2024-03-26T03:11:19.081015+0000","last_peered":"2024-03-26T03:11:19.081015+0000","last_clean":"2024-03-26T03:11:19.081015+0000","last_became_active":"2024-03-26T03:10:44.085377+0000","last_became_peered":"2024-03-26T03:10:44.085377+0000","last_unstale":"2024-03-26T03:11:19.081015+0000","last_undegraded":"2024-03-26T03:11:19.081015+0000","last_fullsized":"2024-03-26T03:11:19.081015+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.548591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.548591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.314046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":198,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.079502+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:11:19.079502+0000","last_peered":"2024-03-26T03:11:19.079502+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:11:19.079502+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078275+0000","last_change":"2024-03-26T03:11:13.331187+0000","last_active":"2024-03-26T03:11:19.078275+0000","last_peered":"2024-03-26T03:11:19.078275+0000","last_clean":"2024-03-26T03:11:19.078275+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:11:19.078275+0000","last_undegraded":"2024-03-26T03:11:19.078275+0000","last_fullsized":"2024-03-26T03:11:19.078275+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.331065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.331065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:16.385611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":442,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.080705+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:11:19.080705+0000","last_peered":"2024-03-26T03:11:19.080705+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:11:19.080705+0000","last_undegraded":"2024-03-26T03:11:19.080705+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":444,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.082203+0000","last_change":"2024-03-26T03:09:57.777835+0000","last_active":"2024-03-26T03:11:19.082203+0000","last_peered":"2024-03-26T03:11:19.082203+0000","last_clean":"2024-03-26T03:11:19.082203+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:11:19.082203+0000","last_undegraded":"2024-03-26T03:11:19.082203+0000","last_fullsized":"2024-03-26T03:11:19.082203+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:57.777717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.777717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.671314+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":565,"reported_epoch":268,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:11:19.081328+0000","last_change":"2024-03-26T03:10:49.282505+0000","last_active":"2024-03-26T03:11:19.081328+0000","last_peered":"2024-03-26T03:11:19.081328+0000","last_clean":"2024-03-26T03:11:19.081328+0000","last_became_active":"2024-03-26T03:10:42.404630+0000","last_became_peered":"2024-03-26T03:10:42.404630+0000","last_unstale":"2024-03-26T03:11:19.081328+0000","last_undegraded":"2024-03-26T03:11:19.081328+0000","last_fullsized":"2024-03-26T03:11:19.081328+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.282258+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.282258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:57.713800+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":397,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.074909+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:11:19.074909+0000","last_peered":"2024-03-26T03:11:19.074909+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:11:19.074909+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":458,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.081372+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:11:19.081372+0000","last_peered":"2024-03-26T03:11:19.081372+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:11:19.081372+0000","last_undegraded":"2024-03-26T03:11:19.081372+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":621,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.090586+0000","last_change":"2024-03-26T03:09:50.922297+0000","last_active":"2024-03-26T03:11:19.090586+0000","last_peered":"2024-03-26T03:11:19.090586+0000","last_clean":"2024-03-26T03:11:19.090586+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:11:19.090586+0000","last_undegraded":"2024-03-26T03:11:19.090586+0000","last_fullsized":"2024-03-26T03:11:19.090586+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:09:50.922184+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:09:50.922184+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":439,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.101386+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:11:19.101386+0000","last_peered":"2024-03-26T03:11:19.101386+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:11:19.101386+0000","last_undegraded":"2024-03-26T03:11:19.101386+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":738,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077210+0000","last_change":"2024-03-26T03:10:49.610796+0000","last_active":"2024-03-26T03:11:19.077210+0000","last_peered":"2024-03-26T03:11:19.077210+0000","last_clean":"2024-03-26T03:11:19.077210+0000","last_became_active":"2024-03-26T03:10:44.085632+0000","last_became_peered":"2024-03-26T03:10:44.085632+0000","last_unstale":"2024-03-26T03:11:19.077210+0000","last_undegraded":"2024-03-26T03:11:19.077210+0000","last_fullsized":"2024-03-26T03:11:19.077210+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.610676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.610676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.952827+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":291,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.082367+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:11:19.082367+0000","last_peered":"2024-03-26T03:11:19.082367+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:11:19.082367+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":443,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.101639+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:11:19.101639+0000","last_peered":"2024-03-26T03:11:19.101639+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:11:19.101639+0000","last_undegraded":"2024-03-26T03:11:19.101639+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":235,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.079364+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:11:19.079364+0000","last_peered":"2024-03-26T03:11:19.079364+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:11:19.079364+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":452,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.085020+0000","last_change":"2024-03-26T03:11:07.697991+0000","last_active":"2024-03-26T03:11:19.085020+0000","last_peered":"2024-03-26T03:11:19.085020+0000","last_clean":"2024-03-26T03:11:19.085020+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:11:19.085020+0000","last_undegraded":"2024-03-26T03:11:19.085020+0000","last_fullsized":"2024-03-26T03:11:19.085020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:11:07.697855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.697855+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:32.059699+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"230'15","reported_seq":485,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073550+0000","last_change":"2024-03-26T03:10:13.175140+0000","last_active":"2024-03-26T03:11:19.073550+0000","last_peered":"2024-03-26T03:11:19.073550+0000","last_clean":"2024-03-26T03:11:19.073550+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:11:19.073550+0000","last_undegraded":"2024-03-26T03:11:19.073550+0000","last_fullsized":"2024-03-26T03:11:19.073550+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'15","last_scrub_stamp":"2024-03-26T03:10:13.174994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:13.174994+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:35.656575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.076736+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:11:19.076736+0000","last_peered":"2024-03-26T03:11:19.076736+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:11:19.076736+0000","last_undegraded":"2024-03-26T03:11:19.076736+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235771+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.102553+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:11:19.102553+0000","last_peered":"2024-03-26T03:11:19.102553+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:11:19.102553+0000","last_undegraded":"2024-03-26T03:11:19.102553+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081213+0000","last_change":"2024-03-26T03:11:13.575433+0000","last_active":"2024-03-26T03:11:19.081213+0000","last_peered":"2024-03-26T03:11:19.081213+0000","last_clean":"2024-03-26T03:11:19.081213+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:11:19.081213+0000","last_undegraded":"2024-03-26T03:11:19.081213+0000","last_fullsized":"2024-03-26T03:11:19.081213+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.575348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.575348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.771654+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":490,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.082240+0000","last_change":"2024-03-26T03:09:48.867245+0000","last_active":"2024-03-26T03:11:19.082240+0000","last_peered":"2024-03-26T03:11:19.082240+0000","last_clean":"2024-03-26T03:11:19.082240+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:11:19.082240+0000","last_undegraded":"2024-03-26T03:11:19.082240+0000","last_fullsized":"2024-03-26T03:11:19.082240+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:09:48.867101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:48.867101+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073563+0000","last_change":"2024-03-26T03:10:54.105195+0000","last_active":"2024-03-26T03:11:19.073563+0000","last_peered":"2024-03-26T03:11:19.073563+0000","last_clean":"2024-03-26T03:11:19.073563+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:11:19.073563+0000","last_undegraded":"2024-03-26T03:11:19.073563+0000","last_fullsized":"2024-03-26T03:11:19.073563+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.105074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.105074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.061523+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":435,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.073768+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:11:19.073768+0000","last_peered":"2024-03-26T03:11:19.073768+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:11:19.073768+0000","last_undegraded":"2024-03-26T03:11:19.073768+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.100806+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:11:19.100806+0000","last_peered":"2024-03-26T03:11:19.100806+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:11:19.100806+0000","last_undegraded":"2024-03-26T03:11:19.100806+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":180,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.072734+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:11:19.072734+0000","last_peered":"2024-03-26T03:11:19.072734+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:11:19.072734+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081149+0000","last_change":"2024-03-26T03:10:48.610870+0000","last_active":"2024-03-26T03:11:19.081149+0000","last_peered":"2024-03-26T03:11:19.081149+0000","last_clean":"2024-03-26T03:11:19.081149+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:11:19.081149+0000","last_undegraded":"2024-03-26T03:11:19.081149+0000","last_fullsized":"2024-03-26T03:11:19.081149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:48.610770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:10:48.610770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:16.686703+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078197+0000","last_change":"2024-03-26T03:10:19.233349+0000","last_active":"2024-03-26T03:11:19.078197+0000","last_peered":"2024-03-26T03:11:19.078197+0000","last_clean":"2024-03-26T03:11:19.078197+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:11:19.078197+0000","last_undegraded":"2024-03-26T03:11:19.078197+0000","last_fullsized":"2024-03-26T03:11:19.078197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_clean_scrub_stamp":"2024-03-26T03:10:19.233223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:22.070407+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":460,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078621+0000","last_change":"2024-03-26T03:11:01.548791+0000","last_active":"2024-03-26T03:11:19.078621+0000","last_peered":"2024-03-26T03:11:19.078621+0000","last_clean":"2024-03-26T03:11:19.078621+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:11:19.078621+0000","last_undegraded":"2024-03-26T03:11:19.078621+0000","last_fullsized":"2024-03-26T03:11:19.078621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.548673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.548673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:25.289328+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":337,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.072168+0000","last_change":"2024-03-26T03:10:51.073689+0000","last_active":"2024-03-26T03:11:19.072168+0000","last_peered":"2024-03-26T03:11:19.072168+0000","last_clean":"2024-03-26T03:11:19.072168+0000","last_became_active":"2024-03-26T03:10:44.079530+0000","last_became_peered":"2024-03-26T03:10:44.079530+0000","last_unstale":"2024-03-26T03:11:19.072168+0000","last_undegraded":"2024-03-26T03:11:19.072168+0000","last_fullsized":"2024-03-26T03:11:19.072168+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.073552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.073552+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:57.802836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":384,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.078524+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:11:19.078524+0000","last_peered":"2024-03-26T03:11:19.078524+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:11:19.078524+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078067+0000","last_change":"2024-03-26T03:09:57.334690+0000","last_active":"2024-03-26T03:11:19.078067+0000","last_peered":"2024-03-26T03:11:19.078067+0000","last_clean":"2024-03-26T03:11:19.078067+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:11:19.078067+0000","last_undegraded":"2024-03-26T03:11:19.078067+0000","last_fullsized":"2024-03-26T03:11:19.078067+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:57.334539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.334539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235826+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":464,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.077666+0000","last_change":"2024-03-26T03:10:14.660979+0000","last_active":"2024-03-26T03:11:19.077666+0000","last_peered":"2024-03-26T03:11:19.077666+0000","last_clean":"2024-03-26T03:11:19.077666+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:11:19.077666+0000","last_undegraded":"2024-03-26T03:11:19.077666+0000","last_fullsized":"2024-03-26T03:11:19.077666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:14.660845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.660845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:43.846150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.101214+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:11:19.101214+0000","last_peered":"2024-03-26T03:11:19.101214+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:11:19.101214+0000","last_undegraded":"2024-03-26T03:11:19.101214+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":490,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.102271+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:11:19.102271+0000","last_peered":"2024-03-26T03:11:19.102271+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:11:19.102271+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.080423+0000","last_change":"2024-03-26T03:10:47.576491+0000","last_active":"2024-03-26T03:11:19.080423+0000","last_peered":"2024-03-26T03:11:19.080423+0000","last_clean":"2024-03-26T03:11:19.080423+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:11:19.080423+0000","last_undegraded":"2024-03-26T03:11:19.080423+0000","last_fullsized":"2024-03-26T03:11:19.080423+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:47.576286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.576286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:14.591656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":624,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078229+0000","last_change":"2024-03-26T03:10:51.299024+0000","last_active":"2024-03-26T03:11:19.078229+0000","last_peered":"2024-03-26T03:11:19.078229+0000","last_clean":"2024-03-26T03:11:19.078229+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:11:19.078229+0000","last_undegraded":"2024-03-26T03:11:19.078229+0000","last_fullsized":"2024-03-26T03:11:19.078229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:10:51.298893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.298893+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.004130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.079557+0000","last_change":"2024-03-26T03:10:57.591571+0000","last_active":"2024-03-26T03:11:19.079557+0000","last_peered":"2024-03-26T03:11:19.079557+0000","last_clean":"2024-03-26T03:11:19.079557+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:11:19.079557+0000","last_undegraded":"2024-03-26T03:11:19.079557+0000","last_fullsized":"2024-03-26T03:11:19.079557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:10:57.591453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.591453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:58.969286+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":503,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078317+0000","last_change":"2024-03-26T03:09:47.385319+0000","last_active":"2024-03-26T03:11:19.078317+0000","last_peered":"2024-03-26T03:11:19.078317+0000","last_clean":"2024-03-26T03:11:19.078317+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:11:19.078317+0000","last_undegraded":"2024-03-26T03:11:19.078317+0000","last_fullsized":"2024-03-26T03:11:19.078317+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:09:47.384690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:47.384690+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":157,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.078008+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:11:19.078008+0000","last_peered":"2024-03-26T03:11:19.078008+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:11:19.078008+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":465,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.102576+0000","last_change":"2024-03-26T03:10:20.281155+0000","last_active":"2024-03-26T03:11:19.102576+0000","last_peered":"2024-03-26T03:11:19.102576+0000","last_clean":"2024-03-26T03:11:19.102576+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:11:19.102576+0000","last_undegraded":"2024-03-26T03:11:19.102576+0000","last_fullsized":"2024-03-26T03:11:19.102576+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:10:20.281044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:33.169884+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":451,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.073343+0000","last_change":"2024-03-26T03:10:57.204904+0000","last_active":"2024-03-26T03:11:19.073343+0000","last_peered":"2024-03-26T03:11:19.073343+0000","last_clean":"2024-03-26T03:11:19.073343+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:11:19.073343+0000","last_undegraded":"2024-03-26T03:11:19.073343+0000","last_fullsized":"2024-03-26T03:11:19.073343+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:57.204763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.204763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.985937+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":489,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.100523+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:11:19.100523+0000","last_peered":"2024-03-26T03:11:19.100523+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:11:19.100523+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.076309+0000","last_change":"2024-03-26T03:11:16.283604+0000","last_active":"2024-03-26T03:11:19.076309+0000","last_peered":"2024-03-26T03:11:19.076309+0000","last_clean":"2024-03-26T03:11:19.076309+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:11:19.076309+0000","last_undegraded":"2024-03-26T03:11:19.076309+0000","last_fullsized":"2024-03-26T03:11:19.076309+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.283475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.283475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:45.688944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.074124+0000","last_change":"2024-03-26T03:10:56.053025+0000","last_active":"2024-03-26T03:11:19.074124+0000","last_peered":"2024-03-26T03:11:19.074124+0000","last_clean":"2024-03-26T03:11:19.074124+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:11:19.074124+0000","last_undegraded":"2024-03-26T03:11:19.074124+0000","last_fullsized":"2024-03-26T03:11:19.074124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:56.052911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:10:56.052911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.280129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":460,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.084486+0000","last_change":"2024-03-26T03:11:15.795645+0000","last_active":"2024-03-26T03:11:19.084486+0000","last_peered":"2024-03-26T03:11:19.084486+0000","last_clean":"2024-03-26T03:11:19.084486+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:11:19.084486+0000","last_undegraded":"2024-03-26T03:11:19.084486+0000","last_fullsized":"2024-03-26T03:11:19.084486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.795433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.795433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:32.594054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":212,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.074027+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:11:19.074027+0000","last_peered":"2024-03-26T03:11:19.074027+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:11:19.074027+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.176040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":148,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.079640+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:11:19.079640+0000","last_peered":"2024-03-26T03:11:19.079640+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:11:19.079640+0000","last_undegraded":"2024-03-26T03:11:19.079640+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":433,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.083638+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:11:19.083638+0000","last_peered":"2024-03-26T03:11:19.083638+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:11:19.083638+0000","last_undegraded":"2024-03-26T03:11:19.083638+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.078406+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:11:19.078406+0000","last_peered":"2024-03-26T03:11:19.078406+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:11:19.078406+0000","last_undegraded":"2024-03-26T03:11:19.078406+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:23.235779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":500,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.082858+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:11:19.082858+0000","last_peered":"2024-03-26T03:11:19.082858+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:11:19.082858+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792841+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.079095+0000","last_change":"2024-03-26T03:10:04.678447+0000","last_active":"2024-03-26T03:11:19.079095+0000","last_peered":"2024-03-26T03:11:19.079095+0000","last_clean":"2024-03-26T03:11:19.079095+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:11:19.079095+0000","last_undegraded":"2024-03-26T03:11:19.079095+0000","last_fullsized":"2024-03-26T03:11:19.079095+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:04.678325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:10:04.678325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:32.014387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":470,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081510+0000","last_change":"2024-03-26T03:11:16.659920+0000","last_active":"2024-03-26T03:11:19.081510+0000","last_peered":"2024-03-26T03:11:19.081510+0000","last_clean":"2024-03-26T03:11:19.081510+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:11:19.081510+0000","last_undegraded":"2024-03-26T03:11:19.081510+0000","last_fullsized":"2024-03-26T03:11:19.081510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.659717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.659717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:28.148413+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":460,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.100444+0000","last_change":"2024-03-26T03:09:59.221838+0000","last_active":"2024-03-26T03:11:19.100444+0000","last_peered":"2024-03-26T03:11:19.100444+0000","last_clean":"2024-03-26T03:11:19.100444+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:11:19.100444+0000","last_undegraded":"2024-03-26T03:11:19.100444+0000","last_fullsized":"2024-03-26T03:11:19.100444+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:59.221596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:59.221596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":462,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.082373+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:11:19.082373+0000","last_peered":"2024-03-26T03:11:19.082373+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:11:19.082373+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:20.792813+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":471,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.082289+0000","last_change":"2024-03-26T03:11:16.999880+0000","last_active":"2024-03-26T03:11:19.082289+0000","last_peered":"2024-03-26T03:11:19.082289+0000","last_clean":"2024-03-26T03:11:19.082289+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:11:19.082289+0000","last_undegraded":"2024-03-26T03:11:19.082289+0000","last_fullsized":"2024-03-26T03:11:19.082289+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.999769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.999769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:22.404177+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":455,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.078303+0000","last_change":"2024-03-26T03:10:02.681646+0000","last_active":"2024-03-26T03:11:19.078303+0000","last_peered":"2024-03-26T03:11:19.078303+0000","last_clean":"2024-03-26T03:11:19.078303+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:11:19.078303+0000","last_undegraded":"2024-03-26T03:11:19.078303+0000","last_fullsized":"2024-03-26T03:11:19.078303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:02.681454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:02.681454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:24.670985+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":438,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.080910+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:11:19.080910+0000","last_peered":"2024-03-26T03:11:19.080910+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:11:19.080910+0000","last_undegraded":"2024-03-26T03:11:19.080910+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":496,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081348+0000","last_change":"2024-03-26T03:09:57.820889+0000","last_active":"2024-03-26T03:11:19.081348+0000","last_peered":"2024-03-26T03:11:19.081348+0000","last_clean":"2024-03-26T03:11:19.081348+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:11:19.081348+0000","last_undegraded":"2024-03-26T03:11:19.081348+0000","last_fullsized":"2024-03-26T03:11:19.081348+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:09:57.820746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:09:57.820746+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":149,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.101721+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:11:19.101721+0000","last_peered":"2024-03-26T03:11:19.101721+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:11:19.101721+0000","last_undegraded":"2024-03-26T03:11:19.101721+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":456,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.072744+0000","last_change":"2024-03-26T03:10:59.074818+0000","last_active":"2024-03-26T03:11:19.072744+0000","last_peered":"2024-03-26T03:11:19.072744+0000","last_clean":"2024-03-26T03:11:19.072744+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:11:19.072744+0000","last_undegraded":"2024-03-26T03:11:19.072744+0000","last_fullsized":"2024-03-26T03:11:19.072744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:59.074364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:59.074364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:14.061608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":450,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.082766+0000","last_change":"2024-03-26T03:09:53.849576+0000","last_active":"2024-03-26T03:11:19.082766+0000","last_peered":"2024-03-26T03:11:19.082766+0000","last_clean":"2024-03-26T03:11:19.082766+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:11:19.082766+0000","last_undegraded":"2024-03-26T03:11:19.082766+0000","last_fullsized":"2024-03-26T03:11:19.082766+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:09:53.849440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:09:53.849440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":260,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:11:19.072046+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:11:19.072046+0000","last_peered":"2024-03-26T03:11:19.072046+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:11:19.072046+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":472,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.101386+0000","last_change":"2024-03-26T03:10:11.215249+0000","last_active":"2024-03-26T03:11:19.101386+0000","last_peered":"2024-03-26T03:11:19.101386+0000","last_clean":"2024-03-26T03:11:19.101386+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:11:19.101386+0000","last_undegraded":"2024-03-26T03:11:19.101386+0000","last_fullsized":"2024-03-26T03:11:19.101386+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:11.215134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:11.215134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:33.383839+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":463,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-03-26T03:11:19.081585+0000","last_change":"2024-03-26T03:10:52.646355+0000","last_active":"2024-03-26T03:11:19.081585+0000","last_peered":"2024-03-26T03:11:19.081585+0000","last_clean":"2024-03-26T03:11:19.081585+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:11:19.081585+0000","last_undegraded":"2024-03-26T03:11:19.081585+0000","last_fullsized":"2024-03-26T03:11:19.081585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:52.646244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:52.646244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.583056+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":148,"reported_epoch":268,"state":"active+undersized","last_fresh":"2024-03-26T03:11:19.078486+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:11:19.078486+0000","last_peered":"2024-03-26T03:11:19.078486+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:11:19.078486+0000","last_undegraded":"2024-03-26T03:11:19.078486+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:11:21.696363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4501536792,"num_objects":68689,"num_object_clones":0,"num_object_copies":206067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":223909,"num_write_kb":9363073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11290030080,"data_stored":11290017864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76269,"ondisk_log_size":76269,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":608258,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":54,"acting":58,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":732,"num_read_kb":727,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1410,"num_read_kb":1135,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":940,"ondisk_log_size":940,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103321,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3117644,"kb_used_data":2531140,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":90627508,"statfs":{"total":95995035648,"available":92802568192,"internally_reserved":0,"allocated":2591887360,"data_stored":2591308474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":600570314},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430365,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241552,"kb_used_data":849412,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92503600,"statfs":{"total":95995035648,"available":94723686400,"internally_reserved":0,"allocated":869797888,"data_stored":869225447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3078152,"kb_used_data":2487480,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":90667000,"statfs":{"total":95995035648,"available":92843008000,"internally_reserved":0,"allocated":2547179520,"data_stored":2546556203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738555,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557972,"kb_used_data":1085320,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92187180,"statfs":{"total":95995035648,"available":94399672320,"internally_reserved":0,"allocated":1111367680,"data_stored":1110791886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411504,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241992,"kb_used_data":831804,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92503160,"statfs":{"total":95995035648,"available":94723235840,"internally_reserved":0,"allocated":851767296,"data_stored":851197561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986343,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157204,"kb_used_data":837196,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92587948,"statfs":{"total":95995035648,"available":94810058752,"internally_reserved":0,"allocated":857288704,"data_stored":856701511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692003,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3084300,"kb_used_data":2521412,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90660852,"statfs":{"total":95995035648,"available":92836712448,"internally_reserved":0,"allocated":2581925888,"data_stored":2581347006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1108279296,"data_stored":1108279296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2517110784,"data_stored":2517106712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":843382784,"data_stored":843382784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":850591744,"data_stored":850591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2545750016,"data_stored":2545745944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":866910208,"data_stored":866910208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2558005248,"data_stored":2558001176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:11:27.428 INFO:tasks.thrashosds.thrasher:Setting 4.17 to [4, 3, 1, 2, 0, 5] 2024-03-26T03:11:27.428 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.17', '4', '3', '1', '2', '0', '5'] 2024-03-26T03:11:27.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 4.17 4 3 1 2 0 5 2024-03-26T03:11:27.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:27.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:27.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:27.572 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:27.614 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:27.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:27.682 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:27 smithi066 ceph-mon[37920]: pgmap v545: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 31657/189918 objects degraded (16.669%); 1/189918 objects misplaced (0.001%) 2024-03-26T03:11:27.682 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:27 smithi066 ceph-mon[29666]: pgmap v545: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 31657/189918 objects degraded (16.669%); 1/189918 objects misplaced (0.001%) 2024-03-26T03:11:27.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:27 smithi192 ceph-mon[33110]: pgmap v545: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.32k op/s; 31657/189918 objects degraded (16.669%); 1/189918 objects misplaced (0.001%) 2024-03-26T03:11:27.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:27.754 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:27.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:27.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:27.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:27.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:28.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:28.023 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:28.054 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:28.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:28.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:28.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.297 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:28.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:28.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:28.538 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:28.580 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:28.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:28.709 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:11:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:11:28.710 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:11:28.710 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:28.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:28.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:28.951 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:28.989 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:28.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:29.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.129 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:29.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:29.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.272 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:29.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:29.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:29.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:29.438 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:11:29.510 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.550 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:29.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:29.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:11:29.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:29.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:29 smithi192 ceph-mon[33110]: pgmap v546: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 31657/189918 objects degraded (16.669%); 1/189918 objects misplaced (0.001%) 2024-03-26T03:11:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:29 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/675846330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 31657/189918 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:29.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:11:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:11:29.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:29.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:29.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:29 smithi066 ceph-mon[29666]: pgmap v546: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 31657/189918 objects degraded (16.669%); 1/189918 objects misplaced (0.001%) 2024-03-26T03:11:29.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:29 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/675846330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:29.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 31657/189918 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:29.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:29 smithi066 ceph-mon[37920]: pgmap v546: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 31657/189918 objects degraded (16.669%); 1/189918 objects misplaced (0.001%) 2024-03-26T03:11:29.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:29 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/675846330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:29.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 31657/189918 objects degraded (16.669%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:29.953 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:29.982 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:29.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:29.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:30.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:30.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:30.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:30.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:30.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:30.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:30.388 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:30.436 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:30.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:30.537 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:30.574 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:30.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:30.675 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:30.710 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:30.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:30.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:11:30.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:11:30.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:30.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/675846330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:30.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[29666]: osdmap e269: 8 total, 7 up, 8 in 2024-03-26T03:11:30.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:11:30.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:11:30.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:30.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/675846330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:30.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:30 smithi066 ceph-mon[37920]: osdmap e269: 8 total, 7 up, 8 in 2024-03-26T03:11:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:30 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:11:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:30 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:11:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:30 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:30 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/675846330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:30 smithi192 ceph-mon[33110]: osdmap e269: 8 total, 7 up, 8 in 2024-03-26T03:11:30.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:30.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:30.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:30.952 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:31.007 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:31.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:31.147 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:31.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:31.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:31.294 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:31.377 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:11:31.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:31.420 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:11:31.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:31.454 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:11:31.510 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:11:31.511 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:31.511 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:31 smithi066 ceph-mon[29666]: pgmap v548: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 29252/177633 objects degraded (16.468%); 1/177633 objects misplaced (0.001%) 2024-03-26T03:11:31.511 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:31 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3372800713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:31.512 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:31.512 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:31 smithi066 ceph-mon[37920]: pgmap v548: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 29252/177633 objects degraded (16.468%); 1/177633 objects misplaced (0.001%) 2024-03-26T03:11:31.512 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:31 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3372800713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:31.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:31.547 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:11:31.581 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:31.596 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:11:31.638 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:11:31.638 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:11:31.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:11:31.684 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:31.697 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:31.697 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:31 smithi192 ceph-mon[33110]: pgmap v548: 168 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s; 29252/177633 objects degraded (16.468%); 1/177633 objects misplaced (0.001%) 2024-03-26T03:11:31.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:31 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3372800713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:31.728 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:31.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:31.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:31.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:31.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:31.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:32.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:32.037 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:11:32.037 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:11:32.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:32.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:32.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:32.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:32.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:32.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:32.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:32.567 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:32.669 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:32.706 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2938553396' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.17", "id": [4, 3, 1, 2, 0, 5]}]: dispatch 2024-03-26T03:11:32.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:32.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2938553396' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.17", "id": [4, 3, 1, 2, 0, 5]}]: dispatch 2024-03-26T03:11:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2938553396' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.17", "id": [4, 3, 1, 2, 0, 5]}]: dispatch 2024-03-26T03:11:32.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:32.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:32.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:33.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:33.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:33.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:33.227 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:33.264 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:33.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:33.365 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:11:33.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:11:33.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:11:33.442 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:33.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:33.499 INFO:teuthology.orchestra.run.smithi066.stderr:set 4.17 pg_upmap_items mapping to [4->3,1->2,0->5] 2024-03-26T03:11:33.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:33.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:33.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-mon[29666]: pgmap v549: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 28892/173148 objects degraded (16.686%); 1/173148 objects misplaced (0.001%) 2024-03-26T03:11:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2938553396' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.17", "id": [4, 3, 1, 2, 0, 5]}]': finished 2024-03-26T03:11:33.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-mon[29666]: osdmap e270: 8 total, 7 up, 8 in 2024-03-26T03:11:33.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-mon[37920]: pgmap v549: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 28892/173148 objects degraded (16.686%); 1/173148 objects misplaced (0.001%) 2024-03-26T03:11:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2938553396' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.17", "id": [4, 3, 1, 2, 0, 5]}]': finished 2024-03-26T03:11:33.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-mon[37920]: osdmap e270: 8 total, 7 up, 8 in 2024-03-26T03:11:33.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:33.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:33 smithi192 ceph-mon[33110]: pgmap v549: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 28892/173148 objects degraded (16.686%); 1/173148 objects misplaced (0.001%) 2024-03-26T03:11:33.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:33 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2938553396' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.17", "id": [4, 3, 1, 2, 0, 5]}]': finished 2024-03-26T03:11:33.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:33 smithi192 ceph-mon[33110]: osdmap e270: 8 total, 7 up, 8 in 2024-03-26T03:11:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:33.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:33.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:33.840 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:33.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:33.878 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:33.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:33.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:34.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:11:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:11:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:11:34.119 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:34.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:34.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:34.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:34.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:34.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:34.532 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:34.561 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:11:34.568 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:34.671 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:34.719 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 28892/173148 objects degraded (16.686%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:34.820 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:34.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 28892/173148 objects degraded (16.686%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 28892/173148 objects degraded (16.686%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:34.861 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:34.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:34.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:34.974 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:35.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:35.103 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:35.140 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:35.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:35.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:35.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:35.368 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:11:35.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:35.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:35.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:35.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:35.646 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:35 smithi192 ceph-mon[33110]: pgmap v551: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 754 op/s; 28446/171810 objects degraded (16.557%); 1/171810 objects misplaced (0.001%) 2024-03-26T03:11:35.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:35.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:35.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:35.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:35.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:35 smithi066 ceph-mon[29666]: pgmap v551: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 754 op/s; 28446/171810 objects degraded (16.557%); 1/171810 objects misplaced (0.001%) 2024-03-26T03:11:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:35 smithi066 ceph-mon[37920]: pgmap v551: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 754 op/s; 28446/171810 objects degraded (16.557%); 1/171810 objects misplaced (0.001%) 2024-03-26T03:11:35.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:35.986 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:35.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:36.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:36.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:36.178 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:11:36.178 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:11:36.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:36.261 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:36.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:36.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:36.478 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:36.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:36.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:36.638 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:36.674 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:36.776 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:36.817 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:36.919 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:36.960 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:36.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:36.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:11:37.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:37.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:37.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:37.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:37.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:37.377 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:37.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:37.492 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:11:37.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:37.560 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:37.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:37.690 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:37 smithi066 ceph-mon[29666]: pgmap v552: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 25302/151572 objects degraded (16.693%); 1/151572 objects misplaced (0.001%) 2024-03-26T03:11:37.690 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/854401118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:11:37.690 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:37 smithi066 ceph-mon[29666]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:11:37.690 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:37 smithi066 ceph-mon[37920]: pgmap v552: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 25302/151572 objects degraded (16.693%); 1/151572 objects misplaced (0.001%) 2024-03-26T03:11:37.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/854401118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:11:37.691 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:37 smithi066 ceph-mon[37920]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:11:37.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:37 smithi192 ceph-mon[33110]: pgmap v552: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 25302/151572 objects degraded (16.693%); 1/151572 objects misplaced (0.001%) 2024-03-26T03:11:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/854401118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:11:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:37 smithi192 ceph-mon[33110]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:11:37.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:37.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:37.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:37.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:37.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:38.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:38.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:38.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:38.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:38.354 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:38.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:38.492 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:38.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:38.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:38.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:38.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:38.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:38.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:38 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:38.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:38 smithi066 ceph-mon[29666]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:11:38.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:38 smithi066 ceph-mon[29666]: osdmap e271: 8 total, 7 up, 8 in 2024-03-26T03:11:38.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:38 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:38.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:38 smithi066 ceph-mon[37920]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:11:38.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:38 smithi066 ceph-mon[37920]: osdmap e271: 8 total, 7 up, 8 in 2024-03-26T03:11:38.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:38.973 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:38.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:38.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:38 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:38.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:38 smithi192 ceph-mon[33110]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:11:38.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:38 smithi192 ceph-mon[33110]: osdmap e271: 8 total, 7 up, 8 in 2024-03-26T03:11:39.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:39.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:39.219 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:39.281 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:39.383 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:39.425 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:39.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:11:39.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:11:39.505 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-26T03:11:39.505 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:11:39.505 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:11:39.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-26T03:11:39.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:39.569 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:39.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:39.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: pgmap v554: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 25302/151572 objects degraded (16.693%); 1/151572 objects misplaced (0.001%) 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: 9.5 scrub starts 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 25302/151572 objects degraded (16.693%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: osdmap e272: 8 total, 7 up, 8 in 2024-03-26T03:11:39.754 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: pgmap v554: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 25302/151572 objects degraded (16.693%); 1/151572 objects misplaced (0.001%) 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: 9.5 scrub starts 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 25302/151572 objects degraded (16.693%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: osdmap e272: 8 total, 7 up, 8 in 2024-03-26T03:11:39.755 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:39.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:39.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:39.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:39.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:11:39.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: pgmap v554: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 25302/151572 objects degraded (16.693%); 1/151572 objects misplaced (0.001%) 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: 9.5 scrub starts 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 25302/151572 objects degraded (16.693%), 30 pgs degraded, 65 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: osdmap e272: 8 total, 7 up, 8 in 2024-03-26T03:11:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:39.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:11:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:11:39.989 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:39.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:11:40.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:11:40.017 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:11:40.091 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.123 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:40.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:40.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:40.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:40.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:40.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:40.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:40.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:40 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4258104103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:40.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:40.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:40 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:11:40.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:40 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:11:40.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:40 smithi192 ceph-mon[33110]: 9.5 scrub ok 2024-03-26T03:11:40.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:40 smithi192 ceph-mon[33110]: pgmap v556: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.60k op/s; 22155/135684 objects degraded (16.328%); 1/135684 objects misplaced (0.001%) 2024-03-26T03:11:40.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:40.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:40.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:40.797 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:40.834 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4258104103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:11:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:11:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[29666]: 9.5 scrub ok 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[29666]: pgmap v556: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.60k op/s; 22155/135684 objects degraded (16.328%); 1/135684 objects misplaced (0.001%) 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4258104103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[37920]: 9.5 scrub ok 2024-03-26T03:11:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:40 smithi066 ceph-mon[37920]: pgmap v556: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.60k op/s; 22155/135684 objects degraded (16.328%); 1/135684 objects misplaced (0.001%) 2024-03-26T03:11:40.936 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:40.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:11:40.969 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:40.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:41.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:41.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:41.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:41.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:41.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:41.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:41.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:41.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:41.611 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:41.612 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:41.612 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:11:41.612 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:11:41.612 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:11:41.612 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 9.6 scrub starts 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:11:41.613 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:41 smithi192 ceph-mon[33110]: 9.6 scrub ok 2024-03-26T03:11:41.646 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:41.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:41.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:41.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:11:41.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:11:41.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 9.6 scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[37920]: 9.6 scrub ok 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 9.6 scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:11:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:11:41.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:41 smithi066 ceph-mon[29666]: 9.6 scrub ok 2024-03-26T03:11:41.886 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:41.927 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:41.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:11:41.958 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:11:41.964 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:42.029 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:42.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:42.174 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:42.208 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:42.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:42.344 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:42.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:42.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:42.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:11:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:11:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 9.a scrub starts 2024-03-26T03:11:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[29666]: pgmap v557: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 21750/130629 objects degraded (16.650%); 1/130629 objects misplaced (0.001%) 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 9.a scrub starts 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:11:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:42 smithi066 ceph-mon[37920]: pgmap v557: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 21750/130629 objects degraded (16.650%); 1/130629 objects misplaced (0.001%) 2024-03-26T03:11:42.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:42.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 9.a scrub starts 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:11:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:42 smithi192 ceph-mon[33110]: pgmap v557: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 21750/130629 objects degraded (16.650%); 1/130629 objects misplaced (0.001%) 2024-03-26T03:11:42.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:42.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:42.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:11:42.874 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:42.911 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:42.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:42.978 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:43.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:43.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:43.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:43.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:43.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:43.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:43.351 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:43.453 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:43.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:43.589 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:43.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:11:43.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:11:43.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[37920]: 9.a scrub ok 2024-03-26T03:11:43.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:11:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:11:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:11:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:11:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[29666]: 9.a scrub ok 2024-03-26T03:11:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:11:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:11:43.628 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:43.730 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:43 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:11:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:43 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:11:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:43 smithi192 ceph-mon[33110]: 9.a scrub ok 2024-03-26T03:11:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:43 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:11:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:43 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:11:43.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:43.871 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:43.906 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:43.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:43.959 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:11:43.983 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:11:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:11:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:11:44.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:44.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:44.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:44.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:44.279 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:44.338 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:44.379 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:44.393 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:44.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:44.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:44.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:44.589 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.632 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:44.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:44.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:44 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:11:44.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:44 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:11:44.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 21750/130629 objects degraded (16.650%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:44.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:44 smithi192 ceph-mon[33110]: pgmap v558: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 918 op/s; 21385/129534 objects degraded (16.509%); 1/129534 objects misplaced (0.001%) 2024-03-26T03:11:44.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:44.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:44.857 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 21750/130629 objects degraded (16.650%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[29666]: pgmap v558: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 918 op/s; 21385/129534 objects degraded (16.509%); 1/129534 objects misplaced (0.001%) 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 21750/130629 objects degraded (16.650%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:44.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:44 smithi066 ceph-mon[37920]: pgmap v558: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 918 op/s; 21385/129534 objects degraded (16.509%); 1/129534 objects misplaced (0.001%) 2024-03-26T03:11:44.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:44.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:44.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:45.005 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:45.046 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:45.148 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:45.189 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:45.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:45.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:45.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:45.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:45.529 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:11:45.561 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:45.603 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[29666]: 9.0 scrub starts 2024-03-26T03:11:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:11:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:11:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3080873132' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:45.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[29666]: from='client.35417 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[37920]: 9.0 scrub starts 2024-03-26T03:11:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:11:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:11:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3080873132' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:45 smithi066 ceph-mon[37920]: from='client.35417 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:45.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:45.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:45.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:45 smithi192 ceph-mon[33110]: 9.0 scrub starts 2024-03-26T03:11:45.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:45 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:11:45.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:45 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:11:45.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3080873132' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:45.865 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:45 smithi192 ceph-mon[33110]: from='client.35417 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:11:45.877 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:45.915 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:45.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:46.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:46.055 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:46.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:46.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:46.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:46.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:46.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:46.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:46.501 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:46.501 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:11:46.501 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:46.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: 9.0 scrub ok 2024-03-26T03:11:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:11:46.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: from='client.35417 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: osdmap e273: 8 total, 7 up, 8 in 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[29666]: pgmap v560: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 18548/111603 objects degraded (16.620%); 1/111603 objects misplaced (0.001%) 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: 9.0 scrub ok 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: from='client.35417 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: osdmap e273: 8 total, 7 up, 8 in 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:11:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:46 smithi066 ceph-mon[37920]: pgmap v560: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 18548/111603 objects degraded (16.620%); 1/111603 objects misplaced (0.001%) 2024-03-26T03:11:46.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:46.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: 9.0 scrub ok 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: from='client.35417 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: osdmap e273: 8 total, 7 up, 8 in 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:11:46.755 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:46 smithi192 ceph-mon[33110]: pgmap v560: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.67k op/s; 18548/111603 objects degraded (16.620%); 1/111603 objects misplaced (0.001%) 2024-03-26T03:11:46.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:46.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:46.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:46.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75536 7.0 7:000359a7:::benchmark_data_smithi066_76917_object75535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 596.42699902499999, 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.004732272, 2024-03-26T03:11:46.955 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75536 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323405+0000", 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323409+0000", 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3119999999999998e-06 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323453+0000", 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4338999999999997e-05 2024-03-26T03:11:46.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323474+0000", 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1109000000000001e-05 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.323488+0000", 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3397000000000001e-05 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.363800+0000", 2024-03-26T03:11:46.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040311903000000003 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.363816+0000", 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6526000000000001e-05 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313106+0000", 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94928997599999998 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313268+0000", 2024-03-26T03:11:46.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016208999999999999 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.313443+0000", 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174558 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.316322+0000", 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028792119999999999 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.317555+0000", 2024-03-26T03:11:46.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012334970000000001 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328022+0000", 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010466392999999999 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328092+0000", 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9900000000000005e-05 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.328138+0000", 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6060000000000003e-05 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.961 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:75538 7.0 7:0739b178:::benchmark_data_smithi066_76917_object75537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 596.20292001500002, 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78758199900000003, 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:46.962 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 75538 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547484+0000", 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547488+0000", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0359999999999998e-06 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547625+0000", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013731600000000001 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547645+0000", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9944000000000001e-05 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547656+0000", 2024-03-26T03:11:46.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1610000000000001e-05 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547729+0000", 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2249999999999994e-05 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:50.547738+0000", 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5040000000000008e-06 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315069+0000", 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.76733058799999998 2024-03-26T03:11:46.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315202+0000", 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132769 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.315298+0000", 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6464000000000003e-05 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.316448+0000", 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011497969999999999 2024-03-26T03:11:46.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.319454+0000", 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003006075 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.334937+0000", 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015482902 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.335019+0000", 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1930999999999999e-05 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:01:51.335066+0000", 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7812999999999998e-05 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:111974 7.0 7:06bf99cb:::benchmark_data_smithi066_76917_object11791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 535.77028002999998, 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14525418500000001, 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 111974 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980124+0000", 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980127+0000", 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0849999999999999e-06 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980134+0000", 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1810000000000004e-06 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980149+0000", 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5702999999999999e-05 2024-03-26T03:11:46.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980159+0000", 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7969999999999995e-06 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980225+0000", 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5648000000000006e-05 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980350+0000", 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125227 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.980649+0000", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029928300000000001 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.982219+0000", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001569916 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:50.983019+0000", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00079931200000000005 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125310+0000", 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14229117699999999 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125347+0000", 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7212000000000002e-05 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.125379+0000", 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1643999999999997e-05 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112720 7.0 7:054d20dd:::benchmark_data_smithi066_76917_object12537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 534.234799716, 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13726976599999999, 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112720 2024-03-26T03:11:46.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515605+0000", 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515607+0000", 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1119999999999999e-06 2024-03-26T03:11:46.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515613+0000", 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1709999999999999e-06 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515629+0000", 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5914e-05 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515638+0000", 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6910000000000003e-06 2024-03-26T03:11:46.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515696+0000", 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7989999999999999e-05 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.515813+0000", 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116997 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.516136+0000", 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032339800000000002 2024-03-26T03:11:46.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.518072+0000", 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001936173 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.521492+0000", 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034199230000000001 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652808+0000", 2024-03-26T03:11:46.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.131315819 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652843+0000", 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4805000000000003e-05 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.652875+0000", 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1773000000000002e-05 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:142785 7.0 7:05427235:::benchmark_data_smithi066_76917_object42602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 497.30415420000003, 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.121159576, 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 142785 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446250+0000", 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446253+0000", 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3099999999999999e-06 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446259+0000", 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2999999999999998e-06 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446275+0000", 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6047999999999999e-05 2024-03-26T03:11:46.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446284+0000", 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446354+0000", 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0160999999999995e-05 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446480+0000", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000126036 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.446868+0000", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038751299999999997 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.449292+0000", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024239219999999998 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.452247+0000", 2024-03-26T03:11:46.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029555039999999999 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567305+0000", 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.115057495 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567363+0000", 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8637000000000001e-05 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:29.567410+0000", 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6641000000000003e-05 2024-03-26T03:11:46.985 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:148920 8.0 8:0ef07389:::benchmark_data_smithi066_100612_object21529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:07:00.249718+0000", 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.50068674300002, 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11961746099999999, 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:11:46.986 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148920 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249718+0000", 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249718+0000", 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249719+0000", 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.531e-06 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249727+0000", 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9480000000000008e-06 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249737+0000", 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0019e-05 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.249755+0000", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7821000000000001e-05 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.305827+0000", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056072151000000001 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.305959+0000", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013224800000000001 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.306094+0000", 2024-03-26T03:11:46.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013411900000000001 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369101+0000", 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.063007103999999994 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369151+0000", 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0281999999999997e-05 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369242+0000", 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1469000000000001e-05 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369335+0000", 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2769000000000006e-05 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24961.0:148929 8.0 8:077252a2:::benchmark_data_smithi066_100612_object21538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:07:00.256786+0000", 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.49361848199999, 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.113071699, 2024-03-26T03:11:46.991 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24961", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148929 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256786+0000", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256786+0000", 2024-03-26T03:11:46.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256788+0000", 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.256795+0000", 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4099999999999996e-06 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.300719+0000", 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.043924419999999999 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.300732+0000", 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3134e-05 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.308563+0000", 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078310540000000005 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.308677+0000", 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011394499999999999 2024-03-26T03:11:46.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.308790+0000", 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112948 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369107+0000", 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.060316579000000002 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369384+0000", 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000277681 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369648+0000", 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026355600000000002 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:07:00.369858+0000", 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000209903 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:105566 9.a 9:5e717e08:::benchmark_data_smithi066_125102_object105565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:54.297971+0000", 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 112.452433449, 2024-03-26T03:11:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.115932777, 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 105566 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.297971+0000", 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.297971+0000", 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.297973+0000", 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3120000000000001e-06 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.298011+0000", 2024-03-26T03:11:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7886999999999998e-05 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.298023+0000", 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1579e-05 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.298030+0000", 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1369999999999996e-06 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.298126+0000", 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5877000000000004e-05 2024-03-26T03:11:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.298280+0000", 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000154487 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.298428+0000", 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014799799999999999 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.304953+0000", 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0065245099999999999 2024-03-26T03:11:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.307448+0000", 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002495513 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.413801+0000", 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.106353012 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.413858+0000", 2024-03-26T03:11:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6613000000000002e-05 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.413904+0000", 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5852000000000001e-05 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:105571 9.a 9:57d9dd21:::benchmark_data_smithi066_125102_object105570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-03-26T03:11:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:09:54.300770+0000", 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 112.449634313, 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.113470478, 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 105571 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.300770+0000", 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.300770+0000", 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.300772+0000", 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.300804+0000", 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2357000000000002e-05 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.300819+0000", 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4602e-05 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.300827+0000", 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0800000000000006e-06 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.301151+0000", 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032421099999999998 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.301267+0000", 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115313 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.301399+0000", 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132774 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.306311+0000", 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049119430000000002 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.307550+0000", 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012385009999999999 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.414063+0000", 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10651351000000001 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.414177+0000", 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113547 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:09:54.414241+0000", 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3754e-05 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.0.0:156959 9.5 e268/225 9:a7b2b3d9:::benchmark_data_smithi066_125102_object80395:head v 268'16550, mlcod=268'16539)", 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:21.785561+0000", 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 24.964843888000001, 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.071671765999999998, 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.785561+0000", 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.785561+0000", 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.785562+0000", 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.404e-06 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.785570+0000", 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1890000000000007e-06 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.785580+0000", 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9899999999999992e-06 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.785587+0000", 2024-03-26T03:11:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1230000000000002e-06 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.814369+0000", 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028781770000000002 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.814392+0000", 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2529999999999999e-05 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.857124+0000", 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.042732602000000001 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:21.857232+0000", 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108158 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:229477 9.5 e268/225 9:a4f329a4:::benchmark_data_smithi066_125102_object83176:head v 268'17258, mlcod=268'17235)", 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:24.737431+0000", 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.012973500000001, 2024-03-26T03:11:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.073172965000000006, 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.737431+0000", 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.737431+0000", 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.737431+0000", 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.737438+0000", 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7270000000000001e-06 2024-03-26T03:11:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.737447+0000", 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5690000000000005e-06 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.737452+0000", 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6200000000000004e-06 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.773193+0000", 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.035740612999999997 2024-03-26T03:11:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.773213+0000", 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0220999999999998e-05 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.810518+0000", 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037305150000000002 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:24.810604+0000", 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5643999999999995e-05 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.0.0:158413 9.5 e268/225 9:aa0207fd:::benchmark_data_smithi066_125102_object135418:head v 268'17318, mlcod=268'17292)", 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.022245+0000", 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.728159861999998, 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.074033189999999999, 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.022245+0000", 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.022245+0000", 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.022245+0000", 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3799999999999996e-07 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.022251+0000", 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2259999999999996e-06 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.022258+0000", 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2389999999999996e-06 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.022264+0000", 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1179999999999997e-06 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.077644+0000", 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055379667 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.077665+0000", 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1392000000000001e-05 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.096086+0000", 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018420914 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.096278+0000", 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019179599999999999 2024-03-26T03:11:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(osd.0.0:158416 9.5 e268/225 9:aa027374:::benchmark_data_smithi066_125102_object111633:head v 268'17319, mlcod=268'17292)", 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.025451+0000", 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.724953451000001, 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.078796027000000005, 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.025451+0000", 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.025451+0000", 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.025452+0000", 2024-03-26T03:11:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7999999999999996e-07 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.025456+0000", 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5349999999999998e-06 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.025461+0000", 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9860000000000002e-06 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.025465+0000", 2024-03-26T03:11:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3919999999999996e-06 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.080927+0000", 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055461057000000001 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.080946+0000", 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9935000000000001e-05 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.104036+0000", 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023089208999999999 2024-03-26T03:11:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.104247+0000", 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021143299999999999 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:229713 9.5 e268/225 9:ab397f8a:::benchmark_data_smithi066_125102_object83412:head v 268'17320, mlcod=268'17297)", 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.029786+0000", 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.720618084000002, 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.074653871999999996, 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.029786+0000", 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.029786+0000", 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.029787+0000", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6899999999999997e-07 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.029793+0000", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.195e-06 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.029799+0000", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.711e-06 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.029804+0000", 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0660000000000003e-06 2024-03-26T03:11:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.089360+0000", 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.059556115999999999 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.089381+0000", 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0465000000000001e-05 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.104251+0000", 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014869950999999999 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.104440+0000", 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018969899999999999 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:229724 9.5 e268/225 9:a439040b:::benchmark_data_smithi066_125102_object83423:head v 268'17321, mlcod=268'17297)", 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.031829+0000", 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.718575815000001, 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.077488614999999997, 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.031829+0000", 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.031829+0000", 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.031830+0000", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3829999999999999e-06 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.031838+0000", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9519999999999994e-06 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.031854+0000", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6099999999999998e-05 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.031863+0000", 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0469999999999999e-06 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.096447+0000", 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.064583743999999998 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.096518+0000", 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1573000000000001e-05 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.109130+0000", 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012611934 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.109317+0000", 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000186882 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:229747 9.5 e268/225 9:a90f5f15:::benchmark_data_smithi066_125102_object83446:head v 268'17322, mlcod=268'17297)", 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.035121+0000", 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.715283373999998, 2024-03-26T03:11:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.079831719999999995, 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035121+0000", 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035121+0000", 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035122+0000", 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035129+0000", 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9859999999999997e-06 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035136+0000", 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2760000000000003e-06 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035142+0000", 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7830000000000004e-06 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.099934+0000", 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.064792335000000006 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.100010+0000", 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5693999999999997e-05 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.114776+0000", 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014765551999999999 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.114953+0000", 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017735799999999999 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:229746 9.d e268/225 9:bcf1ce10:::benchmark_data_smithi066_125102_object83445:head v 268'14531, mlcod=268'14527)", 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.035166+0000", 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.715238174, 2024-03-26T03:11:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.075880400000000001, 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035166+0000", 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035166+0000", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035167+0000", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4300000000000003e-07 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035173+0000", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0759999999999999e-06 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035179+0000", 2024-03-26T03:11:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.035184+0000", 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.135e-06 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.100588+0000", 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.065403647999999995 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.100621+0000", 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3256999999999997e-05 2024-03-26T03:11:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.110636+0000", 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010014924999999999 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.111047+0000", 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041041100000000001 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:229755 9.d e268/225 9:b4720309:::benchmark_data_smithi066_125102_object83454:head v 268'14532, mlcod=268'14527)", 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:25.037921+0000", 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.712483401, 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.083666589, 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.037921+0000", 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.037921+0000", 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.037923+0000", 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1179999999999999e-06 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.037932+0000", 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8289999999999997e-06 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.037943+0000", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1153e-05 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.037951+0000", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8820000000000001e-06 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.102858+0000", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.064906438999999996 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.102881+0000", 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3300000000000001e-05 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.121440+0000", 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018558766000000001 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:25.121588+0000", 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000148102 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:249299 9.f 9:f16601c6:::benchmark_data_smithi066_125102_object102998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:39.852886+0000", 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.8975184199999999, 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.087750015000000001, 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 249299 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.852886+0000", 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.852886+0000", 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.852888+0000", 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.001e-06 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.852894+0000", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.088e-06 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.852909+0000", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5125e-05 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.852917+0000", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9049999999999997e-06 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.853339+0000", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000421297 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.856527+0000", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003188892 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.856645+0000", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117428 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.863710+0000", 2024-03-26T03:11:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070649320000000003 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.940516+0000", 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.076806026999999999 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.940569+0000", 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2902000000000003e-05 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:39.940636+0000", 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7417999999999997e-05 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:251970 9.a 9:5fa781d7:::benchmark_data_smithi066_125102_object105669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:41.941668+0000", 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.8087361199999998, 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.080503818000000005, 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 251970 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:41.941668+0000", 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:41.941668+0000", 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:41.941670+0000", 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.42e-06 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:41.941675+0000", 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9989999999999999e-06 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:41.941684+0000", 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4700000000000008e-06 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:41.941692+0000", 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5800000000000003e-06 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.013723+0000", 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.072031586999999994 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.013863+0000", 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013908099999999999 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.014002+0000", 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013929900000000001 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.021977+0000", 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0079748340000000001 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.021991+0000", 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4687e-05 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.022027+0000", 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5531000000000003e-05 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.022114+0000", 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7510999999999996e-05 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:42.022172+0000", 2024-03-26T03:11:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7819000000000003e-05 2024-03-26T03:11:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:47.057 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:47.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:47.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:47.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:47.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:47.294 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:11:47.323 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:11:47.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:47.356 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:11:47.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:47.396 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:11:47.432 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:11:47.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:47.472 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:11:47.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:11:47.510 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:11:47.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:47.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: 4.12 deep-scrub starts 2024-03-26T03:11:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: 4.12 deep-scrub ok 2024-03-26T03:11:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:11:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:11:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: 6.1a deep-scrub starts 2024-03-26T03:11:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: 6.1a deep-scrub ok 2024-03-26T03:11:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:47 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/680222302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:47.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:47.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: 4.12 deep-scrub starts 2024-03-26T03:11:47.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: 4.12 deep-scrub ok 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: 6.1a deep-scrub starts 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: 6.1a deep-scrub ok 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/680222302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: 4.12 deep-scrub starts 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: 4.12 deep-scrub ok 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:11:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: 6.1a deep-scrub starts 2024-03-26T03:11:47.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: 6.1a deep-scrub ok 2024-03-26T03:11:47.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:47 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/680222302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:47.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:47.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:47.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:47.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:11:48.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:48.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:48.188 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:48.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:48.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:48.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:48.478 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:48.503 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:48.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:48.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:48.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:48 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:11:48.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:48 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:11:48.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:48 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:11:48.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:48 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:11:48.641 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:48 smithi192 ceph-mon[33110]: pgmap v561: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 18548/111603 objects degraded (16.620%); 1/111603 objects misplaced (0.001%) 2024-03-26T03:11:48.649 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:11:48.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:48.756 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:48.801 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:48.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:11:48.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:11:48.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:11:48.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:11:48.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[29666]: pgmap v561: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 18548/111603 objects degraded (16.620%); 1/111603 objects misplaced (0.001%) 2024-03-26T03:11:48.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:11:48.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:11:48.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:11:48.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:11:48.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:48 smithi066 ceph-mon[37920]: pgmap v561: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.45k op/s; 18548/111603 objects degraded (16.620%); 1/111603 objects misplaced (0.001%) 2024-03-26T03:11:48.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:48.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:48.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:49.040 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:49.075 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:49.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:49.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:49.309 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:49.346 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:49.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:49.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:49.586 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:49.624 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 4.6 deep-scrub starts 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 4.6 deep-scrub ok 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 18548/111603 objects degraded (16.620%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:49.670 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-mon[33110]: osdmap e274: 8 total, 7 up, 8 in 2024-03-26T03:11:49.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:49.766 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:49.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:11:49.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 4.6 deep-scrub starts 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 4.6 deep-scrub ok 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 18548/111603 objects degraded (16.620%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[29666]: osdmap e274: 8 total, 7 up, 8 in 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:11:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 4.6 deep-scrub starts 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 4.6 deep-scrub ok 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 18548/111603 objects degraded (16.620%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:49 smithi066 ceph-mon[37920]: osdmap e274: 8 total, 7 up, 8 in 2024-03-26T03:11:49.868 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:49.905 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:49.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:49.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:11:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:11:50.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.050 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:50.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:11:50.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:11:50.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:50.183 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:50.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:50.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:50.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:50.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:50.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:50.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:11:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:11:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[29666]: 6.3 deep-scrub starts 2024-03-26T03:11:50.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[29666]: 6.3 deep-scrub ok 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[29666]: pgmap v563: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 15460/95766 objects degraded (16.144%); 1/95766 objects misplaced (0.001%) 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[37920]: 6.3 deep-scrub starts 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[37920]: 6.3 deep-scrub ok 2024-03-26T03:11:50.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:50 smithi066 ceph-mon[37920]: pgmap v563: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 15460/95766 objects degraded (16.144%); 1/95766 objects misplaced (0.001%) 2024-03-26T03:11:50.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:50.627 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:11:50.627 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:11:50.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:50.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:50.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:50.894 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:50.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:50.958 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:50 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:11:50.959 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:50 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:11:50.959 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:50 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:11:50.959 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:50 smithi192 ceph-mon[33110]: 6.3 deep-scrub starts 2024-03-26T03:11:50.959 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:50 smithi192 ceph-mon[33110]: 6.3 deep-scrub ok 2024-03-26T03:11:50.959 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:50 smithi192 ceph-mon[33110]: pgmap v563: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 15460/95766 objects degraded (16.144%); 1/95766 objects misplaced (0.001%) 2024-03-26T03:11:50.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:51.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:51.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:51.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:51.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:51.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:51.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:51.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:11:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:51.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:51.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:51.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:51.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:51.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:11:51.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:11:51.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[29666]: osdmap e275: 8 total, 7 up, 8 in 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[37920]: osdmap e275: 8 total, 7 up, 8 in 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:11:51.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:51 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:11:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:51 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:11:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:51 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:11:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:51 smithi192 ceph-mon[33110]: osdmap e275: 8 total, 7 up, 8 in 2024-03-26T03:11:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:51 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:11:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:51 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:11:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:51.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:51.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:51.869 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:51.901 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:51.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:52.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:52.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:52.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:11:52.140 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:52.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:52.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:52.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:52.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.489 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:52.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:52.495 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:52.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:52.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:52.731 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:52.770 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:52.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[37920]: pgmap v565: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 760 B/s rd, 0 B/s wr, 1.29k op/s; 15051/90273 objects degraded (16.673%); 1/90273 objects misplaced (0.001%) 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:11:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:52 smithi066 ceph-mon[29666]: pgmap v565: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 760 B/s rd, 0 B/s wr, 1.29k op/s; 15051/90273 objects degraded (16.673%); 1/90273 objects misplaced (0.001%) 2024-03-26T03:11:52.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:52.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:52.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:52 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:11:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:52 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:11:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:52 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:11:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:52 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:11:52.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:52 smithi192 ceph-mon[33110]: pgmap v565: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 760 B/s rd, 0 B/s wr, 1.29k op/s; 15051/90273 objects degraded (16.673%); 1/90273 objects misplaced (0.001%) 2024-03-26T03:11:53.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:53.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:53.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:53.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:53.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:53.403 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:11:53.404 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:11:53.404 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:11:53.404 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:11:53.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:53.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:53.567 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:53.584 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:11:53.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:53.710 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:53.747 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:53.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4282349549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:11:53.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:11:53.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:11:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4282349549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:11:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:11:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:11:53.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:53.888 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:53.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4282349549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:11:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:53 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:11:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:53 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:11:53.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:54.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:54.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:11:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:11:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:11:54.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:54.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:54.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:54.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:54.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:54.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:54.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:54.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:54.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:54.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:11:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4282349549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:11:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: osdmap e276: 8 total, 7 up, 8 in 2024-03-26T03:11:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 15051/90273 objects degraded (16.673%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: pgmap v567: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 14611/88953 objects degraded (16.426%); 1/88953 objects misplaced (0.001%) 2024-03-26T03:11:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:54 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:54.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:54.848 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:54.848 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4282349549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: osdmap e276: 8 total, 7 up, 8 in 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 15051/90273 objects degraded (16.673%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: pgmap v567: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 14611/88953 objects degraded (16.426%); 1/88953 objects misplaced (0.001%) 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4282349549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: osdmap e276: 8 total, 7 up, 8 in 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 15051/90273 objects degraded (16.673%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: pgmap v567: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 1.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.26k op/s; 14611/88953 objects degraded (16.426%); 1/88953 objects misplaced (0.001%) 2024-03-26T03:11:54.849 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:54 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:11:54.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:54.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:54.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:55.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:55.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:55.118 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:55.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:55.225 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:11:55.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:55.262 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:11:55.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:55.302 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:11:55.342 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:11:55.377 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:11:55.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:55.415 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:11:55.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:55.453 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:11:55.454 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:11:55.454 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:11:55.454 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:11:55.454 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:11:55.454 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:11:55.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:11:55.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:11:55.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:11:55.538 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:55.538 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:11:55.538 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:11:55.538 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:11:55.538 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:55.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:55.575 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:55.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:55.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:55.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:55.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:55.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:55.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:55.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:56.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:11:56.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2604252146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:56.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2604252146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:56.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:11:56.182 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2604252146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:11:56.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:56.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:56.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:56.395 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:56.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:56.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:11:56.530 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:56.618 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:11:56.632 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:56.699 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:56.800 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:56.834 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:56.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:56.963 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:11:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:56.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:56.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:57.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:11:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:57.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:11:57.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:57.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:57.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:57 smithi066 ceph-mon[29666]: pgmap v568: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.98k op/s; 11537/69030 objects degraded (16.713%); 1/69030 objects misplaced (0.001%) 2024-03-26T03:11:57.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:57 smithi066 ceph-mon[37920]: pgmap v568: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.98k op/s; 11537/69030 objects degraded (16.713%); 1/69030 objects misplaced (0.001%) 2024-03-26T03:11:57.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:57.387 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:57 smithi192 ceph-mon[33110]: pgmap v568: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.98k op/s; 11537/69030 objects degraded (16.713%); 1/69030 objects misplaced (0.001%) 2024-03-26T03:11:57.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:57.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:57.550 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:57.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:57.689 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:57.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:57.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:57.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:11:57.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:11:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:57.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:57.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:58.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:58.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:58.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:58.234 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:58.285 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:58.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:58.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:58.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:58.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:58.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:58.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:58.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:58.750 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:58.786 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:58.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:58.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:11:58.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:58.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:11:58.939 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:11:59.007 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112085 7.12 7:493de57c:::benchmark_data_smithi066_76917_object11902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 547.72478192699998, 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.024736326, 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112085 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279168+0000", 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279170+0000", 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.593e-06 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279174+0000", 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8470000000000003e-06 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279184+0000", 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6080000000000002e-06 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279193+0000", 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7409999999999998e-06 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279262+0000", 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9488999999999998e-05 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.279273+0000", 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.08e-05 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.291772+0000", 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0124987969999999 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.291923+0000", 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015105899999999999 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.292236+0000", 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031284800000000001 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.296791+0000", 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045547749999999996 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.297868+0000", 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010770319999999999 2024-03-26T03:11:59.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303845+0000", 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059774659999999999 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303877+0000", 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1480999999999999e-05 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303904+0000", 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.779e-05 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112092 7.12 7:4c17d0e1:::benchmark_data_smithi066_76917_object11909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 547.71745313899999, 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0192903719999999, 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112092 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286497+0000", 2024-03-26T03:11:59.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286499+0000", 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.717e-06 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286504+0000", 2024-03-26T03:11:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286516+0000", 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1919999999999999e-05 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286525+0000", 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0359999999999995e-06 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286577+0000", 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2420000000000003e-05 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.286590+0000", 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2469999999999999e-05 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.293804+0000", 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0072144949999999 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.293919+0000", 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115445 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.294102+0000", 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000182339 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298885+0000", 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047836770000000001 2024-03-26T03:11:59.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.303949+0000", 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005063115 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305695+0000", 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017461379999999999 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305746+0000", 2024-03-26T03:11:59.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0810000000000003e-05 2024-03-26T03:11:59.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305787+0000", 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-05 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112198 7.12 7:4c79c881:::benchmark_data_smithi066_76917_object12015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 547.506531774, 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.81316842600000006, 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.023 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112198 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497418+0000", 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497420+0000", 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5859999999999999e-06 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497425+0000", 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2039999999999996e-06 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497438+0000", 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2672e-05 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497445+0000", 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8129999999999996e-06 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497501+0000", 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5503e-05 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.497509+0000", 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6899999999999992e-06 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.294932+0000", 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79742365100000001 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295019+0000", 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7113000000000004e-05 2024-03-26T03:11:59.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295174+0000", 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000155014 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298890+0000", 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037151950000000001 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305657+0000", 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0067669499999999999 2024-03-26T03:11:59.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310476+0000", 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048193130000000004 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310539+0000", 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3424999999999997e-05 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310587+0000", 2024-03-26T03:11:59.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7296999999999998e-05 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:112201 7.12 7:4b665aee:::benchmark_data_smithi066_76917_object12018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 547.50003538800001, 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80677220100000002, 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 112201 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503915+0000", 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503916+0000", 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0890000000000001e-06 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503920+0000", 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.89e-06 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503929+0000", 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7229999999999999e-06 2024-03-26T03:11:59.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.503937+0000", 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0080000000000002e-06 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.504005+0000", 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7787000000000007e-05 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:51.504018+0000", 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.252e-05 2024-03-26T03:11:59.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295896+0000", 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79187867700000003 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.295978+0000", 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.174e-05 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:11:59.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.296092+0000", 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114016 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.298893+0000", 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028010299999999999 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.305824+0000", 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069314609999999999 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310625+0000", 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004800861 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310658+0000", 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2891000000000001e-05 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:02:52.310687+0000", 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8507999999999999e-05 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.036 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157251 7.19 7:9b7ef1c8:::benchmark_data_smithi066_76917_object57068:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 493.035282999, 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.025284095, 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.037 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157251 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968667+0000", 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968672+0000", 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6670000000000004e-06 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968678+0000", 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5309999999999998e-06 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968695+0000", 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7135000000000001e-05 2024-03-26T03:11:59.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.968705+0000", 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4280000000000002e-06 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.969069+0000", 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036450099999999999 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.969082+0000", 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2761e-05 2024-03-26T03:11:59.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.987851+0000", 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0187692269999999 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.987998+0000", 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000146957 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.988367+0000", 2024-03-26T03:11:59.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000368885 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992132+0000", 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037647359999999999 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993640+0000", 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001508604 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993833+0000", 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000192404 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993891+0000", 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8391999999999999e-05 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.993951+0000", 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9867000000000001e-05 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.043 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157261 7.19 7:989eae58:::benchmark_data_smithi066_76917_object57078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 493.02852969499997, 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.026782426, 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.044 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157261 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975420+0000", 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975421+0000", 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7999999999999993e-07 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975426+0000", 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4510000000000002e-06 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975436+0000", 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0815999999999999e-05 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975443+0000", 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975508+0000", 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4856000000000004e-05 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.975521+0000", 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2813000000000001e-05 2024-03-26T03:11:59.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990406+0000", 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.014885641 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990530+0000", 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012369900000000001 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.990814+0000", 2024-03-26T03:11:59.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000283503 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995533+0000", 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047195300000000004 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997610+0000", 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020769709999999999 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002088+0000", 2024-03-26T03:11:59.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004477895 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002142+0000", 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4333000000000003e-05 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002203+0000", 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0304999999999999e-05 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157262 7.19 7:9be70472:::benchmark_data_smithi066_76917_object57079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 493.02604214500002, 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244733150000001, 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157262 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.051 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977908+0000", 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977909+0000", 2024-03-26T03:11:59.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977915+0000", 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3040000000000004e-06 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977925+0000", 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0761000000000001e-05 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.977933+0000", 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2690000000000002e-06 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.978012+0000", 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9157000000000001e-05 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:45.978024+0000", 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2398999999999999e-05 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992170+0000", 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.014145802 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992257+0000", 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6532000000000004e-05 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.992467+0000", 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021020199999999999 2024-03-26T03:11:59.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995538+0000", 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003071514 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997648+0000", 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021097339999999998 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002278+0000", 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046300810000000003 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002337+0000", 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9058000000000003e-05 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002381+0000", 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3902000000000001e-05 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157326 7.19 7:9db2bad9:::benchmark_data_smithi066_76917_object57143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.96748822400002, 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96608487600000004, 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.058 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157326 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036462+0000", 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036464+0000", 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.486e-06 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036474+0000", 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244999999999999e-05 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036490+0000", 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5928000000000001e-05 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036499+0000", 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6670000000000002e-06 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036567+0000", 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8338999999999997e-05 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.036578+0000", 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0682999999999999e-05 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994011+0000", 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95743293299999999 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994083+0000", 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2106000000000004e-05 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.994259+0000", 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017634500000000001 2024-03-26T03:11:59.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997687+0000", 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034280309999999998 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002229+0000", 2024-03-26T03:11:59.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045413550000000004 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002484+0000", 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025479300000000001 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002515+0000", 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1798999999999998e-05 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002547+0000", 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1165999999999999e-05 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157394 7.19 7:9d416a04:::benchmark_data_smithi066_76917_object57211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.90942644, 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90827457, 2024-03-26T03:11:59.066 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157394 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094523+0000", 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094525+0000", 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1820000000000001e-06 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094531+0000", 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6420000000000001e-06 2024-03-26T03:11:59.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094543+0000", 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1473e-05 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094549+0000", 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3749999999999999e-06 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094591+0000", 2024-03-26T03:11:59.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2333999999999998e-05 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.094599+0000", 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1740000000000003e-06 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995590+0000", 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.900991404 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995703+0000", 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112657 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.995979+0000", 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027668000000000001 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002238+0000", 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006258581 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002425+0000", 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018703599999999999 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002629+0000", 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000204358 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002755+0000", 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012557900000000001 2024-03-26T03:11:59.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002798+0000", 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3095e-05 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157412 7.19 7:99cf3667:::benchmark_data_smithi066_76917_object57229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.89496737600001, 2024-03-26T03:11:59.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89402950199999998, 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157412 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108983+0000", 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108984+0000", 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.06e-06 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108988+0000", 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.108996+0000", 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999992e-06 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109003+0000", 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0099999999999998e-06 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109077+0000", 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4035000000000005e-05 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.109090+0000", 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2683e-05 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997735+0000", 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88864550600000003 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.997869+0000", 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013324800000000001 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.998178+0000", 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030980100000000001 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002441+0000", 2024-03-26T03:11:59.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042626239999999996 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002581+0000", 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014011500000000001 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002909+0000", 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032837200000000001 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002971+0000", 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1194999999999999e-05 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003012+0000", 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1405000000000001e-05 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157451 7.19 7:9ac14505:::benchmark_data_smithi066_76917_object57268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:11:59.080 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.86701483000002, 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87424788600000003, 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157451 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:11:59.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136935+0000", 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136936+0000", 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3880000000000001e-06 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136941+0000", 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9099999999999996e-06 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136956+0000", 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4181e-05 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.136971+0000", 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5316999999999999e-05 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.137210+0000", 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023869700000000001 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.137224+0000", 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4207e-05 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.999829+0000", 2024-03-26T03:11:59.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86260561300000005 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.999973+0000", 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014313800000000001 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.000235+0000", 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026196699999999999 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002445+0000", 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022107519999999999 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.002855+0000", 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040992999999999997 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011105+0000", 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082499259999999994 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011144+0000", 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8408000000000002e-05 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011183+0000", 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9452000000000001e-05 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.087 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157623 7.19 7:9ae45d3c:::benchmark_data_smithi066_76917_object57440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.70328213099998, 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.71074928199999998, 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157623 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300668+0000", 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300670+0000", 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300674+0000", 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4469999999999999e-06 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300678+0000", 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.163e-06 2024-03-26T03:11:59.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300717+0000", 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.300724+0000", 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0079999999999996e-06 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003059+0000", 2024-03-26T03:11:59.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70233499399999999 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003162+0000", 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103136 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.003429+0000", 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000266395 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011275+0000", 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078460179999999997 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011290+0000", 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5318000000000001e-05 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011330+0000", 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9536999999999998e-05 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011379+0000", 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9342000000000003e-05 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011417+0000", 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8158999999999998e-05 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157629 7.19 7:9d222119:::benchmark_data_smithi066_76917_object57446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.69725239399997, 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70924085000000003, 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157629 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306698+0000", 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306699+0000", 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.702e-06 2024-03-26T03:11:59.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306705+0000", 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1109999999999996e-06 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306718+0000", 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2378000000000001e-05 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306725+0000", 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2250000000000003e-06 2024-03-26T03:11:59.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306795+0000", 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0272999999999996e-05 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.306806+0000", 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0883000000000001e-05 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005375+0000", 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.69856929300000004 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005481+0000", 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105763 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.005761+0000", 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027942199999999998 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011295+0000", 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055347560000000001 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011461+0000", 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000166088 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015840+0000", 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043782969999999997 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015892+0000", 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2256999999999998e-05 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015938+0000", 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6402000000000001e-05 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157655 7.19 7:9b64c26d:::benchmark_data_smithi066_76917_object57472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.101 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.656400201, 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67836821199999997, 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157655 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347550+0000", 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347551+0000", 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.85e-07 2024-03-26T03:11:59.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347554+0000", 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1889999999999999e-06 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347564+0000", 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8030000000000008e-06 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347568+0000", 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9520000000000002e-06 2024-03-26T03:11:59.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347621+0000", 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2417000000000003e-05 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.347628+0000", 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5120000000000002e-06 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.008556+0000", 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66092736900000004 2024-03-26T03:11:59.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.008679+0000", 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012314300000000001 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.009015+0000", 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033638899999999998 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011425+0000", 2024-03-26T03:11:59.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00241004 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015997+0000", 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045720439999999999 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025826+0000", 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0098289510000000007 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025878+0000", 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1316000000000001e-05 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.025918+0000", 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0302000000000001e-05 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.14841.0:157664 7.19 7:9fd64715:::benchmark_data_smithi066_76917_object57481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:11:59.109 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 492.642843056, 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66498246000000005, 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.14841", 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3671590330", 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157664 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361107+0000", 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361108+0000", 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6899999999999997e-07 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361110+0000", 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9270000000000001e-06 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361117+0000", 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9659999999999999e-06 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361122+0000", 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2459999999999997e-06 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361169+0000", 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6907999999999999e-05 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:46.361176+0000", 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5129999999999999e-06 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011489+0000", 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65031320299999995 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011597+0000", 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107469 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.011895+0000", 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029828400000000001 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.015953+0000", 2024-03-26T03:11:59.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040577169999999997 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.016041+0000", 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8707000000000003e-05 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026023+0000", 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0099818719999999993 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026054+0000", 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0925e-05 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:03:47.026089+0000", 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5054e-05 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:270840 9.1 9:8e4a34ac:::benchmark_data_smithi066_125102_object124539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:55.573952+0000", 2024-03-26T03:11:59.116 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 3.429997583, 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031149926000000001, 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 270840 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.573952+0000", 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.573952+0000", 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.573955+0000", 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3089999999999998e-06 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.573966+0000", 2024-03-26T03:11:59.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0947e-05 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.573984+0000", 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8151999999999998e-05 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.573993+0000", 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4280000000000002e-06 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.580180+0000", 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0061869610000000004 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.580495+0000", 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031526500000000001 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.580886+0000", 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039020299999999999 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.587444+0000", 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0065581709999999998 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.602539+0000", 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015094725999999999 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.605010+0000", 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024713449999999998 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:55.605102+0000", 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2418999999999997e-05 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:271699 9.1 9:80663b2f:::benchmark_data_smithi066_125102_object125398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:56.221369+0000", 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.7825808520000002, 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.044949047999999998, 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.122 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 271699 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221369+0000", 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221369+0000", 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221371+0000", 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221380+0000", 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0559999999999994e-06 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221391+0000", 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0823e-05 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221398+0000", 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.469e-06 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221448+0000", 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9521000000000002e-05 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221553+0000", 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010537800000000001 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.221659+0000", 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010555799999999999 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.231458+0000", 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0097995779999999998 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.266126+0000", 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034667361000000001 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.266224+0000", 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7922000000000002e-05 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.266318+0000", 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4338000000000004e-05 2024-03-26T03:11:59.127 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:272155 9.1 9:8e536950:::benchmark_data_smithi066_125102_object125854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:56.579629+0000", 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.4243208259999998, 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.035828790999999999, 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:59.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 272155 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.579629+0000", 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.579629+0000", 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.579631+0000", 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8530000000000001e-06 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.579638+0000", 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8120000000000002e-06 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.579650+0000", 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2465e-05 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.579659+0000", 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5229999999999995e-06 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.592303+0000", 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012644659000000001 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.592621+0000", 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031750499999999999 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.593055+0000", 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00043436599999999999 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.597195+0000", 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041396710000000001 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.615244+0000", 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018049490000000001 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.615345+0000", 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010063 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:56.615458+0000", 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112817 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:273131 9.1 9:87d1c906:::benchmark_data_smithi066_125102_object126830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:57.413643+0000", 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.5903071870000001, 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034380030999999998, 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 273131 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413643+0000", 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413643+0000", 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413645+0000", 2024-03-26T03:11:59.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7480000000000002e-06 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413654+0000", 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8009999999999993e-06 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413674+0000", 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9400999999999999e-05 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413684+0000", 2024-03-26T03:11:59.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0199e-05 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.413877+0000", 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019264799999999999 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.414178+0000", 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030139000000000001 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.414564+0000", 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038594900000000001 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.426736+0000", 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012172145000000001 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.447580+0000", 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020844331000000001 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.447738+0000", 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015801300000000001 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.448023+0000", 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028440600000000001 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.139 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:273140 9.1 9:877e2ea6:::benchmark_data_smithi066_125102_object126839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:11:57.418004+0000", 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.585945986, 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033050799999999998, 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 273140 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418004+0000", 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418004+0000", 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418006+0000", 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5289999999999998e-06 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418016+0000", 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9820000000000003e-06 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418036+0000", 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9366999999999999e-05 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418046+0000", 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8549999999999997e-06 2024-03-26T03:11:59.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418240+0000", 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019396999999999999 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418371+0000", 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013099300000000001 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.418490+0000", 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119263 2024-03-26T03:11:59.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.426757+0000", 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082670480000000008 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.448169+0000", 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021411682000000001 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.448375+0000", 2024-03-26T03:11:59.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020666700000000001 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:11:57.451055+0000", 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002679444 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:11:59.145 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:11:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:59.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:59.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:59.248 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:11:59.304 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:59.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:11:59.350 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:59 smithi066 ceph-mon[29666]: pgmap v569: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 11537/69030 objects degraded (16.713%); 1/69030 objects misplaced (0.001%) 2024-03-26T03:11:59.350 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:11:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 11537/69030 objects degraded (16.713%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:59.350 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:59 smithi066 ceph-mon[37920]: pgmap v569: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 11537/69030 objects degraded (16.713%); 1/69030 objects misplaced (0.001%) 2024-03-26T03:11:59.350 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:11:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 11537/69030 objects degraded (16.713%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 276, 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:11:53.561980+0000", 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T03:06:19.254688+0000", 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5801984, 2024-03-26T03:11:59.352 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "noscrub", 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:11:59.353 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 9, 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:11:59.354 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 21, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 21, 2024-03-26T03:11:59.355 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 21, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 21, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 21, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "254", 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.356 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "252", 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.357 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.358 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.359 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.360 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2699999809265137, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2699999809265137, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.362 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:59.363 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.364 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.365 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.366 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.367 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:11:59.368 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:11:59.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:11:59.370 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:11:59.370 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:11:59.370 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:11:59.370 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.371 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.371 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.371 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.371 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.371 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.371 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.372 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:11:59.373 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.374 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.375 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.376 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.377 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.378 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.379 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.380 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.381 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2400000095367432, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2400000095367432, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.382 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.383 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.384 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.385 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.386 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.387 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:59.388 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:59.389 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:11:59.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.391 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.392 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.393 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:11:59.394 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 9, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_2", 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:08:41.897072+0000", 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:11:59.395 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 16, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 16, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 16, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 16, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 16, 2024-03-26T03:11:59.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "228", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:11:59.397 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:11:59.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:11:59.399 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:11:59.400 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.401 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.402 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.403 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:11:59.404 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.405 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.406 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:11:59.407 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:11:59.408 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.410 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:11:59.411 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:11:59.412 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 0, 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.413 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 198, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.414 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:11:59.415 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:11:59.416 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "exists" 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.417 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:11:59.420 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:11:59.421 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.422 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.423 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.424 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.425 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.426 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.427 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.428 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.429 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:11:59.430 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.431 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.433 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.434 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:11:59.437 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.437 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:11:59.437 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:29.133643+0000", 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.438 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:10:39.727465+0000", 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:50.281295+0000", 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:05:36.091229+0000", 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 197 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.440 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:52.802839+0000", 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:05.177898+0000", 2024-03-26T03:11:59.441 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:07.662058+0000", 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:11:59.442 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:09.431840+0000", 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [ 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.f", 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:11:59.443 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.f", 2024-03-26T03:11:59.444 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 4, 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 7 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 0, 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 1 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.445 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.446 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [ 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.c", 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:11:59.447 INFO:teuthology.orchestra.run.smithi066.stdout: 6, 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: 5 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.e", 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: 7, 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: 0 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.11", 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:59.448 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: 5, 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: 2 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.13", 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: 6 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:11:59.449 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:11:59.450 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:11:59.451 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:11:59.452 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:11:59.490 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:59.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:59.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:11:59.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:11:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:59.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:11:59.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:59 smithi192 ceph-mon[33110]: pgmap v569: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 11537/69030 objects degraded (16.713%); 1/69030 objects misplaced (0.001%) 2024-03-26T03:11:59.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:11:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 11537/69030 objects degraded (16.713%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:11:59.728 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:11:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:59.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:59.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:11:59.870 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:11:59.911 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:11:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:11:59.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:11:59.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:11:59.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:11:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:12:00.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:00.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:00.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:00.155 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.197 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:00.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:00.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:00.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:00.444 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:00.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:00.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:00.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4143358786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:12:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4143358786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:12:00.629 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:00.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4143358786' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:12:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:00.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:00.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:12:00.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:00.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:00.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:01.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:01.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:01.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:01.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:01.251 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:12:01.326 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:01.364 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:01.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:01.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:01.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:01 smithi066 ceph-mon[29666]: pgmap v570: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s; 8570/53700 objects degraded (15.959%); 1/53700 objects misplaced (0.002%) 2024-03-26T03:12:01.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:01.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:01 smithi066 ceph-mon[37920]: pgmap v570: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s; 8570/53700 objects degraded (15.959%); 1/53700 objects misplaced (0.002%) 2024-03-26T03:12:01.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:01 smithi192 ceph-mon[33110]: pgmap v570: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 1.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.45k op/s; 8570/53700 objects degraded (15.959%); 1/53700 objects misplaced (0.002%) 2024-03-26T03:12:01.747 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:01.784 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:01.826 INFO:tasks.thrashosds.thrasher:j is {'epoch': 276, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:11:53.561980+0000', 'last_up_change': '2024-03-26T03:06:19.254688+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '252', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-26T03:08:41.897072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 172, 'down_at': 198, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:29.133643+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:10:39.727465+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:50.281295+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:05:36.091229+0000', 'dead_epoch': 197}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:52.802839+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:05.177898+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:07.662058+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:09.431840+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [4, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.f', 'mappings': [{'from': 4, 'to': 7}, {'from': 0, 'to': 1}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [0, 6, 5]}, {'pgid': '1.e', 'osds': [4, 7, 0]}, {'pgid': '1.11', 'osds': [0, 5, 2]}, {'pgid': '1.13', 'osds': [1, 4, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:12:01.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:12:01.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:01.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:01.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:02.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:02.059 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:02.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:02.206 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:02.308 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:02.346 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:02.363 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:02.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:02.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:02.599 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:02 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:12:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:02 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:12:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:02 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:12:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:02 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:12:02.638 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:02 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:12:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:02 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:12:02.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:02.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:02.876 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:02.920 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:02.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:03.021 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:03.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:03.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:03.199 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:03.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:03.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:03.365 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:12:03.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:03.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:03.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[29666]: pgmap v571: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 995 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 8165/48423 objects degraded (16.862%); 1/48423 objects misplaced (0.002%) 2024-03-26T03:12:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[29666]: osdmap e277: 8 total, 7 up, 8 in 2024-03-26T03:12:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1235995650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[29666]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[37920]: pgmap v571: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 995 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 8165/48423 objects degraded (16.862%); 1/48423 objects misplaced (0.002%) 2024-03-26T03:12:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[37920]: osdmap e277: 8 total, 7 up, 8 in 2024-03-26T03:12:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1235995650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-mon[37920]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:03.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:03.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:03 smithi192 ceph-mon[33110]: pgmap v571: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 995 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 8165/48423 objects degraded (16.862%); 1/48423 objects misplaced (0.002%) 2024-03-26T03:12:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:03 smithi192 ceph-mon[33110]: osdmap e277: 8 total, 7 up, 8 in 2024-03-26T03:12:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1235995650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:03 smithi192 ceph-mon[33110]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:03.754 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:03.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:03.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:03.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:03.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:04.024 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:12:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:12:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:12:04.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:04.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:04.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:04.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:04.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:04.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:04.539 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.579 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:04.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:04.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:04.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[37920]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[37920]: osdmap e278: 8 total, 7 up, 8 in 2024-03-26T03:12:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 8165/48423 objects degraded (16.862%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[29666]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[29666]: osdmap e278: 8 total, 7 up, 8 in 2024-03-26T03:12:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 8165/48423 objects degraded (16.862%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:04.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:04.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:04 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:04 smithi192 ceph-mon[33110]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:04 smithi192 ceph-mon[33110]: osdmap e278: 8 total, 7 up, 8 in 2024-03-26T03:12:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 8165/48423 objects degraded (16.862%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:04.817 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.856 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:04.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:04.957 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:04.994 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:04.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:05.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:05.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:05.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:05.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:05.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:05.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:05.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:05.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:05.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:05 smithi066 ceph-mon[29666]: pgmap v574: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 967 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 914 op/s; 7718/47082 objects degraded (16.393%); 1/47082 objects misplaced (0.002%) 2024-03-26T03:12:05.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:05 smithi066 ceph-mon[37920]: pgmap v574: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 967 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 914 op/s; 7718/47082 objects degraded (16.393%); 1/47082 objects misplaced (0.002%) 2024-03-26T03:12:05.634 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:05 smithi192 ceph-mon[33110]: pgmap v574: 168 pgs: 1 active+clean+scrubbing, 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 95 active+clean; 967 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 914 op/s; 7718/47082 objects degraded (16.393%); 1/47082 objects misplaced (0.002%) 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:284534 9.d 9:b46450e1:::benchmark_data_smithi066_125102_object138233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.626097+0000", 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0072010149999999998, 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072331339999999996, 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:12:05.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 284534 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626097+0000", 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626097+0000", 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626099+0000", 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0090000000000002e-06 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626104+0000", 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5199999999999997e-06 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626116+0000", 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1929999999999999e-05 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626124+0000", 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2740000000000007e-06 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626839+0000", 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00071431800000000003 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626969+0000", 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000130316 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.627092+0000", 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012265400000000001 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.632553+0000", 2024-03-26T03:12:05.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054615569999999997 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.632603+0000", 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9872e-05 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:284532 9.c e278/225 9:3988d2f9:::benchmark_data_smithi066_125102_object138231:head v 278'17792, mlcod=278'17790)", 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.626275+0000", 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0070230860000000004, 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.007149463, 2024-03-26T03:12:05.641 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626275+0000", 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626275+0000", 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626275+0000", 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5700000000000001e-07 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626281+0000", 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6899999999999997e-06 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626290+0000", 2024-03-26T03:12:05.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5720000000000003e-06 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626304+0000", 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4479e-05 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626539+0000", 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023514400000000001 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626560+0000", 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0343000000000002e-05 2024-03-26T03:12:05.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15360.0:284537 9.1 e278/225 9:82bd26c6:::benchmark_data_smithi066_125102_object138236:head v 278'18002, mlcod=278'18000)", 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.626368+0000", 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0069296719999999996, 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071186979999999997, 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626368+0000", 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626368+0000", 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626369+0000", 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-03-26T03:12:05.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626374+0000", 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9690000000000002e-06 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626379+0000", 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9220000000000002e-06 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626385+0000", 2024-03-26T03:12:05.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.028e-06 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.627239+0000", 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085410300000000005 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.627257+0000", 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8868e-05 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:284539 9.d 9:bfee6b0c:::benchmark_data_smithi066_125102_object138238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.626475+0000", 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0068230000000000001, 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070729410000000001, 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 284539 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.649 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626475+0000", 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626475+0000", 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626476+0000", 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.609e-06 2024-03-26T03:12:05.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626480+0000", 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.031e-06 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626489+0000", 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8419999999999994e-06 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.626496+0000", 2024-03-26T03:12:05.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7159999999999997e-06 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630058+0000", 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003562553 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630168+0000", 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010967 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630276+0000", 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107996 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.652 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:284544 9.e 9:72b92291:::benchmark_data_smithi066_125102_object138243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.630924+0000", 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002373315, 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002700895, 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:12:05.653 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 284544 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630924+0000", 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630924+0000", 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630926+0000", 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1299999999999999e-06 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630935+0000", 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9099999999999994e-06 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630979+0000", 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.405e-05 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.630988+0000", 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3930000000000008e-06 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631086+0000", 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7819999999999995e-05 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631184+0000", 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8753999999999997e-05 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631365+0000", 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018010199999999999 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:284545 9.3 9:c85ec54c:::benchmark_data_smithi066_125102_object138244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.631006+0000", 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002291562, 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026999300000000001, 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 284545 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631006+0000", 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631006+0000", 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631007+0000", 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7599999999999996e-07 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631011+0000", 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7859999999999998e-06 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631017+0000", 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1449999999999996e-06 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631022+0000", 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.519e-06 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631064+0000", 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461999999999997e-05 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631185+0000", 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120696 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631296+0000", 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011176299999999999 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:284546 9.3 9.4fb34503 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:12:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:05.631231+0000", 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0020671029999999998, 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025641990000000001, 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 284546 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631231+0000", 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631231+0000", 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631232+0000", 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0890000000000001e-06 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631236+0000", 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2599999999999999e-06 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631243+0000", 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9689999999999997e-06 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.631250+0000", 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0299999999999996e-06 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.664 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:12:05.665 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.665 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:12:05.665 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:12:05.665 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 7 2024-03-26T03:12:05.665 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:12:05.665 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:05.700 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:05.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:05.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:05.937 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:05.979 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:05.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:06.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.113 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:06.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:06.214 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:06.253 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:06.355 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:06.393 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:06.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:06.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:12:06.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:06.632 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.670 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:06.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:06.771 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:06.807 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:06.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:06.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:06.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:06.966 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:12:07.056 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:07.092 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:07.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:07.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:07.334 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:07.372 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:07 smithi192 ceph-mon[33110]: pgmap v575: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 566 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 4717/27831 objects degraded (16.949%); 1/27831 objects misplaced (0.004%) 2024-03-26T03:12:07.373 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:07.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:07.445 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:07.474 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:07.518 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:07.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:07 smithi066 ceph-mon[29666]: pgmap v575: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 566 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 4717/27831 objects degraded (16.949%); 1/27831 objects misplaced (0.004%) 2024-03-26T03:12:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:07 smithi066 ceph-mon[37920]: pgmap v575: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 566 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 4717/27831 objects degraded (16.949%); 1/27831 objects misplaced (0.004%) 2024-03-26T03:12:07.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:07.659 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:07.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.808 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:07.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:07.909 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:07.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:07.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:08.057 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:08.093 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:08.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:08.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:08.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:08.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:08.482 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.530 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:08.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:08.631 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:08.670 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:08.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:08.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:08.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:08.947 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:12:08.948 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:12:08.948 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:12:08.948 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:12:08.948 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:12:08.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:08.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:09.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:09.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:09.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:09.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:09.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:09.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:09.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:09.525 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:09.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:09 smithi066 ceph-mon[37920]: pgmap v576: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 566 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 4717/27831 objects degraded (16.949%); 1/27831 objects misplaced (0.004%) 2024-03-26T03:12:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 4717/27831 objects degraded (16.949%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:09 smithi066 ceph-mon[29666]: pgmap v576: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 566 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 4717/27831 objects degraded (16.949%); 1/27831 objects misplaced (0.004%) 2024-03-26T03:12:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 4717/27831 objects degraded (16.949%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:09.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:09.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:09.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:09 smithi192 ceph-mon[33110]: pgmap v576: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 566 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 4717/27831 objects degraded (16.949%); 1/27831 objects misplaced (0.004%) 2024-03-26T03:12:09.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 4717/27831 objects degraded (16.949%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:09.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.805 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:09.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:09.845 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:12:09.878 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:12:09.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:09.914 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:12:09.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:09.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:09.947 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:12:09.950 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:12:09.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:12:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:12:09.992 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:12:10.031 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:12:10.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:10.071 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:12:10.071 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:12:10.071 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:12:10.071 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:12:10.071 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:12:10.072 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:12:10.072 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:12:10.072 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:12:10.072 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:12:10.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:10.179 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.219 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:10.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:10.320 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:10.361 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:10.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:10.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:10.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:10.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:10 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1165463287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:10 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1165463287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:10.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:10.725 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:10 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1165463287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:10.770 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:10.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:10.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:12:10.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:10.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:10.981 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:12:10.985 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:12:11.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:11.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:12:11.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:12:11.157 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:11.195 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:11.264 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Removed 146298 objects 2024-03-26T03:12:11.265 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Clean up completed and total clean up time :106.325 2024-03-26T03:12:11.289 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:11.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:11.302 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:12:11.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:11.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:11.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:11.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:11.513 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:11 smithi192 ceph-mon[33110]: pgmap v577: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 239 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 1667/12126 objects degraded (13.747%); 1/12126 objects misplaced (0.008%) 2024-03-26T03:12:11.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:11.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:11.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:11.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:11.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:11.821 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:11.821 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:11 smithi066 ceph-mon[29666]: pgmap v577: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 239 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 1667/12126 objects degraded (13.747%); 1/12126 objects misplaced (0.008%) 2024-03-26T03:12:11.821 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:11.822 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:11 smithi066 ceph-mon[37920]: pgmap v577: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 239 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 1667/12126 objects degraded (13.747%); 1/12126 objects misplaced (0.008%) 2024-03-26T03:12:11.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:11.852 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2024-03-26T03:12:10.255334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":250372002,"num_objects":4042,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1667,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":2223,"num_read_kb":1946,"num_write":289705,"num_write_kb":9363191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":54528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79987,"ondisk_log_size":79987,"up":432,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":502,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6893748,"kb_used_data":3063540,"kb_used_omap":12,"kb_used_meta":3829747,"kb_avail":743067468,"statfs":{"total":767960285184,"available":760901087232,"internally_reserved":0,"allocated":3137064960,"data_stored":3132213770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3921661488},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,4,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-792920064,"num_objects":-12099,"num_object_clones":0,"num_object_copies":-36297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12099,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12099,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004273"},"pg_stats":[{"pgid":"1.d","version":"82'128","reported_seq":357,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358804+0000","last_change":"2024-03-26T03:10:50.602175+0000","last_active":"2024-03-26T03:12:03.358804+0000","last_peered":"2024-03-26T03:12:03.358804+0000","last_clean":"2024-03-26T03:12:03.358804+0000","last_became_active":"2024-03-26T03:10:44.090472+0000","last_became_peered":"2024-03-26T03:10:44.090472+0000","last_unstale":"2024-03-26T03:12:03.358804+0000","last_undegraded":"2024-03-26T03:12:03.358804+0000","last_fullsized":"2024-03-26T03:12:03.358804+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.602062+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.602062+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:16.544511+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"278'21510","reported_seq":21815,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:05.692200+0000","last_change":"2024-03-26T03:11:39.734959+0000","last_active":"2024-03-26T03:12:05.692200+0000","last_peered":"2024-03-26T03:12:05.692200+0000","last_clean":"2024-03-26T03:12:05.692200+0000","last_became_active":"2024-03-26T03:08:43.065248+0000","last_became_peered":"2024-03-26T03:08:43.065248+0000","last_unstale":"2024-03-26T03:12:05.692200+0000","last_undegraded":"2024-03-26T03:12:05.692200+0000","last_fullsized":"2024-03-26T03:12:05.692200+0000","mapping_epoch":225,"log_start":"268'16202","ondisk_log_start":"268'16202","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'19230","last_scrub_stamp":"2024-03-26T03:11:39.734819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:11:39.734819+0000","objects_scrubbed":2804,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:47.568427+0000","scrub_duration":1167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31457280,"num_objects":480,"num_object_clones":0,"num_object_copies":1440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17958,"num_write_kb":590016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":653,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:04.545507+0000","last_change":"2024-03-26T03:10:55.601026+0000","last_active":"2024-03-26T03:12:04.545507+0000","last_peered":"2024-03-26T03:12:04.545507+0000","last_clean":"2024-03-26T03:12:04.545507+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:12:04.545507+0000","last_undegraded":"2024-03-26T03:12:04.545507+0000","last_fullsized":"2024-03-26T03:12:04.545507+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:10:55.600815+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.600815+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.420458+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":571,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.353690+0000","last_change":"2024-03-26T03:11:39.721484+0000","last_active":"2024-03-26T03:12:03.353690+0000","last_peered":"2024-03-26T03:12:03.353690+0000","last_clean":"2024-03-26T03:12:03.353690+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:12:03.353690+0000","last_undegraded":"2024-03-26T03:12:03.353690+0000","last_fullsized":"2024-03-26T03:12:03.353690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:11:39.721355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:39.721355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:06.244718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":471,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.353529+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:12:03.353529+0000","last_peered":"2024-03-26T03:12:03.353529+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:12:03.353529+0000","last_undegraded":"2024-03-26T03:12:03.353529+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":293,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.360515+0000","last_change":"2024-03-26T03:10:59.286571+0000","last_active":"2024-03-26T03:12:03.360515+0000","last_peered":"2024-03-26T03:12:03.360515+0000","last_clean":"2024-03-26T03:12:03.360515+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:12:03.360515+0000","last_undegraded":"2024-03-26T03:12:03.360515+0000","last_fullsized":"2024-03-26T03:12:03.360515+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:59.286437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:59.286437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.053590+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":354,"reported_epoch":278,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:03.364883+0000","last_change":"2024-03-26T03:10:50.577501+0000","last_active":"2024-03-26T03:12:03.364883+0000","last_peered":"2024-03-26T03:12:03.364883+0000","last_clean":"2024-03-26T03:12:03.364883+0000","last_became_active":"2024-03-26T03:10:44.082186+0000","last_became_peered":"2024-03-26T03:10:44.082186+0000","last_unstale":"2024-03-26T03:12:03.364883+0000","last_undegraded":"2024-03-26T03:12:03.364883+0000","last_fullsized":"2024-03-26T03:12:03.364883+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.577205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.577205+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.267018+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"278'18002","reported_seq":18243,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:09.519177+0000","last_change":"2024-03-26T03:11:41.057805+0000","last_active":"2024-03-26T03:12:09.519177+0000","last_peered":"2024-03-26T03:12:09.519177+0000","last_clean":"2024-03-26T03:12:09.519177+0000","last_became_active":"2024-03-26T03:08:43.063988+0000","last_became_peered":"2024-03-26T03:08:43.063988+0000","last_unstale":"2024-03-26T03:12:09.519177+0000","last_undegraded":"2024-03-26T03:12:09.519177+0000","last_fullsized":"2024-03-26T03:12:09.519177+0000","mapping_epoch":225,"log_start":"260'11853","ondisk_log_start":"260'11853","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15622","last_scrub_stamp":"2024-03-26T03:11:41.057612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.057612+0000","objects_scrubbed":2578,"log_size":6149,"log_dups_size":0,"ondisk_log_size":6149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.964720+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18002,"num_write_kb":581760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":530,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358737+0000","last_change":"2024-03-26T03:11:11.587482+0000","last_active":"2024-03-26T03:12:03.358737+0000","last_peered":"2024-03-26T03:12:03.358737+0000","last_clean":"2024-03-26T03:12:03.358737+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:12:03.358737+0000","last_undegraded":"2024-03-26T03:12:03.358737+0000","last_fullsized":"2024-03-26T03:12:03.358737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:11:11.587327+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:27.886569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":471,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.362692+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:12:03.362692+0000","last_peered":"2024-03-26T03:12:03.362692+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:12:03.362692+0000","last_undegraded":"2024-03-26T03:12:03.362692+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236948+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":456,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.351920+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:12:03.351920+0000","last_peered":"2024-03-26T03:12:03.351920+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:12:03.351920+0000","last_undegraded":"2024-03-26T03:12:03.351920+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":481,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358943+0000","last_change":"2024-03-26T03:11:43.664485+0000","last_active":"2024-03-26T03:12:03.358943+0000","last_peered":"2024-03-26T03:12:03.358943+0000","last_clean":"2024-03-26T03:12:03.358943+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:12:03.358943+0000","last_undegraded":"2024-03-26T03:12:03.358943+0000","last_fullsized":"2024-03-26T03:12:03.358943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:43.664357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:11:43.664357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.152927+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":565,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361707+0000","last_change":"2024-03-26T03:10:49.362862+0000","last_active":"2024-03-26T03:12:03.361707+0000","last_peered":"2024-03-26T03:12:03.361707+0000","last_clean":"2024-03-26T03:12:03.361707+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:12:03.361707+0000","last_undegraded":"2024-03-26T03:12:03.361707+0000","last_fullsized":"2024-03-26T03:12:03.361707+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.362737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.362737+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.713660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"278'18240","reported_seq":18354,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.584460+0000","last_change":"2024-03-26T03:08:54.927812+0000","last_active":"2024-03-26T03:12:09.584460+0000","last_peered":"2024-03-26T03:12:09.584460+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.069288+0000","last_became_peered":"2024-03-26T03:08:43.069288+0000","last_unstale":"2024-03-26T03:12:09.584460+0000","last_undegraded":"2024-03-26T03:08:54.925664+0000","last_fullsized":"2024-03-26T03:08:43.062900+0000","mapping_epoch":225,"log_start":"268'14102","ondisk_log_start":"268'14102","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.918499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18240,"num_write_kb":588608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":154}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":584,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.360741+0000","last_change":"2024-03-26T03:11:12.308404+0000","last_active":"2024-03-26T03:12:03.360741+0000","last_peered":"2024-03-26T03:12:03.360741+0000","last_clean":"2024-03-26T03:12:03.360741+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:12:03.360741+0000","last_undegraded":"2024-03-26T03:12:03.360741+0000","last_fullsized":"2024-03-26T03:12:03.360741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:11:12.308255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:12.308255+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:29.673061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":508,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.352456+0000","last_change":"2024-03-26T03:11:51.169305+0000","last_active":"2024-03-26T03:12:03.352456+0000","last_peered":"2024-03-26T03:12:03.352456+0000","last_clean":"2024-03-26T03:12:03.352456+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:12:03.352456+0000","last_undegraded":"2024-03-26T03:12:03.352456+0000","last_fullsized":"2024-03-26T03:12:03.352456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.169187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.169187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:15.669587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.351931+0000","last_change":"2024-03-26T03:11:46.249753+0000","last_active":"2024-03-26T03:12:03.351931+0000","last_peered":"2024-03-26T03:12:03.351931+0000","last_clean":"2024-03-26T03:12:03.351931+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:12:03.351931+0000","last_undegraded":"2024-03-26T03:12:03.351931+0000","last_fullsized":"2024-03-26T03:12:03.351931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:46.249597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.249597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:08.763925+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":452,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358070+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:12:03.358070+0000","last_peered":"2024-03-26T03:12:03.358070+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:12:03.358070+0000","last_undegraded":"2024-03-26T03:12:03.358070+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"278'18267","reported_seq":18501,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:09.968001+0000","last_change":"2024-03-26T03:11:44.986339+0000","last_active":"2024-03-26T03:12:09.968001+0000","last_peered":"2024-03-26T03:12:09.968001+0000","last_clean":"2024-03-26T03:12:09.968001+0000","last_became_active":"2024-03-26T03:08:43.063932+0000","last_became_peered":"2024-03-26T03:08:43.063932+0000","last_unstale":"2024-03-26T03:12:09.968001+0000","last_undegraded":"2024-03-26T03:12:09.968001+0000","last_fullsized":"2024-03-26T03:12:09.968001+0000","mapping_epoch":225,"log_start":"266'13870","ondisk_log_start":"266'13870","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'16029","last_scrub_stamp":"2024-03-26T03:11:44.986128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:11:44.986128+0000","objects_scrubbed":2392,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:06.532658+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18267,"num_write_kb":588736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":544,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.364276+0000","last_change":"2024-03-26T03:10:56.581354+0000","last_active":"2024-03-26T03:12:03.364276+0000","last_peered":"2024-03-26T03:12:03.364276+0000","last_clean":"2024-03-26T03:12:03.364276+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:12:03.364276+0000","last_undegraded":"2024-03-26T03:12:03.364276+0000","last_fullsized":"2024-03-26T03:12:03.364276+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:10:56.581216+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:10:56.581216+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100304+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358979+0000","last_change":"2024-03-26T03:11:41.872736+0000","last_active":"2024-03-26T03:12:03.358979+0000","last_peered":"2024-03-26T03:12:03.358979+0000","last_clean":"2024-03-26T03:12:03.358979+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:12:03.358979+0000","last_undegraded":"2024-03-26T03:12:03.358979+0000","last_fullsized":"2024-03-26T03:12:03.358979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:41.872620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.872620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:50.631490+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":476,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.360126+0000","last_change":"2024-03-26T03:10:47.065658+0000","last_active":"2024-03-26T03:12:03.360126+0000","last_peered":"2024-03-26T03:12:03.360126+0000","last_clean":"2024-03-26T03:12:03.360126+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:12:03.360126+0000","last_undegraded":"2024-03-26T03:12:03.360126+0000","last_fullsized":"2024-03-26T03:12:03.360126+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:47.065556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.065556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.028651+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":459,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.352731+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:12:03.352731+0000","last_peered":"2024-03-26T03:12:03.352731+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:12:03.352731+0000","last_undegraded":"2024-03-26T03:12:03.352731+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219278+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":541,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.350970+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:12:03.350970+0000","last_peered":"2024-03-26T03:12:03.350970+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:12:03.350970+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"278'18325","reported_seq":18438,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.339100+0000","last_change":"2024-03-26T03:08:54.926696+0000","last_active":"2024-03-26T03:12:09.339100+0000","last_peered":"2024-03-26T03:12:09.339100+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.066721+0000","last_became_peered":"2024-03-26T03:08:43.066721+0000","last_unstale":"2024-03-26T03:12:09.339100+0000","last_undegraded":"2024-03-26T03:08:54.926086+0000","last_fullsized":"2024-03-26T03:08:43.059549+0000","mapping_epoch":225,"log_start":"261'12894","ondisk_log_start":"261'12894","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5431,"log_dups_size":0,"ondisk_log_size":5431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.195921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18325,"num_write_kb":591936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":173}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":732,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359316+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:12:03.359316+0000","last_peered":"2024-03-26T03:12:03.359316+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:12:03.359316+0000","last_undegraded":"2024-03-26T03:12:03.359316+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":496,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.364870+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:12:03.364870+0000","last_peered":"2024-03-26T03:12:03.364870+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:12:03.364870+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100179+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":464,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358998+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:12:03.358998+0000","last_peered":"2024-03-26T03:12:03.358998+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:12:03.358998+0000","last_undegraded":"2024-03-26T03:12:03.358998+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236908+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":660,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.352337+0000","last_change":"2024-03-26T03:10:55.115703+0000","last_active":"2024-03-26T03:12:03.352337+0000","last_peered":"2024-03-26T03:12:03.352337+0000","last_clean":"2024-03-26T03:12:03.352337+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:12:03.352337+0000","last_undegraded":"2024-03-26T03:12:03.352337+0000","last_fullsized":"2024-03-26T03:12:03.352337+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:10:55.115576+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.115576+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:19.781331+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":471,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.365464+0000","last_change":"2024-03-26T03:10:47.638717+0000","last_active":"2024-03-26T03:12:03.365464+0000","last_peered":"2024-03-26T03:12:03.365464+0000","last_clean":"2024-03-26T03:12:03.365464+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:12:03.365464+0000","last_undegraded":"2024-03-26T03:12:03.365464+0000","last_fullsized":"2024-03-26T03:12:03.365464+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:47.638593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.638593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100271+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"278'18146","reported_seq":18260,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.981885+0000","last_change":"2024-03-26T03:08:54.925459+0000","last_active":"2024-03-26T03:12:09.981885+0000","last_peered":"2024-03-26T03:12:09.981885+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.057035+0000","last_became_peered":"2024-03-26T03:08:43.057035+0000","last_unstale":"2024-03-26T03:12:09.981885+0000","last_undegraded":"2024-03-26T03:08:54.924597+0000","last_fullsized":"2024-03-26T03:08:43.055113+0000","mapping_epoch":225,"log_start":"266'13772","ondisk_log_start":"266'13772","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.241841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18146,"num_write_kb":584192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":110}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":579,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.351736+0000","last_change":"2024-03-26T03:11:52.158484+0000","last_active":"2024-03-26T03:12:03.351736+0000","last_peered":"2024-03-26T03:12:03.351736+0000","last_clean":"2024-03-26T03:12:03.351736+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:12:03.351736+0000","last_undegraded":"2024-03-26T03:12:03.351736+0000","last_fullsized":"2024-03-26T03:12:03.351736+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:11:52.158163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:11:52.158163+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:07.642900+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":510,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.358458+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:12:03.358458+0000","last_peered":"2024-03-26T03:12:03.358458+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:12:03.358458+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236936+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":464,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.356370+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:12:03.356370+0000","last_peered":"2024-03-26T03:12:03.356370+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:12:03.356370+0000","last_undegraded":"2024-03-26T03:12:03.356370+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":650,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.365409+0000","last_change":"2024-03-26T03:11:40.792357+0000","last_active":"2024-03-26T03:12:03.365409+0000","last_peered":"2024-03-26T03:12:03.365409+0000","last_clean":"2024-03-26T03:12:03.365409+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:12:03.365409+0000","last_undegraded":"2024-03-26T03:12:03.365409+0000","last_fullsized":"2024-03-26T03:12:03.365409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:11:40.791990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.791990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:50.059248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362233+0000","last_change":"2024-03-26T03:11:01.239298+0000","last_active":"2024-03-26T03:12:03.362233+0000","last_peered":"2024-03-26T03:12:03.362233+0000","last_clean":"2024-03-26T03:12:03.362233+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:12:03.362233+0000","last_undegraded":"2024-03-26T03:12:03.362233+0000","last_fullsized":"2024-03-26T03:12:03.362233+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.239012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.239012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:25.557947+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":583,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359546+0000","last_change":"2024-03-26T03:10:51.016455+0000","last_active":"2024-03-26T03:12:03.359546+0000","last_peered":"2024-03-26T03:12:03.359546+0000","last_clean":"2024-03-26T03:12:03.359546+0000","last_became_active":"2024-03-26T03:10:44.079636+0000","last_became_peered":"2024-03-26T03:10:44.079636+0000","last_unstale":"2024-03-26T03:12:03.359546+0000","last_undegraded":"2024-03-26T03:12:03.359546+0000","last_fullsized":"2024-03-26T03:12:03.359546+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.016116+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.016116+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:11.231426+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"278'17998","reported_seq":18111,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.594761+0000","last_change":"2024-03-26T03:08:54.940154+0000","last_active":"2024-03-26T03:12:09.594761+0000","last_peered":"2024-03-26T03:12:09.594761+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.064645+0000","last_became_peered":"2024-03-26T03:08:43.064645+0000","last_unstale":"2024-03-26T03:12:09.594761+0000","last_undegraded":"2024-03-26T03:08:54.937570+0000","last_fullsized":"2024-03-26T03:08:43.060747+0000","mapping_epoch":225,"log_start":"268'13880","ondisk_log_start":"268'13880","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.918485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043968,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17998,"num_write_kb":580352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":138}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"231'25","reported_seq":530,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.365693+0000","last_change":"2024-03-26T03:11:06.580747+0000","last_active":"2024-03-26T03:12:03.365693+0000","last_peered":"2024-03-26T03:12:03.365693+0000","last_clean":"2024-03-26T03:12:03.365693+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:12:03.365693+0000","last_undegraded":"2024-03-26T03:12:03.365693+0000","last_fullsized":"2024-03-26T03:12:03.365693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:11:06.580543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:06.580543+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.318740+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358224+0000","last_change":"2024-03-26T03:11:00.578809+0000","last_active":"2024-03-26T03:12:03.358224+0000","last_peered":"2024-03-26T03:12:03.358224+0000","last_clean":"2024-03-26T03:12:03.358224+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:12:03.358224+0000","last_undegraded":"2024-03-26T03:12:03.358224+0000","last_fullsized":"2024-03-26T03:12:03.358224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:00.578704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:00.578704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.102616+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":291,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.364241+0000","last_change":"2024-03-26T03:11:48.965867+0000","last_active":"2024-03-26T03:12:03.364241+0000","last_peered":"2024-03-26T03:12:03.364241+0000","last_clean":"2024-03-26T03:12:03.364241+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:12:03.364241+0000","last_undegraded":"2024-03-26T03:12:03.364241+0000","last_fullsized":"2024-03-26T03:12:03.364241+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.965732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.965732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:12.010040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.361131+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:12:03.361131+0000","last_peered":"2024-03-26T03:12:03.361131+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:12:03.361131+0000","last_undegraded":"2024-03-26T03:12:03.361131+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"278'18120","reported_seq":18234,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.213110+0000","last_change":"2024-03-26T03:08:54.944412+0000","last_active":"2024-03-26T03:12:09.213110+0000","last_peered":"2024-03-26T03:12:09.213110+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.063578+0000","last_became_peered":"2024-03-26T03:08:43.063578+0000","last_unstale":"2024-03-26T03:12:09.213110+0000","last_undegraded":"2024-03-26T03:08:54.938583+0000","last_fullsized":"2024-03-26T03:08:43.056078+0000","mapping_epoch":225,"log_start":"264'13339","ondisk_log_start":"264'13339","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:11.121806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18120,"num_write_kb":585856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":188}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":329,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.357230+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:12:03.357230+0000","last_peered":"2024-03-26T03:12:03.357230+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:12:03.357230+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":542,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362973+0000","last_change":"2024-03-26T03:10:15.759744+0000","last_active":"2024-03-26T03:12:03.362973+0000","last_peered":"2024-03-26T03:12:03.362973+0000","last_clean":"2024-03-26T03:12:03.362973+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:12:03.362973+0000","last_undegraded":"2024-03-26T03:12:03.362973+0000","last_fullsized":"2024-03-26T03:12:03.362973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:10:15.759584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:15.759584+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100241+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361959+0000","last_change":"2024-03-26T03:11:07.220709+0000","last_active":"2024-03-26T03:12:03.361959+0000","last_peered":"2024-03-26T03:12:03.361959+0000","last_clean":"2024-03-26T03:12:03.361959+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:12:03.361959+0000","last_undegraded":"2024-03-26T03:12:03.361959+0000","last_fullsized":"2024-03-26T03:12:03.361959+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:07.220578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.220578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:08.544006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":481,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359322+0000","last_change":"2024-03-26T03:11:49.921518+0000","last_active":"2024-03-26T03:12:03.359322+0000","last_peered":"2024-03-26T03:12:03.359322+0000","last_clean":"2024-03-26T03:12:03.359322+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:12:03.359322+0000","last_undegraded":"2024-03-26T03:12:03.359322+0000","last_fullsized":"2024-03-26T03:12:03.359322+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.921407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:05.885611+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":651,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358809+0000","last_change":"2024-03-26T03:10:39.370375+0000","last_active":"2024-03-26T03:12:03.358809+0000","last_peered":"2024-03-26T03:12:03.358809+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:10:39.370375+0000","last_became_peered":"2024-03-26T03:10:39.370375+0000","last_unstale":"2024-03-26T03:12:03.358809+0000","last_undegraded":"2024-03-26T03:12:03.358809+0000","last_fullsized":"2024-03-26T03:10:39.363411+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:03.801819+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"278'18307","reported_seq":18498,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:09.530471+0000","last_change":"2024-03-26T03:10:54.851702+0000","last_active":"2024-03-26T03:12:09.530471+0000","last_peered":"2024-03-26T03:12:09.530471+0000","last_clean":"2024-03-26T03:12:09.530471+0000","last_became_active":"2024-03-26T03:08:43.068853+0000","last_became_peered":"2024-03-26T03:08:43.068853+0000","last_unstale":"2024-03-26T03:12:09.530471+0000","last_undegraded":"2024-03-26T03:12:09.530471+0000","last_fullsized":"2024-03-26T03:12:09.530471+0000","mapping_epoch":225,"log_start":"268'14200","ondisk_log_start":"268'14200","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'12055","last_scrub_stamp":"2024-03-26T03:10:54.851285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.851285+0000","objects_scrubbed":6498,"log_size":4107,"log_dups_size":0,"ondisk_log_size":4107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.921249+0000","scrub_duration":2294,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437208,"num_objects":145,"num_object_clones":0,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18307,"num_write_kb":590401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":455,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.363343+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:12:03.363343+0000","last_peered":"2024-03-26T03:12:03.363343+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:12:03.363343+0000","last_undegraded":"2024-03-26T03:12:03.363343+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":295,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.358369+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:12:03.358369+0000","last_peered":"2024-03-26T03:12:03.358369+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:12:03.358369+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":495,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.363564+0000","last_change":"2024-03-26T03:11:41.297116+0000","last_active":"2024-03-26T03:12:03.363564+0000","last_peered":"2024-03-26T03:12:03.363564+0000","last_clean":"2024-03-26T03:12:03.363564+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:12:03.363564+0000","last_undegraded":"2024-03-26T03:12:03.363564+0000","last_fullsized":"2024-03-26T03:12:03.363564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:11:41.296291+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.296291+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:51.235662+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.362727+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:12:03.362727+0000","last_peered":"2024-03-26T03:12:03.362727+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:12:03.362727+0000","last_undegraded":"2024-03-26T03:12:03.362727+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100153+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":576,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.364567+0000","last_change":"2024-03-26T03:10:51.609357+0000","last_active":"2024-03-26T03:12:03.364567+0000","last_peered":"2024-03-26T03:12:03.364567+0000","last_clean":"2024-03-26T03:12:03.364567+0000","last_became_active":"2024-03-26T03:10:39.368150+0000","last_became_peered":"2024-03-26T03:10:39.368150+0000","last_unstale":"2024-03-26T03:12:03.364567+0000","last_undegraded":"2024-03-26T03:12:03.364567+0000","last_fullsized":"2024-03-26T03:12:03.364567+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.609183+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.609183+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:14.028320+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"278'18060","reported_seq":18272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:09.532152+0000","last_change":"2024-03-26T03:10:21.408571+0000","last_active":"2024-03-26T03:12:09.532152+0000","last_peered":"2024-03-26T03:12:09.532152+0000","last_clean":"2024-03-26T03:12:09.532152+0000","last_became_active":"2024-03-26T03:08:43.067345+0000","last_became_peered":"2024-03-26T03:08:43.067345+0000","last_unstale":"2024-03-26T03:12:09.532152+0000","last_undegraded":"2024-03-26T03:12:09.532152+0000","last_fullsized":"2024-03-26T03:12:09.532152+0000","mapping_epoch":225,"log_start":"268'13984","ondisk_log_start":"268'13984","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'8823","last_scrub_stamp":"2024-03-26T03:10:21.407979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:21.407979+0000","objects_scrubbed":8714,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.918635+0000","scrub_duration":2784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18060,"num_write_kb":582144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":485,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.360799+0000","last_change":"2024-03-26T03:10:54.285604+0000","last_active":"2024-03-26T03:12:03.360799+0000","last_peered":"2024-03-26T03:12:03.360799+0000","last_clean":"2024-03-26T03:12:03.360799+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:12:03.360799+0000","last_undegraded":"2024-03-26T03:12:03.360799+0000","last_fullsized":"2024-03-26T03:12:03.360799+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.285471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.285471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:11.594353+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":560,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358411+0000","last_change":"2024-03-26T03:11:47.624747+0000","last_active":"2024-03-26T03:12:03.358411+0000","last_peered":"2024-03-26T03:12:03.358411+0000","last_clean":"2024-03-26T03:12:03.358411+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:12:03.358411+0000","last_undegraded":"2024-03-26T03:12:03.358411+0000","last_fullsized":"2024-03-26T03:12:03.358411+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:11:47.624655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.624655+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:17.291201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":485,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359917+0000","last_change":"2024-03-26T03:10:54.596765+0000","last_active":"2024-03-26T03:12:03.359917+0000","last_peered":"2024-03-26T03:12:03.359917+0000","last_clean":"2024-03-26T03:12:03.359917+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:12:03.359917+0000","last_undegraded":"2024-03-26T03:12:03.359917+0000","last_fullsized":"2024-03-26T03:12:03.359917+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.596513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.596513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.924242+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.356642+0000","last_change":"2024-03-26T03:11:16.222874+0000","last_active":"2024-03-26T03:12:03.356642+0000","last_peered":"2024-03-26T03:12:03.356642+0000","last_clean":"2024-03-26T03:12:03.356642+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:12:03.356642+0000","last_undegraded":"2024-03-26T03:12:03.356642+0000","last_fullsized":"2024-03-26T03:12:03.356642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.222744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.635224+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":550,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.361267+0000","last_change":"2024-03-26T03:10:39.372465+0000","last_active":"2024-03-26T03:12:03.361267+0000","last_peered":"2024-03-26T03:12:03.361267+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:10:39.372465+0000","last_became_peered":"2024-03-26T03:10:39.372465+0000","last_unstale":"2024-03-26T03:12:03.361267+0000","last_undegraded":"2024-03-26T03:12:03.361267+0000","last_fullsized":"2024-03-26T03:10:39.360974+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.170163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"278'17842","reported_seq":18027,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:05.673214+0000","last_change":"2024-03-26T03:10:57.300217+0000","last_active":"2024-03-26T03:12:05.673214+0000","last_peered":"2024-03-26T03:12:05.673214+0000","last_clean":"2024-03-26T03:12:05.673214+0000","last_became_active":"2024-03-26T03:08:43.066837+0000","last_became_peered":"2024-03-26T03:08:43.066837+0000","last_unstale":"2024-03-26T03:12:05.673214+0000","last_undegraded":"2024-03-26T03:12:05.673214+0000","last_fullsized":"2024-03-26T03:12:05.673214+0000","mapping_epoch":225,"log_start":"260'12544","ondisk_log_start":"260'12544","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'12163","last_scrub_stamp":"2024-03-26T03:10:57.299961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.299961+0000","objects_scrubbed":6295,"log_size":5298,"log_dups_size":0,"ondisk_log_size":5298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.920679+0000","scrub_duration":1960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35520512,"num_objects":542,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17842,"num_write_kb":588288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.354794+0000","last_change":"2024-03-26T03:11:13.563915+0000","last_active":"2024-03-26T03:12:03.354794+0000","last_peered":"2024-03-26T03:12:03.354794+0000","last_clean":"2024-03-26T03:12:03.354794+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:12:03.354794+0000","last_undegraded":"2024-03-26T03:12:03.354794+0000","last_fullsized":"2024-03-26T03:12:03.354794+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.563699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.563699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.093265+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":539,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359086+0000","last_change":"2024-03-26T03:11:48.575617+0000","last_active":"2024-03-26T03:12:03.359086+0000","last_peered":"2024-03-26T03:12:03.359086+0000","last_clean":"2024-03-26T03:12:03.359086+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:12:03.359086+0000","last_undegraded":"2024-03-26T03:12:03.359086+0000","last_fullsized":"2024-03-26T03:12:03.359086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.575462+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:48.657881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358146+0000","last_change":"2024-03-26T03:11:03.629624+0000","last_active":"2024-03-26T03:12:03.358146+0000","last_peered":"2024-03-26T03:12:03.358146+0000","last_clean":"2024-03-26T03:12:03.358146+0000","last_became_active":"2024-03-26T02:58:56.925660+0000","last_became_peered":"2024-03-26T02:58:56.925660+0000","last_unstale":"2024-03-26T03:12:03.358146+0000","last_undegraded":"2024-03-26T03:12:03.358146+0000","last_fullsized":"2024-03-26T03:12:03.358146+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:03.629514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:03.629514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.141143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":479,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358061+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:12:03.358061+0000","last_peered":"2024-03-26T03:12:03.358061+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:12:03.358061+0000","last_undegraded":"2024-03-26T03:12:03.358061+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":460,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.360220+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:12:03.360220+0000","last_peered":"2024-03-26T03:12:03.360220+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:12:03.360220+0000","last_undegraded":"2024-03-26T03:12:03.360220+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":576,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359652+0000","last_change":"2024-03-26T03:10:52.006406+0000","last_active":"2024-03-26T03:12:03.359652+0000","last_peered":"2024-03-26T03:12:03.359652+0000","last_clean":"2024-03-26T03:12:03.359652+0000","last_became_active":"2024-03-26T03:10:39.379213+0000","last_became_peered":"2024-03-26T03:10:39.379213+0000","last_unstale":"2024-03-26T03:12:03.359652+0000","last_undegraded":"2024-03-26T03:12:03.359652+0000","last_fullsized":"2024-03-26T03:12:03.359652+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:52.006290+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:52.006290+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:08.240502+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"278'17378","reported_seq":17492,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:05.637611+0000","last_change":"2024-03-26T03:08:54.931470+0000","last_active":"2024-03-26T03:12:05.637611+0000","last_peered":"2024-03-26T03:12:05.637611+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.065260+0000","last_became_peered":"2024-03-26T03:08:43.065260+0000","last_unstale":"2024-03-26T03:12:05.637611+0000","last_undegraded":"2024-03-26T03:08:54.924879+0000","last_fullsized":"2024-03-26T03:08:43.057979+0000","mapping_epoch":225,"log_start":"260'12028","ondisk_log_start":"260'12028","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:09.453150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":522,"num_object_clones":0,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17378,"num_write_kb":572800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":485,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359085+0000","last_change":"2024-03-26T03:11:47.792917+0000","last_active":"2024-03-26T03:12:03.359085+0000","last_peered":"2024-03-26T03:12:03.359085+0000","last_clean":"2024-03-26T03:12:03.359085+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:12:03.359085+0000","last_undegraded":"2024-03-26T03:12:03.359085+0000","last_fullsized":"2024-03-26T03:12:03.359085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:47.792700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.792700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:54.932511+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"231'28","reported_seq":688,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361538+0000","last_change":"2024-03-26T03:10:48.343491+0000","last_active":"2024-03-26T03:12:03.361538+0000","last_peered":"2024-03-26T03:12:03.361538+0000","last_clean":"2024-03-26T03:12:03.361538+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:12:03.361538+0000","last_undegraded":"2024-03-26T03:12:03.361538+0000","last_fullsized":"2024-03-26T03:12:03.361538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:10:48.343341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:10:48.343341+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":751,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358188+0000","last_change":"2024-03-26T03:10:47.568899+0000","last_active":"2024-03-26T03:12:03.358188+0000","last_peered":"2024-03-26T03:12:03.358188+0000","last_clean":"2024-03-26T03:12:03.358188+0000","last_became_active":"2024-03-26T03:10:39.378619+0000","last_became_peered":"2024-03-26T03:10:39.378619+0000","last_unstale":"2024-03-26T03:12:03.358188+0000","last_undegraded":"2024-03-26T03:12:03.358188+0000","last_fullsized":"2024-03-26T03:12:03.358188+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:47.568755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.568755+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.757752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":514,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.353836+0000","last_change":"2024-03-26T03:11:14.586398+0000","last_active":"2024-03-26T03:12:03.353836+0000","last_peered":"2024-03-26T03:12:03.353836+0000","last_clean":"2024-03-26T03:12:03.353836+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:12:03.353836+0000","last_undegraded":"2024-03-26T03:12:03.353836+0000","last_fullsized":"2024-03-26T03:12:03.353836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:14.586295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:11:14.586295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:20.767744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":195,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.354392+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:12:03.354392+0000","last_peered":"2024-03-26T03:12:03.354392+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:12:03.354392+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.910034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":459,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.354863+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:12:03.354863+0000","last_peered":"2024-03-26T03:12:03.354863+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:12:03.354863+0000","last_undegraded":"2024-03-26T03:12:03.354863+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.910004+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"278'18080","reported_seq":18320,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:09.503351+0000","last_change":"2024-03-26T03:11:42.709422+0000","last_active":"2024-03-26T03:12:09.503351+0000","last_peered":"2024-03-26T03:12:09.503351+0000","last_clean":"2024-03-26T03:12:09.503351+0000","last_became_active":"2024-03-26T03:08:43.066310+0000","last_became_peered":"2024-03-26T03:08:43.066310+0000","last_unstale":"2024-03-26T03:12:09.503351+0000","last_undegraded":"2024-03-26T03:12:09.503351+0000","last_fullsized":"2024-03-26T03:12:09.503351+0000","mapping_epoch":225,"log_start":"266'13585","ondisk_log_start":"266'13585","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'15734","last_scrub_stamp":"2024-03-26T03:11:42.709159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.709159+0000","objects_scrubbed":2521,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:10.859412+0000","scrub_duration":987,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":132,"num_object_clones":0,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18080,"num_write_kb":582784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.353840+0000","last_change":"2024-03-26T03:11:15.024122+0000","last_active":"2024-03-26T03:12:03.353840+0000","last_peered":"2024-03-26T03:12:03.353840+0000","last_clean":"2024-03-26T03:12:03.353840+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:12:03.353840+0000","last_undegraded":"2024-03-26T03:12:03.353840+0000","last_fullsized":"2024-03-26T03:12:03.353840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.024002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.024002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:34.521104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":534,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359681+0000","last_change":"2024-03-26T03:11:11.116895+0000","last_active":"2024-03-26T03:12:03.359681+0000","last_peered":"2024-03-26T03:12:03.359681+0000","last_clean":"2024-03-26T03:12:03.359681+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:12:03.359681+0000","last_undegraded":"2024-03-26T03:12:03.359681+0000","last_fullsized":"2024-03-26T03:12:03.359681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:11:11.116642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:11.116642+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.643762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":472,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.351471+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:12:03.351471+0000","last_peered":"2024-03-26T03:12:03.351471+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:12:03.351471+0000","last_undegraded":"2024-03-26T03:12:03.351471+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219290+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1293,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:08.172058+0000","last_change":"2024-03-26T03:11:48.252850+0000","last_active":"2024-03-26T03:12:08.172058+0000","last_peered":"2024-03-26T03:12:08.172058+0000","last_clean":"2024-03-26T03:12:08.172058+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:12:08.172058+0000","last_undegraded":"2024-03-26T03:12:08.172058+0000","last_fullsized":"2024-03-26T03:12:08.172058+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:11:48.252717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.252717+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:03.591867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":782,"num_read_kb":777,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":460,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359789+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:12:03.359789+0000","last_peered":"2024-03-26T03:12:03.359789+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:12:03.359789+0000","last_undegraded":"2024-03-26T03:12:03.359789+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":758,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358681+0000","last_change":"2024-03-26T03:10:46.593971+0000","last_active":"2024-03-26T03:12:03.358681+0000","last_peered":"2024-03-26T03:12:03.358681+0000","last_clean":"2024-03-26T03:12:03.358681+0000","last_became_active":"2024-03-26T03:10:39.379191+0000","last_became_peered":"2024-03-26T03:10:39.379191+0000","last_unstale":"2024-03-26T03:12:03.358681+0000","last_undegraded":"2024-03-26T03:12:03.358681+0000","last_fullsized":"2024-03-26T03:12:03.358681+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:46.593847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:46.593847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"278'17861","reported_seq":18039,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:05.673639+0000","last_change":"2024-03-26T03:11:07.255990+0000","last_active":"2024-03-26T03:12:05.673639+0000","last_peered":"2024-03-26T03:12:05.673639+0000","last_clean":"2024-03-26T03:12:05.673639+0000","last_became_active":"2024-03-26T03:08:43.065826+0000","last_became_peered":"2024-03-26T03:08:43.065826+0000","last_unstale":"2024-03-26T03:12:05.673639+0000","last_undegraded":"2024-03-26T03:12:05.673639+0000","last_fullsized":"2024-03-26T03:12:05.673639+0000","mapping_epoch":225,"log_start":"260'12534","ondisk_log_start":"260'12534","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'13144","last_scrub_stamp":"2024-03-26T03:11:07.255669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.255669+0000","objects_scrubbed":5305,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:22.559544+0000","scrub_duration":1702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17861,"num_write_kb":588160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":485,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358516+0000","last_change":"2024-03-26T03:11:40.584646+0000","last_active":"2024-03-26T03:12:03.358516+0000","last_peered":"2024-03-26T03:12:03.358516+0000","last_clean":"2024-03-26T03:12:03.358516+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:12:03.358516+0000","last_undegraded":"2024-03-26T03:12:03.358516+0000","last_fullsized":"2024-03-26T03:12:03.358516+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:40.584536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.584536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:49.955813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"231'58","reported_seq":717,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359764+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:12:03.359764+0000","last_peered":"2024-03-26T03:12:03.359764+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:12:03.359764+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236922+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358190+0000","last_change":"2024-03-26T03:11:04.160279+0000","last_active":"2024-03-26T03:12:03.358190+0000","last_peered":"2024-03-26T03:12:03.358190+0000","last_clean":"2024-03-26T03:12:03.358190+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:12:03.358190+0000","last_undegraded":"2024-03-26T03:12:03.358190+0000","last_fullsized":"2024-03-26T03:12:03.358190+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:04.160195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:04.160195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:10.320752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.365559+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:12:03.365559+0000","last_peered":"2024-03-26T03:12:03.365559+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:12:03.365559+0000","last_undegraded":"2024-03-26T03:12:03.365559+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":585,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.357950+0000","last_change":"2024-03-26T03:12:02.353776+0000","last_active":"2024-03-26T03:12:03.357950+0000","last_peered":"2024-03-26T03:12:03.357950+0000","last_clean":"2024-03-26T03:12:03.357950+0000","last_became_active":"2024-03-26T03:10:39.376563+0000","last_became_peered":"2024-03-26T03:10:39.376563+0000","last_unstale":"2024-03-26T03:12:03.357950+0000","last_undegraded":"2024-03-26T03:12:03.357950+0000","last_fullsized":"2024-03-26T03:12:03.357950+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:47.195197+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.195197+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"278'18058","reported_seq":18172,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.537260+0000","last_change":"2024-03-26T03:08:54.940176+0000","last_active":"2024-03-26T03:12:09.537260+0000","last_peered":"2024-03-26T03:12:09.537260+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.067789+0000","last_became_peered":"2024-03-26T03:08:43.067789+0000","last_unstale":"2024-03-26T03:12:09.537260+0000","last_undegraded":"2024-03-26T03:08:54.937224+0000","last_fullsized":"2024-03-26T03:08:43.062110+0000","mapping_epoch":225,"log_start":"266'13545","ondisk_log_start":"266'13545","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18058,"num_write_kb":582720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":152}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":654,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359212+0000","last_change":"2024-03-26T03:11:14.528793+0000","last_active":"2024-03-26T03:12:03.359212+0000","last_peered":"2024-03-26T03:12:03.359212+0000","last_clean":"2024-03-26T03:12:03.359212+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:12:03.359212+0000","last_undegraded":"2024-03-26T03:12:03.359212+0000","last_fullsized":"2024-03-26T03:12:03.359212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:11:14.528661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:14.528661+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.412124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":532,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359657+0000","last_change":"2024-03-26T03:10:55.068251+0000","last_active":"2024-03-26T03:12:03.359657+0000","last_peered":"2024-03-26T03:12:03.359657+0000","last_clean":"2024-03-26T03:12:03.359657+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:12:03.359657+0000","last_undegraded":"2024-03-26T03:12:03.359657+0000","last_fullsized":"2024-03-26T03:12:03.359657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:10:55.067970+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.067970+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.607085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359432+0000","last_change":"2024-03-26T03:11:10.075730+0000","last_active":"2024-03-26T03:12:03.359432+0000","last_peered":"2024-03-26T03:12:03.359432+0000","last_clean":"2024-03-26T03:12:03.359432+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:12:03.359432+0000","last_undegraded":"2024-03-26T03:12:03.359432+0000","last_fullsized":"2024-03-26T03:12:03.359432+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:10.075607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:10.075607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:26.228686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":460,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.357659+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:12:03.357659+0000","last_peered":"2024-03-26T03:12:03.357659+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:12:03.357659+0000","last_undegraded":"2024-03-26T03:12:03.357659+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":550,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.361804+0000","last_change":"2024-03-26T03:10:39.371201+0000","last_active":"2024-03-26T03:12:03.361804+0000","last_peered":"2024-03-26T03:12:03.361804+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:10:39.371201+0000","last_became_peered":"2024-03-26T03:10:39.371201+0000","last_unstale":"2024-03-26T03:12:03.361804+0000","last_undegraded":"2024-03-26T03:12:03.361804+0000","last_fullsized":"2024-03-26T03:10:39.361046+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.170182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":355,"reported_epoch":278,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:03.358364+0000","last_change":"2024-03-26T03:10:50.310997+0000","last_active":"2024-03-26T03:12:03.358364+0000","last_peered":"2024-03-26T03:12:03.358364+0000","last_clean":"2024-03-26T03:12:03.358364+0000","last_became_active":"2024-03-26T03:10:44.081411+0000","last_became_peered":"2024-03-26T03:10:44.081411+0000","last_unstale":"2024-03-26T03:12:03.358364+0000","last_undegraded":"2024-03-26T03:12:03.358364+0000","last_fullsized":"2024-03-26T03:12:03.358364+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.310701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.310701+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.015695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"278'18144","reported_seq":18258,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:09.944419+0000","last_change":"2024-03-26T03:08:54.931125+0000","last_active":"2024-03-26T03:12:09.944419+0000","last_peered":"2024-03-26T03:12:09.944419+0000","last_clean":"2024-03-26T03:08:42.037937+0000","last_became_active":"2024-03-26T03:08:43.058021+0000","last_became_peered":"2024-03-26T03:08:43.058021+0000","last_unstale":"2024-03-26T03:12:09.944419+0000","last_undegraded":"2024-03-26T03:08:54.930155+0000","last_fullsized":"2024-03-26T03:08:43.054400+0000","mapping_epoch":225,"log_start":"266'13737","ondisk_log_start":"266'13737","created":225,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:42.037937+0000","last_clean_scrub_stamp":"2024-03-26T03:08:42.037937+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.241848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18144,"num_write_kb":584320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.351443+0000","last_change":"2024-03-26T03:10:50.022827+0000","last_active":"2024-03-26T03:12:03.351443+0000","last_peered":"2024-03-26T03:12:03.351443+0000","last_clean":"2024-03-26T03:12:03.351443+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:12:03.351443+0000","last_undegraded":"2024-03-26T03:12:03.351443+0000","last_fullsized":"2024-03-26T03:12:03.351443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:50.022721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.022721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:08.998611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":537,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.363256+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:12:03.363256+0000","last_peered":"2024-03-26T03:12:03.363256+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:12:03.363256+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":523,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.364080+0000","last_change":"2024-03-26T03:10:58.571618+0000","last_active":"2024-03-26T03:12:03.364080+0000","last_peered":"2024-03-26T03:12:03.364080+0000","last_clean":"2024-03-26T03:12:03.364080+0000","last_became_active":"2024-03-26T03:10:08.555235+0000","last_became_peered":"2024-03-26T03:10:08.555235+0000","last_unstale":"2024-03-26T03:12:03.364080+0000","last_undegraded":"2024-03-26T03:12:03.364080+0000","last_fullsized":"2024-03-26T03:12:03.364080+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:10:58.571492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:58.571492+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.423266+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359076+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:12:03.359076+0000","last_peered":"2024-03-26T03:12:03.359076+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:12:03.359076+0000","last_undegraded":"2024-03-26T03:12:03.359076+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":579,"reported_epoch":278,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:03.359307+0000","last_change":"2024-03-26T03:10:48.023253+0000","last_active":"2024-03-26T03:12:03.359307+0000","last_peered":"2024-03-26T03:12:03.359307+0000","last_clean":"2024-03-26T03:12:03.359307+0000","last_became_active":"2024-03-26T03:10:42.404423+0000","last_became_peered":"2024-03-26T03:10:42.404423+0000","last_unstale":"2024-03-26T03:12:03.359307+0000","last_undegraded":"2024-03-26T03:12:03.359307+0000","last_fullsized":"2024-03-26T03:12:03.359307+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:48.023049+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:10:48.023049+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.241070+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":516,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359195+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:12:03.359195+0000","last_peered":"2024-03-26T03:12:03.359195+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:12:03.359195+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":506,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.351307+0000","last_change":"2024-03-26T03:11:14.000120+0000","last_active":"2024-03-26T03:12:03.351307+0000","last_peered":"2024-03-26T03:12:03.351307+0000","last_clean":"2024-03-26T03:12:03.351307+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:12:03.351307+0000","last_undegraded":"2024-03-26T03:12:03.351307+0000","last_fullsized":"2024-03-26T03:12:03.351307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:11:13.999986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.999986+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:19.357060+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358659+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:12:03.358659+0000","last_peered":"2024-03-26T03:12:03.358659+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:12:03.358659+0000","last_undegraded":"2024-03-26T03:12:03.358659+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":488,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.352228+0000","last_change":"2024-03-26T03:11:38.225205+0000","last_active":"2024-03-26T03:12:03.352228+0000","last_peered":"2024-03-26T03:12:03.352228+0000","last_clean":"2024-03-26T03:12:03.352228+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:12:03.352228+0000","last_undegraded":"2024-03-26T03:12:03.352228+0000","last_fullsized":"2024-03-26T03:12:03.352228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:38.225092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:11:38.225092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:46.994914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":355,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.353543+0000","last_change":"2024-03-26T03:10:50.548723+0000","last_active":"2024-03-26T03:12:03.353543+0000","last_peered":"2024-03-26T03:12:03.353543+0000","last_clean":"2024-03-26T03:12:03.353543+0000","last_became_active":"2024-03-26T03:10:44.085377+0000","last_became_peered":"2024-03-26T03:10:44.085377+0000","last_unstale":"2024-03-26T03:12:03.353543+0000","last_undegraded":"2024-03-26T03:12:03.353543+0000","last_fullsized":"2024-03-26T03:12:03.353543+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.548591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.548591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.314046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":218,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.360016+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:12:03.360016+0000","last_peered":"2024-03-26T03:12:03.360016+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:12:03.360016+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361644+0000","last_change":"2024-03-26T03:11:13.331187+0000","last_active":"2024-03-26T03:12:03.361644+0000","last_peered":"2024-03-26T03:12:03.361644+0000","last_clean":"2024-03-26T03:12:03.361644+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:12:03.361644+0000","last_undegraded":"2024-03-26T03:12:03.361644+0000","last_fullsized":"2024-03-26T03:12:03.361644+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.331065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.331065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:16.385611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":462,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.354191+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:12:03.354191+0000","last_peered":"2024-03-26T03:12:03.354191+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:12:03.354191+0000","last_undegraded":"2024-03-26T03:12:03.354191+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.910020+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":472,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359084+0000","last_change":"2024-03-26T03:11:45.964602+0000","last_active":"2024-03-26T03:12:03.359084+0000","last_peered":"2024-03-26T03:12:03.359084+0000","last_clean":"2024-03-26T03:12:03.359084+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:12:03.359084+0000","last_undegraded":"2024-03-26T03:12:03.359084+0000","last_fullsized":"2024-03-26T03:12:03.359084+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:45.964365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:45.964365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:11.924887+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":585,"reported_epoch":278,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:03.357570+0000","last_change":"2024-03-26T03:10:49.282505+0000","last_active":"2024-03-26T03:12:03.357570+0000","last_peered":"2024-03-26T03:12:03.357570+0000","last_clean":"2024-03-26T03:12:03.357570+0000","last_became_active":"2024-03-26T03:10:42.404630+0000","last_became_peered":"2024-03-26T03:10:42.404630+0000","last_unstale":"2024-03-26T03:12:03.357570+0000","last_undegraded":"2024-03-26T03:12:03.357570+0000","last_fullsized":"2024-03-26T03:12:03.357570+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.282258+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.282258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":417,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359996+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:12:03.359996+0000","last_peered":"2024-03-26T03:12:03.359996+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:12:03.359996+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":478,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.354093+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:12:03.354093+0000","last_peered":"2024-03-26T03:12:03.354093+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:12:03.354093+0000","last_undegraded":"2024-03-26T03:12:03.354093+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.910058+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":659,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.355244+0000","last_change":"2024-03-26T03:11:38.754437+0000","last_active":"2024-03-26T03:12:03.355244+0000","last_peered":"2024-03-26T03:12:03.355244+0000","last_clean":"2024-03-26T03:12:03.355244+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:12:03.355244+0000","last_undegraded":"2024-03-26T03:12:03.355244+0000","last_fullsized":"2024-03-26T03:12:03.355244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:11:38.754331+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:11:38.754331+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:43.119696+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":459,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358746+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:12:03.358746+0000","last_peered":"2024-03-26T03:12:03.358746+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:12:03.358746+0000","last_undegraded":"2024-03-26T03:12:03.358746+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":758,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.357816+0000","last_change":"2024-03-26T03:10:49.610796+0000","last_active":"2024-03-26T03:12:03.357816+0000","last_peered":"2024-03-26T03:12:03.357816+0000","last_clean":"2024-03-26T03:12:03.357816+0000","last_became_active":"2024-03-26T03:10:44.085632+0000","last_became_peered":"2024-03-26T03:10:44.085632+0000","last_unstale":"2024-03-26T03:12:03.357816+0000","last_undegraded":"2024-03-26T03:12:03.357816+0000","last_fullsized":"2024-03-26T03:12:03.357816+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.610676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.610676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:13.952827+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":312,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359580+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:12:03.359580+0000","last_peered":"2024-03-26T03:12:03.359580+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:12:03.359580+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":463,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358567+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:12:03.358567+0000","last_peered":"2024-03-26T03:12:03.358567+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:12:03.358567+0000","last_undegraded":"2024-03-26T03:12:03.358567+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":265,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359289+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:12:03.359289+0000","last_peered":"2024-03-26T03:12:03.359289+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:12:03.359289+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":472,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.364331+0000","last_change":"2024-03-26T03:11:07.697991+0000","last_active":"2024-03-26T03:12:03.364331+0000","last_peered":"2024-03-26T03:12:03.364331+0000","last_clean":"2024-03-26T03:12:03.364331+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:12:03.364331+0000","last_undegraded":"2024-03-26T03:12:03.364331+0000","last_fullsized":"2024-03-26T03:12:03.364331+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:11:07.697855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.697855+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:32.059699+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":517,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359185+0000","last_change":"2024-03-26T03:11:46.227621+0000","last_active":"2024-03-26T03:12:03.359185+0000","last_peered":"2024-03-26T03:12:03.359185+0000","last_clean":"2024-03-26T03:12:03.359185+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:12:03.359185+0000","last_undegraded":"2024-03-26T03:12:03.359185+0000","last_fullsized":"2024-03-26T03:12:03.359185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.227482+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:11.154098+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":471,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.361329+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:12:03.361329+0000","last_peered":"2024-03-26T03:12:03.361329+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:12:03.361329+0000","last_undegraded":"2024-03-26T03:12:03.361329+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359027+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:12:03.359027+0000","last_peered":"2024-03-26T03:12:03.359027+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:12:03.359027+0000","last_undegraded":"2024-03-26T03:12:03.359027+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":483,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.357544+0000","last_change":"2024-03-26T03:11:13.575433+0000","last_active":"2024-03-26T03:12:03.357544+0000","last_peered":"2024-03-26T03:12:03.357544+0000","last_clean":"2024-03-26T03:12:03.357544+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:12:03.357544+0000","last_undegraded":"2024-03-26T03:12:03.357544+0000","last_fullsized":"2024-03-26T03:12:03.357544+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.575348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.575348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.771654+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":518,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.363699+0000","last_change":"2024-03-26T03:11:42.830091+0000","last_active":"2024-03-26T03:12:03.363699+0000","last_peered":"2024-03-26T03:12:03.363699+0000","last_clean":"2024-03-26T03:12:03.363699+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:12:03.363699+0000","last_undegraded":"2024-03-26T03:12:03.363699+0000","last_fullsized":"2024-03-26T03:12:03.363699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:11:42.829912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.829912+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:08.247461+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362607+0000","last_change":"2024-03-26T03:10:54.105195+0000","last_active":"2024-03-26T03:12:03.362607+0000","last_peered":"2024-03-26T03:12:03.362607+0000","last_clean":"2024-03-26T03:12:03.362607+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:12:03.362607+0000","last_undegraded":"2024-03-26T03:12:03.362607+0000","last_fullsized":"2024-03-26T03:12:03.362607+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.105074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.105074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.061523+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":455,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359306+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:12:03.359306+0000","last_peered":"2024-03-26T03:12:03.359306+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:12:03.359306+0000","last_undegraded":"2024-03-26T03:12:03.359306+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236895+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358261+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:12:03.358261+0000","last_peered":"2024-03-26T03:12:03.358261+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:12:03.358261+0000","last_undegraded":"2024-03-26T03:12:03.358261+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":200,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.357692+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:12:03.357692+0000","last_peered":"2024-03-26T03:12:03.357692+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:12:03.357692+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":499,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.354474+0000","last_change":"2024-03-26T03:10:48.610870+0000","last_active":"2024-03-26T03:12:03.354474+0000","last_peered":"2024-03-26T03:12:03.354474+0000","last_clean":"2024-03-26T03:12:03.354474+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:12:03.354474+0000","last_undegraded":"2024-03-26T03:12:03.354474+0000","last_fullsized":"2024-03-26T03:12:03.354474+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:48.610770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:10:48.610770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:16.686703+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361548+0000","last_change":"2024-03-26T03:11:48.104641+0000","last_active":"2024-03-26T03:12:03.361548+0000","last_peered":"2024-03-26T03:12:03.361548+0000","last_clean":"2024-03-26T03:12:03.361548+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:12:03.361548+0000","last_undegraded":"2024-03-26T03:12:03.361548+0000","last_fullsized":"2024-03-26T03:12:03.361548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.104415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.104415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:48.998188+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":480,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358727+0000","last_change":"2024-03-26T03:11:01.548791+0000","last_active":"2024-03-26T03:12:03.358727+0000","last_peered":"2024-03-26T03:12:03.358727+0000","last_clean":"2024-03-26T03:12:03.358727+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:12:03.358727+0000","last_undegraded":"2024-03-26T03:12:03.358727+0000","last_fullsized":"2024-03-26T03:12:03.358727+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.548673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.548673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:25.289328+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":357,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.352051+0000","last_change":"2024-03-26T03:10:51.073689+0000","last_active":"2024-03-26T03:12:03.352051+0000","last_peered":"2024-03-26T03:12:03.352051+0000","last_clean":"2024-03-26T03:12:03.352051+0000","last_became_active":"2024-03-26T03:10:44.079530+0000","last_became_peered":"2024-03-26T03:10:44.079530+0000","last_unstale":"2024-03-26T03:12:03.352051+0000","last_undegraded":"2024-03-26T03:12:03.352051+0000","last_fullsized":"2024-03-26T03:12:03.352051+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.073552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.073552+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219339+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":404,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359667+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:12:03.359667+0000","last_peered":"2024-03-26T03:12:03.359667+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:12:03.359667+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361914+0000","last_change":"2024-03-26T03:11:42.129279+0000","last_active":"2024-03-26T03:12:03.361914+0000","last_peered":"2024-03-26T03:12:03.361914+0000","last_clean":"2024-03-26T03:12:03.361914+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:12:03.361914+0000","last_undegraded":"2024-03-26T03:12:03.361914+0000","last_fullsized":"2024-03-26T03:12:03.361914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:42.129153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.129153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.397417+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358343+0000","last_change":"2024-03-26T03:10:14.660979+0000","last_active":"2024-03-26T03:12:03.358343+0000","last_peered":"2024-03-26T03:12:03.358343+0000","last_clean":"2024-03-26T03:12:03.358343+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:12:03.358343+0000","last_undegraded":"2024-03-26T03:12:03.358343+0000","last_fullsized":"2024-03-26T03:12:03.358343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:14.660845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:10:14.660845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.357957+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:12:03.357957+0000","last_peered":"2024-03-26T03:12:03.357957+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:12:03.357957+0000","last_undegraded":"2024-03-26T03:12:03.357957+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":510,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.359758+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:12:03.359758+0000","last_peered":"2024-03-26T03:12:03.359758+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:12:03.359758+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":499,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.353588+0000","last_change":"2024-03-26T03:10:47.576491+0000","last_active":"2024-03-26T03:12:03.353588+0000","last_peered":"2024-03-26T03:12:03.353588+0000","last_clean":"2024-03-26T03:12:03.353588+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:12:03.353588+0000","last_undegraded":"2024-03-26T03:12:03.353588+0000","last_fullsized":"2024-03-26T03:12:03.353588+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:47.576286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.576286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:14.591656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":655,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362062+0000","last_change":"2024-03-26T03:10:51.299024+0000","last_active":"2024-03-26T03:12:03.362062+0000","last_peered":"2024-03-26T03:12:03.362062+0000","last_clean":"2024-03-26T03:12:03.362062+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:12:03.362062+0000","last_undegraded":"2024-03-26T03:12:03.362062+0000","last_fullsized":"2024-03-26T03:12:03.362062+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:10:51.298893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.298893+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:15.004130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"71'1","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359388+0000","last_change":"2024-03-26T03:10:57.591571+0000","last_active":"2024-03-26T03:12:03.359388+0000","last_peered":"2024-03-26T03:12:03.359388+0000","last_clean":"2024-03-26T03:12:03.359388+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:12:03.359388+0000","last_undegraded":"2024-03-26T03:12:03.359388+0000","last_fullsized":"2024-03-26T03:12:03.359388+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:10:57.591453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.591453+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":532,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362047+0000","last_change":"2024-03-26T03:11:40.118723+0000","last_active":"2024-03-26T03:12:03.362047+0000","last_peered":"2024-03-26T03:12:03.362047+0000","last_clean":"2024-03-26T03:12:03.362047+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:12:03.362047+0000","last_undegraded":"2024-03-26T03:12:03.362047+0000","last_fullsized":"2024-03-26T03:12:03.362047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:11:40.118572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.118572+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:56.770542+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":177,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.360387+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:12:03.360387+0000","last_peered":"2024-03-26T03:12:03.360387+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:12:03.360387+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":493,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359974+0000","last_change":"2024-03-26T03:11:49.540875+0000","last_active":"2024-03-26T03:12:03.359974+0000","last_peered":"2024-03-26T03:12:03.359974+0000","last_clean":"2024-03-26T03:12:03.359974+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:12:03.359974+0000","last_undegraded":"2024-03-26T03:12:03.359974+0000","last_fullsized":"2024-03-26T03:12:03.359974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.540763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.540763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:52.443627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":472,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362339+0000","last_change":"2024-03-26T03:10:57.204904+0000","last_active":"2024-03-26T03:12:03.362339+0000","last_peered":"2024-03-26T03:12:03.362339+0000","last_clean":"2024-03-26T03:12:03.362339+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:12:03.362339+0000","last_undegraded":"2024-03-26T03:12:03.362339+0000","last_fullsized":"2024-03-26T03:12:03.362339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:57.204763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.204763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.985937+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":510,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.352656+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:12:03.352656+0000","last_peered":"2024-03-26T03:12:03.352656+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:12:03.352656+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":500,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.360717+0000","last_change":"2024-03-26T03:11:16.283604+0000","last_active":"2024-03-26T03:12:03.360717+0000","last_peered":"2024-03-26T03:12:03.360717+0000","last_clean":"2024-03-26T03:12:03.360717+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:12:03.360717+0000","last_undegraded":"2024-03-26T03:12:03.360717+0000","last_fullsized":"2024-03-26T03:12:03.360717+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.283475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.283475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:45.688944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362877+0000","last_change":"2024-03-26T03:10:56.053025+0000","last_active":"2024-03-26T03:12:03.362877+0000","last_peered":"2024-03-26T03:12:03.362877+0000","last_clean":"2024-03-26T03:12:03.362877+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:12:03.362877+0000","last_undegraded":"2024-03-26T03:12:03.362877+0000","last_fullsized":"2024-03-26T03:12:03.362877+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:56.052911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:10:56.052911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:17.280129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":480,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.365193+0000","last_change":"2024-03-26T03:11:15.795645+0000","last_active":"2024-03-26T03:12:03.365193+0000","last_peered":"2024-03-26T03:12:03.365193+0000","last_clean":"2024-03-26T03:12:03.365193+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:12:03.365193+0000","last_undegraded":"2024-03-26T03:12:03.365193+0000","last_fullsized":"2024-03-26T03:12:03.365193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.795433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.795433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:32.594054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":232,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.362699+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:12:03.362699+0000","last_peered":"2024-03-26T03:12:03.362699+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:12:03.362699+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:07.236822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359195+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:12:03.359195+0000","last_peered":"2024-03-26T03:12:03.359195+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:12:03.359195+0000","last_undegraded":"2024-03-26T03:12:03.359195+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":454,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.364686+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:12:03.364686+0000","last_peered":"2024-03-26T03:12:03.364686+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:12:03.364686+0000","last_undegraded":"2024-03-26T03:12:03.364686+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.362112+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:12:03.362112+0000","last_peered":"2024-03-26T03:12:03.362112+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:12:03.362112+0000","last_undegraded":"2024-03-26T03:12:03.362112+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":521,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.362340+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:12:03.362340+0000","last_peered":"2024-03-26T03:12:03.362340+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:12:03.362340+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358828+0000","last_change":"2024-03-26T03:10:04.678447+0000","last_active":"2024-03-26T03:12:03.358828+0000","last_peered":"2024-03-26T03:12:03.358828+0000","last_clean":"2024-03-26T03:12:03.358828+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:12:03.358828+0000","last_undegraded":"2024-03-26T03:12:03.358828+0000","last_fullsized":"2024-03-26T03:12:03.358828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:04.678325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:10:04.678325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":491,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.353786+0000","last_change":"2024-03-26T03:11:16.659920+0000","last_active":"2024-03-26T03:12:03.353786+0000","last_peered":"2024-03-26T03:12:03.353786+0000","last_clean":"2024-03-26T03:12:03.353786+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:12:03.353786+0000","last_undegraded":"2024-03-26T03:12:03.353786+0000","last_fullsized":"2024-03-26T03:12:03.353786+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.659717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.659717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:28.148413+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":488,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358348+0000","last_change":"2024-03-26T03:11:44.673836+0000","last_active":"2024-03-26T03:12:03.358348+0000","last_peered":"2024-03-26T03:12:03.358348+0000","last_clean":"2024-03-26T03:12:03.358348+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:12:03.358348+0000","last_undegraded":"2024-03-26T03:12:03.358348+0000","last_fullsized":"2024-03-26T03:12:03.358348+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:44.673348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:44.673348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.606713+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":483,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.364364+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:12:03.364364+0000","last_peered":"2024-03-26T03:12:03.364364+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:12:03.364364+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:05.100102+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":492,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359222+0000","last_change":"2024-03-26T03:11:16.999880+0000","last_active":"2024-03-26T03:12:03.359222+0000","last_peered":"2024-03-26T03:12:03.359222+0000","last_clean":"2024-03-26T03:12:03.359222+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:12:03.359222+0000","last_undegraded":"2024-03-26T03:12:03.359222+0000","last_fullsized":"2024-03-26T03:12:03.359222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.999769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.999769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:22.404177+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":484,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.358589+0000","last_change":"2024-03-26T03:11:48.834864+0000","last_active":"2024-03-26T03:12:03.358589+0000","last_peered":"2024-03-26T03:12:03.358589+0000","last_clean":"2024-03-26T03:12:03.358589+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:12:03.358589+0000","last_undegraded":"2024-03-26T03:12:03.358589+0000","last_fullsized":"2024-03-26T03:12:03.358589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.834753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.834753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:51.805383+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":458,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.353602+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:12:03.353602+0000","last_peered":"2024-03-26T03:12:03.353602+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:12:03.353602+0000","last_undegraded":"2024-03-26T03:12:03.353602+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.910047+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":524,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.361816+0000","last_change":"2024-03-26T03:11:41.830501+0000","last_active":"2024-03-26T03:12:03.361816+0000","last_peered":"2024-03-26T03:12:03.361816+0000","last_clean":"2024-03-26T03:12:03.361816+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:12:03.361816+0000","last_undegraded":"2024-03-26T03:12:03.361816+0000","last_fullsized":"2024-03-26T03:12:03.361816+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:11:41.830357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.830357+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:58.510180+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.359409+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:12:03.359409+0000","last_peered":"2024-03-26T03:12:03.359409+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:12:03.359409+0000","last_undegraded":"2024-03-26T03:12:03.359409+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":476,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.356560+0000","last_change":"2024-03-26T03:10:59.074818+0000","last_active":"2024-03-26T03:12:03.356560+0000","last_peered":"2024-03-26T03:12:03.356560+0000","last_clean":"2024-03-26T03:12:03.356560+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:12:03.356560+0000","last_undegraded":"2024-03-26T03:12:03.356560+0000","last_fullsized":"2024-03-26T03:12:03.356560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:59.074364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:59.074364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:14.061608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":479,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362258+0000","last_change":"2024-03-26T03:11:46.936315+0000","last_active":"2024-03-26T03:12:03.362258+0000","last_peered":"2024-03-26T03:12:03.362258+0000","last_clean":"2024-03-26T03:12:03.362258+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:12:03.362258+0000","last_undegraded":"2024-03-26T03:12:03.362258+0000","last_fullsized":"2024-03-26T03:12:03.362258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.290695+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":280,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:03.351392+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:12:03.351392+0000","last_peered":"2024-03-26T03:12:03.351392+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:12:03.351392+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:04.219304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":501,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.359459+0000","last_change":"2024-03-26T03:11:51.538045+0000","last_active":"2024-03-26T03:12:03.359459+0000","last_peered":"2024-03-26T03:12:03.359459+0000","last_clean":"2024-03-26T03:12:03.359459+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:12:03.359459+0000","last_undegraded":"2024-03-26T03:12:03.359459+0000","last_fullsized":"2024-03-26T03:12:03.359459+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.537921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.537921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:21.217203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":483,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-03-26T03:12:03.362064+0000","last_change":"2024-03-26T03:10:52.646355+0000","last_active":"2024-03-26T03:12:03.362064+0000","last_peered":"2024-03-26T03:12:03.362064+0000","last_clean":"2024-03-26T03:12:03.362064+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:12:03.362064+0000","last_undegraded":"2024-03-26T03:12:03.362064+0000","last_fullsized":"2024-03-26T03:12:03.362064+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:52.646244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:52.646244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:21.583056+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":169,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-03-26T03:12:03.358847+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:12:03.358847+0000","last_peered":"2024-03-26T03:12:03.358847+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:12:03.358847+0000","last_undegraded":"2024-03-26T03:12:03.358847+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":249757720,"num_objects":3812,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":288786,"num_write_kb":9363073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":477245440,"data_stored":477233224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76269,"ondisk_log_size":76269,"up":40,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":608258,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":54,"acting":58,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":782,"num_read_kb":777,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1410,"num_read_kb":1135,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103331,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763344,"kb_used_data":118280,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92981808,"statfs":{"total":95995035648,"available":95213371392,"internally_reserved":0,"allocated":121118720,"data_stored":120482852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":660273610},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430375,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":431684,"kb_used_data":39548,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93313468,"statfs":{"total":95995035648,"available":95552991232,"internally_reserved":0,"allocated":40497152,"data_stored":39872930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":743608,"kb_used_data":104368,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":93001544,"statfs":{"total":95995035648,"available":95233581056,"internally_reserved":0,"allocated":106872832,"data_stored":106197734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738564,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642824,"kb_used_data":138240,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93102328,"statfs":{"total":95995035648,"available":95336783872,"internally_reserved":0,"allocated":141557760,"data_stored":140930185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411514,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451912,"kb_used_data":41728,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93293240,"statfs":{"total":95995035648,"available":95532277760,"internally_reserved":0,"allocated":42729472,"data_stored":42102755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986353,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":375832,"kb_used_data":37776,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93369320,"statfs":{"total":95995035648,"available":95610183680,"internally_reserved":0,"allocated":38682624,"data_stored":38038449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692013,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":729936,"kb_used_data":121160,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93015216,"statfs":{"total":95995035648,"available":95247581184,"internally_reserved":0,"allocated":124067840,"data_stored":123431976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134938624,"data_stored":134938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66326528,"data_stored":66322456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31653888,"data_stored":31653888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34340864,"data_stored":34340864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":89526272,"data_stored":89522200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35717120,"data_stored":35717120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":84742144,"data_stored":84738072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:12:11.859 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [6, 5, 7, 3, 2, 0] 2024-03-26T03:12:11.860 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '6', '5', '7', '3', '2', '0'] 2024-03-26T03:12:11.860 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 2.0 6 5 7 3 2 0 2024-03-26T03:12:11.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:11.956 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:11.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:12.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:12.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:12.182 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:12.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:12.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:12.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:12.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:12 smithi192 ceph-mon[33110]: from='client.15606 v1:172.21.15.66:0/291426949' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:12:12.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:12.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:12.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:12.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:12 smithi066 ceph-mon[29666]: from='client.15606 v1:172.21.15.66:0/291426949' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:12:12.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:12 smithi066 ceph-mon[37920]: from='client.15606 v1:172.21.15.66:0/291426949' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:12:12.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:12.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:12.825 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:12.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:12.935 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:12.955 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:12.970 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:12:12.970 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:12:12.984 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:12.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:13.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:13.102 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:12:13.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:13.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:13.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:13.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:13.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:13.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:13.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:13.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:13 smithi066 ceph-mon[29666]: pgmap v578: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 122 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.41k op/s; 1191/6507 objects degraded (18.303%); 1/6507 objects misplaced (0.015%) 2024-03-26T03:12:13.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:13 smithi066 ceph-mon[37920]: pgmap v578: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 122 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.41k op/s; 1191/6507 objects degraded (18.303%); 1/6507 objects misplaced (0.015%) 2024-03-26T03:12:13.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:13.632 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:13 smithi192 ceph-mon[33110]: pgmap v578: 168 pgs: 4 active+clean+remapped, 30 active+undersized+degraded, 38 active+undersized, 96 active+clean; 122 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.41k op/s; 1191/6507 objects degraded (18.303%); 1/6507 objects misplaced (0.015%) 2024-03-26T03:12:13.658 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:12:13.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.741 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:13.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:13.842 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:13.870 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:13.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:13.974 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:12:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:12:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:12:13.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:13.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:14.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:14.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:14.154 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:14.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:14.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:14.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:14.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:14.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.504 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:14.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:14.604 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 1191/6507 objects degraded (18.303%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:14.604 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 1191/6507 objects degraded (18.303%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:14.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:14.633 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:14.735 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 1191/6507 objects degraded (18.303%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:14.764 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:14.865 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:14.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:14.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:14.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:15.022 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:15.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:15.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:15.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:15.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:15.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:15.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:15.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:15.536 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:15.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:15 smithi066 ceph-mon[29666]: pgmap v579: 168 pgs: 4 active+clean+remapped, 29 active+undersized+degraded, 39 active+undersized, 96 active+clean; 110 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 1.30k op/s; 1003/5943 objects degraded (16.877%); 1/5943 objects misplaced (0.017%) 2024-03-26T03:12:15.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:15 smithi066 ceph-mon[37920]: pgmap v579: 168 pgs: 4 active+clean+remapped, 29 active+undersized+degraded, 39 active+undersized, 96 active+clean; 110 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 1.30k op/s; 1003/5943 objects degraded (16.877%); 1/5943 objects misplaced (0.017%) 2024-03-26T03:12:15.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:15.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:15 smithi192 ceph-mon[33110]: pgmap v579: 168 pgs: 4 active+clean+remapped, 29 active+undersized+degraded, 39 active+undersized, 96 active+clean; 110 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 B/s wr, 1.30k op/s; 1003/5943 objects degraded (16.877%); 1/5943 objects misplaced (0.017%) 2024-03-26T03:12:15.767 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:15.796 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:15.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:15.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:15.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:16.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:16.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:16.065 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:16.156 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:16.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:16.216 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-26T03:12:16.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-26T03:12:16.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:16.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:16.390 INFO:teuthology.orchestra.run.smithi066.stderr:set 2.0 pg_upmap_items mapping to [6->5,7->3,2->0] 2024-03-26T03:12:16.390 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:12:16.415 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:16.445 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:16.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:16.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:16.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:16.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3719147065' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [6, 5, 7, 3, 2, 0]}]: dispatch 2024-03-26T03:12:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3345699275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:12:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:16 smithi192 ceph-mon[33110]: from='client.35465 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:12:16.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:16.835 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:16.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3719147065' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [6, 5, 7, 3, 2, 0]}]: dispatch 2024-03-26T03:12:16.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3345699275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:12:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:16 smithi066 ceph-mon[37920]: from='client.35465 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:12:16.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3719147065' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [6, 5, 7, 3, 2, 0]}]: dispatch 2024-03-26T03:12:16.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3345699275' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:12:16.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:16 smithi066 ceph-mon[29666]: from='client.35465 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:12:16.936 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:16.964 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:16.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:17.065 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.095 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:17.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:17.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:17.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:17.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:17.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:17.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:17.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:17.582 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:17.611 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:17.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: pgmap v580: 168 pgs: 4 active+clean+remapped, 27 active+undersized+degraded, 41 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3719147065' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [6, 5, 7, 3, 2, 0]}]': finished 2024-03-26T03:12:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: from='client.35465 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:12:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: osdmap e279: 8 total, 7 up, 8 in 2024-03-26T03:12:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:12:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:17 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:12:17.743 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: pgmap v580: 168 pgs: 4 active+clean+remapped, 27 active+undersized+degraded, 41 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3719147065' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [6, 5, 7, 3, 2, 0]}]': finished 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: from='client.35465 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: osdmap e279: 8 total, 7 up, 8 in 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:12:17.819 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: pgmap v580: 168 pgs: 4 active+clean+remapped, 27 active+undersized+degraded, 41 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:17.820 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:17.820 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3719147065' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [6, 5, 7, 3, 2, 0]}]': finished 2024-03-26T03:12:17.820 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: from='client.35465 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:12:17.820 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: osdmap e279: 8 total, 7 up, 8 in 2024-03-26T03:12:17.820 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:12:17.820 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:17 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:12:17.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:17.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:17.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:17.974 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.005 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:18.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:18.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:18.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:18.235 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:18.264 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:18.366 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:18.397 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:18.403 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:18.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:18.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:18.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:18.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:18 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:12:18.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:18 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:12:18.653 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:18 smithi066 ceph-mon[29666]: osdmap e280: 8 total, 7 up, 8 in 2024-03-26T03:12:18.653 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:18 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:12:18.653 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:18 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:12:18.653 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:18 smithi066 ceph-mon[37920]: osdmap e280: 8 total, 7 up, 8 in 2024-03-26T03:12:18.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:18 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:12:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:18 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:12:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:18 smithi192 ceph-mon[33110]: osdmap e280: 8 total, 7 up, 8 in 2024-03-26T03:12:18.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:18.785 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:18.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:18.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:18.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:19.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:19.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:19.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:19.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:19.272 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:19.300 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:19.401 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:19.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:19.530 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:19.557 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:12:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:12:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: pgmap v583: 168 pgs: 4 active+clean+remapped, 27 active+undersized+degraded, 41 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 476 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: osdmap e281: 8 total, 7 up, 8 in 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 27 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[29666]: osdmap e282: 8 total, 7 up, 8 in 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:12:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: pgmap v583: 168 pgs: 4 active+clean+remapped, 27 active+undersized+degraded, 41 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 476 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: osdmap e281: 8 total, 7 up, 8 in 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 27 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:19.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:19 smithi066 ceph-mon[37920]: osdmap e282: 8 total, 7 up, 8 in 2024-03-26T03:12:19.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:19.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: pgmap v583: 168 pgs: 4 active+clean+remapped, 27 active+undersized+degraded, 41 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 476 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: osdmap e281: 8 total, 7 up, 8 in 2024-03-26T03:12:19.702 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:12:19.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:12:19.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:12:19.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:12:19.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 27 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:19.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-mon[33110]: osdmap e282: 8 total, 7 up, 8 in 2024-03-26T03:12:19.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:19.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:19.919 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:19.948 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:19.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:19.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:12:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:12:20.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:20.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:20.123 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:12:20.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:20.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:20.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:20.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:20.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:20.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:20.465 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:20.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:20.571 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:12:20.571 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:12:20.593 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:20.695 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:20.724 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:20.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:20.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:20.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:20.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:20.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:21.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:21.113 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:21.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:21.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:21.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:21.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:21.475 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:21.496 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:21.496 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:12:21.496 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:12:21.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[37920]: pgmap v586: 168 pgs: 5 active+clean+remapped, 23 active+undersized+degraded, 45 active+undersized, 95 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:21.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:21.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:12:21.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:12:21.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:21 smithi066 ceph-mon[29666]: pgmap v586: 168 pgs: 5 active+clean+remapped, 23 active+undersized+degraded, 45 active+undersized, 95 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:21.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:21.606 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.634 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:21.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:21.735 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:21 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:12:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:21 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:12:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:21 smithi192 ceph-mon[33110]: pgmap v586: 168 pgs: 5 active+clean+remapped, 23 active+undersized+degraded, 45 active+undersized, 95 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:21.762 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:21.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:21.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:21.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:21.934 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:12:21.994 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:22.021 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:22.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:22.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:22.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:22.279 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:22.381 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:22.410 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:22.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:12:22.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:12:22.487 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:12:22.488 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:12:22.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:22.516 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[29666]: 1.14 scrub starts 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[29666]: 1.14 scrub ok 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:12:22.517 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[37920]: 1.14 scrub starts 2024-03-26T03:12:22.518 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[37920]: 1.14 scrub ok 2024-03-26T03:12:22.518 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:12:22.518 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:22 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:12:22.546 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:22.561 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:22.578 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:12:22.598 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:22.603 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:12:22.628 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:12:22.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:22.651 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:12:22.673 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:12:22.676 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:22.694 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:12:22.716 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:12:22.717 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:12:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:22 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:12:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:22 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:12:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:22 smithi192 ceph-mon[33110]: 1.14 scrub starts 2024-03-26T03:12:22.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:22 smithi192 ceph-mon[33110]: 1.14 scrub ok 2024-03-26T03:12:22.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:22 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:12:22.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:22 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:12:22.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:22.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:22.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:22.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:22.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:23.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:23.057 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:23.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:23.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:23.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:23.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:23.410 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:23.433 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_2' removed 2024-03-26T03:12:23.450 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:23.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:23.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:23.681 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:23.709 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: pgmap v587: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 46 active+undersized, 95 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 870 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: 1.13 scrub starts 2024-03-26T03:12:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: 1.13 scrub ok 2024-03-26T03:12:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/639837898' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='client.35471 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/475871287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:12:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:23 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:23.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: pgmap v587: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 46 active+undersized, 95 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 870 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: 1.13 scrub starts 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: 1.13 scrub ok 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/639837898' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='client.35471 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/475871287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:12:23.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: pgmap v587: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 46 active+undersized, 95 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 870 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: 1.13 scrub starts 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: 1.13 scrub ok 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/639837898' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='client.35471 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/475871287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:12:23.789 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:23.812 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:23.839 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:23.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:23.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:23.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:24.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:24.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:12:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:12:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:12:24.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:24.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:24.229 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:12:24.230 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:12:24.230 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-26T03:12:24.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-26T03:12:24.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:24.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:24.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:24.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:24.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:24.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:12:24.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:12:24.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: from='client.35471 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:12:24.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: osdmap e283: 8 total, 7 up, 8 in 2024-03-26T03:12:24.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 22 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[29666]: osdmap e284: 8 total, 7 up, 8 in 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: from='client.35471 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: osdmap e283: 8 total, 7 up, 8 in 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:12:24.601 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:12:24.602 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:12:24.602 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:12:24.602 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:12:24.602 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:12:24.602 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 22 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:24.602 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:24 smithi066 ceph-mon[37920]: osdmap e284: 8 total, 7 up, 8 in 2024-03-26T03:12:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.603 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:24.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:12:24.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:12:24.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:24.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:12:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:12:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: from='client.35471 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:12:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: osdmap e283: 8 total, 7 up, 8 in 2024-03-26T03:12:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 22 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:24 smithi192 ceph-mon[33110]: osdmap e284: 8 total, 7 up, 8 in 2024-03-26T03:12:24.829 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.858 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:24.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:24.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:24.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:24.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:25.001 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:25.084 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:25.112 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:25.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:25.242 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:25.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:25.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:25.470 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:25.498 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:25.600 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: pgmap v590: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 696 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: 1.c scrub starts 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: 1.c scrub ok 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: pgmap v590: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 696 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: 1.c scrub starts 2024-03-26T03:12:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: 1.c scrub ok 2024-03-26T03:12:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:12:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:12:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:12:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:25 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:12:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.627 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:25.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:25.695 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:25.728 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: pgmap v590: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 696 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: 1.c scrub starts 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: 1.c scrub ok 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:12:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:25 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:12:25.755 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:25.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:25.884 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:25.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:25.986 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:26.013 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:26.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:26.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:26.244 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.273 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:26.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:26.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:26.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:26.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:26.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:26.627 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:12:26.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:26.659 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:26.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:26.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:26.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:26.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:26.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:27.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:27.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:27.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:27.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:27.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:27.306 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:27.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:27.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:27.443 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:12:27.538 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:27.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:27.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:27 smithi066 ceph-mon[29666]: pgmap v591: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:27.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/409056205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:27 smithi066 ceph-mon[37920]: pgmap v591: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:27.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/409056205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:27.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:27.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:27 smithi192 ceph-mon[33110]: pgmap v591: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/409056205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:27.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:27.826 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:27.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:27.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:27.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:28.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:28.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:28.191 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:28.219 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:28.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:28.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:28.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:28.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:28.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:28.580 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:28.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:12:28.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:12:28.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:12:28.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/409056205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:28.588 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[29666]: osdmap e285: 8 total, 7 up, 8 in 2024-03-26T03:12:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:12:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:12:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:12:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/409056205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:28.588 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:28 smithi066 ceph-mon[37920]: osdmap e285: 8 total, 7 up, 8 in 2024-03-26T03:12:28.609 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:28.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:28 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:12:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:28 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:12:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:28 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:12:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/409056205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:28 smithi192 ceph-mon[33110]: osdmap e285: 8 total, 7 up, 8 in 2024-03-26T03:12:28.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:28.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:28.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:28.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:28.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:28.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:29.100 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:29.129 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:29.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:29.301 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:29.306 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:29.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:29.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:29.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: pgmap v593: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[29666]: osdmap e286: 8 total, 7 up, 8 in 2024-03-26T03:12:29.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: pgmap v593: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:29.663 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:12:29.663 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:12:29.663 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:12:29.663 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:12:29.663 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:29.663 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:29 smithi066 ceph-mon[37920]: osdmap e286: 8 total, 7 up, 8 in 2024-03-26T03:12:29.663 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:29.691 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: pgmap v593: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 114/690 objects degraded (16.522%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:29.709 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-mon[33110]: osdmap e286: 8 total, 7 up, 8 in 2024-03-26T03:12:29.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:29.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:29.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:29.950 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:29.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:29.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:12:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:12:30.052 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:30.079 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:30.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:30.180 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:30.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:30.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:30.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:30.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:30.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:12:30.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:12:30.496 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/541181528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:12:30.497 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:30 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/541181528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:12:30.530 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:12:30.530 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:12:30.530 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 286, 2024-03-26T03:12:30.530 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:12:30.530 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:12:30.530 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:12:29.086726+0000", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T03:06:19.254688+0000", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T02:58:26.140451+0000", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5799936, 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 18, 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:12:30.531 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 9, 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:12:30.532 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 21, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 21, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 21, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 21, 2024-03-26T03:12:30.533 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 21, 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "254", 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "252", 2024-03-26T03:12:30.534 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:12:30.535 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:12:30.536 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:12:30.537 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2699999809265137, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2699999809265137, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:12:30.538 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:12:30.539 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:12:30.540 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:12:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:12:30.542 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.543 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:12:30.544 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:12:30.545 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:12:30.546 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:12:30.547 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:12:30.548 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:12:30.549 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:12:30.550 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:12:30.551 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:12:30.552 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:12:30.553 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:12:30.554 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.555 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2400000095367432, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2400000095367432, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:12:30.556 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:12:30.557 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:12:30.558 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:12:30.559 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:12:30.560 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:12:30.561 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:12:30.562 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:12:30.563 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:12:30.564 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:12:30.565 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:12:30.566 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:12:30.567 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.94957733154296875, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.568 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.569 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:12:30.570 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 280, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.573 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.574 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.575 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.576 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:12:30.577 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.580 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 0, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 26, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 172, 2024-03-26T03:12:30.581 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 198, 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.582 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.583 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2366374823 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.584 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/2366374823", 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/2366374823", 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/2366374823", 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/2366374823", 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "exists" 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:12:30.585 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.586 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.587 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:12:30.588 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.589 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.590 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:12:30.591 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.594 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:12:30.595 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.596 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.597 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 257, 2024-03-26T03:12:30.598 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.599 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.600 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:12:30.601 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:29.133643+0000", 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:12:01.602008+0000", 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.603 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:50.281295+0000", 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:05:36.091229+0000", 2024-03-26T03:12:30.604 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 197 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:52.802839+0000", 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.605 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:05.177898+0000", 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.606 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:07.662058+0000", 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:11:09.431840+0000", 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:12:30.607 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [ 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.f", 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.0", 2024-03-26T03:12:30.608 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 3 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.3", 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 7, 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.f", 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 4, 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 7 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 0, 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 1 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.610 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:12:30.611 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.612 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [ 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.c", 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: 6, 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: 5 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.e", 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:12:30.613 INFO:teuthology.orchestra.run.smithi066.stdout: 7, 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: 0 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.11", 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: 5, 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: 2 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.13", 2024-03-26T03:12:30.614 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: 6 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "2.0", 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: 0, 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:12:30.615 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:12:30.616 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:12:30.617 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:12:30.618 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:12:30.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:30.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:12:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:12:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:12:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:12:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:12:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:12:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:12:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:12:30.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:30 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/541181528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:12:30.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:30.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:30.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:30.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:30.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:31.007 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:31.036 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:31.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:31.165 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:31.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:31.294 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:31.351 INFO:tasks.thrashosds.thrasher:j is {'epoch': 286, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:12:29.086726+0000', 'last_up_change': '2024-03-26T03:06:19.254688+0000', 'last_in_change': '2024-03-26T02:58:26.140451+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '252', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9495773315429688, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 172, 'down_at': 198, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 2366374823}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 2366374823}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 2366374823}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 2366374823}]}, 'public_addr': '172.21.15.66:6813/2366374823', 'cluster_addr': '172.21.15.66:6814/2366374823', 'heartbeat_back_addr': '172.21.15.66:6816/2366374823', 'heartbeat_front_addr': '172.21.15.66:6815/2366374823', 'state': ['exists']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:29.133643+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:12:01.602008+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:50.281295+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:05:36.091229+0000', 'dead_epoch': 197}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:52.802839+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:05.177898+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:07.662058+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:11:09.431840+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [4, 1, 7]}], 'pg_upmap_items': [{'pgid': '2.0', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '2.f', 'mappings': [{'from': 4, 'to': 7}, {'from': 0, 'to': 1}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.c', 'osds': [0, 6, 5]}, {'pgid': '1.e', 'osds': [4, 7, 0]}, {'pgid': '1.11', 'osds': [0, 5, 2]}, {'pgid': '1.13', 'osds': [1, 4, 6]}, {'pgid': '2.0', 'osds': [1, 0, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:12:31.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:12:31.395 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:31.423 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:31.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:31.554 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:31.604 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:12:31.604 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:12:31.605 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:12:31.605 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:12:31.605 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: pgmap v595: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: osdmap e287: 8 total, 7 up, 8 in 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:12:31.605 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: 1.e scrub starts 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[29666]: 1.e scrub ok 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: pgmap v595: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: osdmap e287: 8 total, 7 up, 8 in 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: 1.e scrub starts 2024-03-26T03:12:31.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:31 smithi066 ceph-mon[37920]: 1.e scrub ok 2024-03-26T03:12:31.655 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:31.683 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: pgmap v595: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: osdmap e287: 8 total, 7 up, 8 in 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: 1.e scrub starts 2024-03-26T03:12:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:31 smithi192 ceph-mon[33110]: 1.e scrub ok 2024-03-26T03:12:31.784 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:31.812 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:31.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:31.941 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:31.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:32.042 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:32.070 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:32.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:32.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:32.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:32.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:32.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:32.460 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:32.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:32.578 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:32.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:32.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:32.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:32.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:32.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:32.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:32.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:32.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:33.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:33.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:33.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:33.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:33.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:33.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:33.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:33.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:33.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:33.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:33.725 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:12:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:12:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: pgmap v597: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: 1.b scrub starts 2024-03-26T03:12:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: 1.b scrub ok 2024-03-26T03:12:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:12:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:12:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:12:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:33 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:12:33.754 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:12:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:12:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: pgmap v597: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: 1.b scrub starts 2024-03-26T03:12:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: 1.b scrub ok 2024-03-26T03:12:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: pgmap v597: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: 1.b scrub starts 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: 1.b scrub ok 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:12:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:12:33.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:12:33.856 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:33.883 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:33.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:33.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:34.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:12:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:12:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:12:34.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:34.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:34.164 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:34.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:34.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:34.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:34.400 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:34.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:34.529 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:34.631 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:34.658 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:34 smithi192 ceph-mon[33110]: osdmap e288: 8 total, 7 up, 8 in 2024-03-26T03:12:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:34 smithi192 ceph-mon[33110]: 6.1c deep-scrub starts 2024-03-26T03:12:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:34 smithi192 ceph-mon[33110]: 6.1c deep-scrub ok 2024-03-26T03:12:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3612940420' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T03:12:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:34 smithi192 ceph-mon[33110]: from='client.35495 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T03:12:34.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:34.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:34.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[37920]: osdmap e288: 8 total, 7 up, 8 in 2024-03-26T03:12:34.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[37920]: 6.1c deep-scrub starts 2024-03-26T03:12:34.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[37920]: 6.1c deep-scrub ok 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3612940420' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[37920]: from='client.35495 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[29666]: osdmap e288: 8 total, 7 up, 8 in 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[29666]: 6.1c deep-scrub starts 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[29666]: 6.1c deep-scrub ok 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3612940420' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T03:12:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:34 smithi066 ceph-mon[29666]: from='client.35495 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T03:12:34.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:34.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:34.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:34.938 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:12:35.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:35.044 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:35.108 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_3' created 2024-03-26T03:12:35.146 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:35.176 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:35.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:35.287 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:12:35.312 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:12:35.314 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:35.333 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:12:35.354 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:12:35.375 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:12:35.395 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:12:35.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:35.417 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:12:35.417 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:12:35.417 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:12:35.417 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:12:35.417 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:12:35.417 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:12:35.418 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:12:35.418 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:12:35.418 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:12:35.444 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:35.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:35.574 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:35.676 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.705 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:35.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: pgmap v599: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: from='client.35495 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: osdmap e289: 8 total, 7 up, 8 in 2024-03-26T03:12:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:35 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/770786787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:35.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:35.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:35.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:12:35.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:12:35.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: pgmap v599: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: from='client.35495 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: osdmap e289: 8 total, 7 up, 8 in 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/770786787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:12:35.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: pgmap v599: 152 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: from='client.35495 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: osdmap e289: 8 total, 7 up, 8 in 2024-03-26T03:12:35.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:35 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/770786787' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:35.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:35.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:35.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:36.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:36.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:36.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:36.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:36.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:36.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:36.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:36.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:36.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:36.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:36.710 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:36 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:12:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:36 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:12:36.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:36 smithi192 ceph-mon[33110]: osdmap e290: 8 total, 7 up, 8 in 2024-03-26T03:12:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.742 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:36.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:36.742 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:36 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:12:36.742 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:36 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:12:36.742 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:36 smithi066 ceph-mon[29666]: osdmap e290: 8 total, 7 up, 8 in 2024-03-26T03:12:36.742 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:36 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:12:36.743 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:36 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:12:36.743 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:36 smithi066 ceph-mon[37920]: osdmap e290: 8 total, 7 up, 8 in 2024-03-26T03:12:36.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-26T03:12:36.843 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:36.880 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:36.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:36.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:37.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:37.112 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:37.140 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:37.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:37.269 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:37.371 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.399 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:37.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:37.500 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.531 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:37.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:37.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:37.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: 1.5 deep-scrub starts 2024-03-26T03:12:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: 1.5 deep-scrub ok 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: pgmap v602: 168 pgs: 3 creating+peering, 13 unknown, 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 879 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:12:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:37 smithi192 ceph-mon[33110]: osdmap e291: 8 total, 7 up, 8 in 2024-03-26T03:12:37.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: 1.5 deep-scrub starts 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: 1.5 deep-scrub ok 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: pgmap v602: 168 pgs: 3 creating+peering, 13 unknown, 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 879 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:12:37.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[29666]: osdmap e291: 8 total, 7 up, 8 in 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: 1.5 deep-scrub starts 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: 1.5 deep-scrub ok 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: pgmap v602: 168 pgs: 3 creating+peering, 13 unknown, 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 879 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:12:37.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:12:37.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:37 smithi066 ceph-mon[37920]: osdmap e291: 8 total, 7 up, 8 in 2024-03-26T03:12:37.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:12:37.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:12:37.891 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:37.919 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:37.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:38.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:38.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:38.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:38.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:38.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:38.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:38.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:38.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:38.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:38.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:38.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:12:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:12:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[29666]: 4.e deep-scrub starts 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[29666]: 4.e deep-scrub ok 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[37920]: 4.e deep-scrub starts 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[37920]: 4.e deep-scrub ok 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:12:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:38 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:12:38.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:38.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:38 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:12:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:38 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:12:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:38 smithi192 ceph-mon[33110]: 4.e deep-scrub starts 2024-03-26T03:12:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:38 smithi192 ceph-mon[33110]: 4.e deep-scrub ok 2024-03-26T03:12:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:38 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:12:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:38 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:12:38.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:38.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:38.926 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:38.953 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:38.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:39.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:39.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:39.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:39.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:39.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:39.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:39.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:39.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:39.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:39.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:39.605 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:12:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:12:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: pgmap v604: 168 pgs: 3 creating+peering, 13 unknown, 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:12:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:12:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: osdmap e292: 8 total, 7 up, 8 in 2024-03-26T03:12:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:39.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:12:39.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:12:39.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: pgmap v604: 168 pgs: 3 creating+peering, 13 unknown, 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:39.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:12:39.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:12:39.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: osdmap e292: 8 total, 7 up, 8 in 2024-03-26T03:12:39.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:39.613 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:39.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:39.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:39.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:12:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: pgmap v604: 168 pgs: 3 creating+peering, 13 unknown, 5 active+clean+remapped, 22 active+undersized+degraded, 38 active+undersized, 87 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: osdmap e292: 8 total, 7 up, 8 in 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:39.793 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:39.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:39.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:39.879 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:12:39.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:39.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:39.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:40.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:40.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:40.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:40.241 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:12:40.244 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:12:40.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:40.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:40.378 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:40.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:40.489 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:12:40.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1544330028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[29666]: from='client.35507 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1544330028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:12:40.586 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:40 smithi066 ceph-mon[37920]: from='client.35507 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:12:40.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:40.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:40.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:40 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:12:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:40 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:12:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:40 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1544330028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:12:40.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:40 smithi192 ceph-mon[33110]: from='client.35507 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:12:40.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:40.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:40.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:40.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:40.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:41.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:41.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:41.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:41.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:41.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:41.345 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2024-03-26T03:12:39.091051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614282,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2247,"num_read_kb":1970,"num_write":919,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":54528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3719,"ondisk_log_size":3719,"up":399,"acting":404,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":462,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6299928,"kb_used_data":2474240,"kb_used_omap":12,"kb_used_meta":3825587,"kb_avail":743661288,"statfs":{"total":767960285184,"available":761509158912,"internally_reserved":0,"allocated":2533621760,"data_stored":2528388808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3917401648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.824678"},"pg_stats":[{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":289,"state":"creating+peering","last_fresh":"2024-03-26T03:12:35.101464+0000","last_change":"2024-03-26T03:12:35.101464+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.101464+0000","last_undegraded":"2024-03-26T03:12:35.101464+0000","last_fullsized":"2024-03-26T03:12:35.101464+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":377,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139259+0000","last_change":"2024-03-26T03:10:50.602175+0000","last_active":"2024-03-26T03:12:34.139259+0000","last_peered":"2024-03-26T03:12:34.139259+0000","last_clean":"2024-03-26T03:12:34.139259+0000","last_became_active":"2024-03-26T03:10:44.090472+0000","last_became_peered":"2024-03-26T03:10:44.090472+0000","last_unstale":"2024-03-26T03:12:34.139259+0000","last_undegraded":"2024-03-26T03:12:34.139259+0000","last_fullsized":"2024-03-26T03:12:34.139259+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.602062+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.602062+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":678,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138455+0000","last_change":"2024-03-26T03:10:55.601026+0000","last_active":"2024-03-26T03:12:34.138455+0000","last_peered":"2024-03-26T03:12:34.138455+0000","last_clean":"2024-03-26T03:12:34.138455+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:12:34.138455+0000","last_undegraded":"2024-03-26T03:12:34.138455+0000","last_fullsized":"2024-03-26T03:12:34.138455+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:10:55.600815+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.600815+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":591,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.634296+0000","last_change":"2024-03-26T03:11:39.721484+0000","last_active":"2024-03-26T03:12:33.634296+0000","last_peered":"2024-03-26T03:12:33.634296+0000","last_clean":"2024-03-26T03:12:33.634296+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:12:33.634296+0000","last_undegraded":"2024-03-26T03:12:33.634296+0000","last_fullsized":"2024-03-26T03:12:33.634296+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:11:39.721355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:39.721355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:06.244718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":492,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.461780+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:12:33.461780+0000","last_peered":"2024-03-26T03:12:33.461780+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:12:33.461780+0000","last_undegraded":"2024-03-26T03:12:33.461780+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:33.900338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":321,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210172+0000","last_change":"2024-03-26T03:12:27.221535+0000","last_active":"2024-03-26T03:12:34.210172+0000","last_peered":"2024-03-26T03:12:34.210172+0000","last_clean":"2024-03-26T03:12:34.210172+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:12:34.210172+0000","last_undegraded":"2024-03-26T03:12:34.210172+0000","last_fullsized":"2024-03-26T03:12:34.210172+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:27.221439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:27.221439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:28.393425+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":382,"reported_epoch":288,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:33.460676+0000","last_change":"2024-03-26T03:12:31.058456+0000","last_active":"2024-03-26T03:12:33.460676+0000","last_peered":"2024-03-26T03:12:33.460676+0000","last_clean":"2024-03-26T03:12:33.460676+0000","last_became_active":"2024-03-26T03:10:44.082186+0000","last_became_peered":"2024-03-26T03:10:44.082186+0000","last_unstale":"2024-03-26T03:12:33.460676+0000","last_undegraded":"2024-03-26T03:12:33.460676+0000","last_fullsized":"2024-03-26T03:12:33.460676+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:31.058290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:31.058290+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:32.950088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":561,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.106359+0000","last_change":"2024-03-26T03:12:30.521625+0000","last_active":"2024-03-26T03:12:35.106359+0000","last_peered":"2024-03-26T03:12:35.106359+0000","last_clean":"2024-03-26T03:12:35.106359+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:12:35.106359+0000","last_undegraded":"2024-03-26T03:12:35.106359+0000","last_fullsized":"2024-03-26T03:12:35.106359+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:12:30.521522+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.521522+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:31.997671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":492,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.299895+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:12:34.299895+0000","last_peered":"2024-03-26T03:12:34.299895+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:12:34.299895+0000","last_undegraded":"2024-03-26T03:12:34.299895+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":476,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.460897+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:12:33.460897+0000","last_peered":"2024-03-26T03:12:33.460897+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:12:33.460897+0000","last_undegraded":"2024-03-26T03:12:33.460897+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:33.900310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":503,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.105095+0000","last_change":"2024-03-26T03:11:43.664485+0000","last_active":"2024-03-26T03:12:35.105095+0000","last_peered":"2024-03-26T03:12:35.105095+0000","last_clean":"2024-03-26T03:12:35.105095+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:12:35.105095+0000","last_undegraded":"2024-03-26T03:12:35.105095+0000","last_fullsized":"2024-03-26T03:12:35.105095+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:43.664357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:11:43.664357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.152927+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":289,"state":"creating+peering","last_fresh":"2024-03-26T03:12:35.101460+0000","last_change":"2024-03-26T03:12:35.101460+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.101460+0000","last_undegraded":"2024-03-26T03:12:35.101460+0000","last_fullsized":"2024-03-26T03:12:35.101460+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":585,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210848+0000","last_change":"2024-03-26T03:10:49.362862+0000","last_active":"2024-03-26T03:12:34.210848+0000","last_peered":"2024-03-26T03:12:34.210848+0000","last_clean":"2024-03-26T03:12:34.210848+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:12:34.210848+0000","last_undegraded":"2024-03-26T03:12:34.210848+0000","last_fullsized":"2024-03-26T03:12:34.210848+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.362737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.362737+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":604,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210376+0000","last_change":"2024-03-26T03:11:12.308404+0000","last_active":"2024-03-26T03:12:34.210376+0000","last_peered":"2024-03-26T03:12:34.210376+0000","last_clean":"2024-03-26T03:12:34.210376+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:12:34.210376+0000","last_undegraded":"2024-03-26T03:12:34.210376+0000","last_fullsized":"2024-03-26T03:12:34.210376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:11:12.308255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:12.308255+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":529,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461332+0000","last_change":"2024-03-26T03:11:51.169305+0000","last_active":"2024-03-26T03:12:33.461332+0000","last_peered":"2024-03-26T03:12:33.461332+0000","last_clean":"2024-03-26T03:12:33.461332+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:12:33.461332+0000","last_undegraded":"2024-03-26T03:12:33.461332+0000","last_fullsized":"2024-03-26T03:12:33.461332+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.169187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.169187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:15.669587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461174+0000","last_change":"2024-03-26T03:11:46.249753+0000","last_active":"2024-03-26T03:12:33.461174+0000","last_peered":"2024-03-26T03:12:33.461174+0000","last_clean":"2024-03-26T03:12:33.461174+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:12:33.461174+0000","last_undegraded":"2024-03-26T03:12:33.461174+0000","last_fullsized":"2024-03-26T03:12:33.461174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:46.249597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.249597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:08.763925+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":472,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.137995+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:12:34.137995+0000","last_peered":"2024-03-26T03:12:34.137995+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:12:34.137995+0000","last_undegraded":"2024-03-26T03:12:34.137995+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":573,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460891+0000","last_change":"2024-03-26T03:12:30.037525+0000","last_active":"2024-03-26T03:12:33.460891+0000","last_peered":"2024-03-26T03:12:33.460891+0000","last_clean":"2024-03-26T03:12:33.460891+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:12:33.460891+0000","last_undegraded":"2024-03-26T03:12:33.460891+0000","last_fullsized":"2024-03-26T03:12:33.460891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:12:30.037403+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.037403+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.381867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138590+0000","last_change":"2024-03-26T03:11:41.872736+0000","last_active":"2024-03-26T03:12:34.138590+0000","last_peered":"2024-03-26T03:12:34.138590+0000","last_clean":"2024-03-26T03:12:34.138590+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:12:34.138590+0000","last_undegraded":"2024-03-26T03:12:34.138590+0000","last_fullsized":"2024-03-26T03:12:34.138590+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:41.872620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.872620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:50.631490+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":505,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299528+0000","last_change":"2024-03-26T03:12:23.358746+0000","last_active":"2024-03-26T03:12:34.299528+0000","last_peered":"2024-03-26T03:12:34.299528+0000","last_clean":"2024-03-26T03:12:34.299528+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:12:34.299528+0000","last_undegraded":"2024-03-26T03:12:34.299528+0000","last_fullsized":"2024-03-26T03:12:34.299528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:23.358630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.358630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:37.326581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":480,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.462193+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:12:33.462193+0000","last_peered":"2024-03-26T03:12:33.462193+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:12:33.462193+0000","last_undegraded":"2024-03-26T03:12:33.462193+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.914104+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":561,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.460683+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:12:33.460683+0000","last_peered":"2024-03-26T03:12:33.460683+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:12:33.460683+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:33.900331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":752,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.137998+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:12:34.137998+0000","last_peered":"2024-03-26T03:12:34.137998+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:12:34.137998+0000","last_undegraded":"2024-03-26T03:12:34.137998+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":517,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.460377+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:12:33.460377+0000","last_peered":"2024-03-26T03:12:33.460377+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:12:33.460377+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033851+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":484,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.299335+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:12:34.299335+0000","last_peered":"2024-03-26T03:12:34.299335+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:12:34.299335+0000","last_undegraded":"2024-03-26T03:12:34.299335+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":694,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461747+0000","last_change":"2024-03-26T03:12:28.903130+0000","last_active":"2024-03-26T03:12:33.461747+0000","last_peered":"2024-03-26T03:12:33.461747+0000","last_clean":"2024-03-26T03:12:33.461747+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:12:33.461747+0000","last_undegraded":"2024-03-26T03:12:33.461747+0000","last_fullsized":"2024-03-26T03:12:33.461747+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:12:28.903038+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.903038+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:42.184864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":499,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461047+0000","last_change":"2024-03-26T03:12:25.078525+0000","last_active":"2024-03-26T03:12:33.461047+0000","last_peered":"2024-03-26T03:12:33.461047+0000","last_clean":"2024-03-26T03:12:33.461047+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:12:33.461047+0000","last_undegraded":"2024-03-26T03:12:33.461047+0000","last_fullsized":"2024-03-26T03:12:33.461047+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:25.078426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:25.078426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:37.085323+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":599,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461427+0000","last_change":"2024-03-26T03:11:52.158484+0000","last_active":"2024-03-26T03:12:33.461427+0000","last_peered":"2024-03-26T03:12:33.461427+0000","last_clean":"2024-03-26T03:12:33.461427+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:12:33.461427+0000","last_undegraded":"2024-03-26T03:12:33.461427+0000","last_fullsized":"2024-03-26T03:12:33.461427+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:11:52.158163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:11:52.158163+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:07.642900+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":531,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.299677+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:12:34.299677+0000","last_peered":"2024-03-26T03:12:34.299677+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:12:34.299677+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":484,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.298744+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:12:34.298744+0000","last_peered":"2024-03-26T03:12:34.298744+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:12:34.298744+0000","last_undegraded":"2024-03-26T03:12:34.298744+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":676,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461047+0000","last_change":"2024-03-26T03:11:40.792357+0000","last_active":"2024-03-26T03:12:33.461047+0000","last_peered":"2024-03-26T03:12:33.461047+0000","last_clean":"2024-03-26T03:12:33.461047+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:12:33.461047+0000","last_undegraded":"2024-03-26T03:12:33.461047+0000","last_fullsized":"2024-03-26T03:12:33.461047+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:11:40.791990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.791990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:50.059248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":305,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299480+0000","last_change":"2024-03-26T03:11:01.239298+0000","last_active":"2024-03-26T03:12:34.299480+0000","last_peered":"2024-03-26T03:12:34.299480+0000","last_clean":"2024-03-26T03:12:34.299480+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:12:34.299480+0000","last_undegraded":"2024-03-26T03:12:34.299480+0000","last_fullsized":"2024-03-26T03:12:34.299480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.239012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.239012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":611,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.298932+0000","last_change":"2024-03-26T03:12:32.478786+0000","last_active":"2024-03-26T03:12:34.298932+0000","last_peered":"2024-03-26T03:12:34.298932+0000","last_clean":"2024-03-26T03:12:34.298932+0000","last_became_active":"2024-03-26T03:10:44.079636+0000","last_became_peered":"2024-03-26T03:10:44.079636+0000","last_unstale":"2024-03-26T03:12:34.298932+0000","last_undegraded":"2024-03-26T03:12:34.298932+0000","last_fullsized":"2024-03-26T03:12:34.298932+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:32.478665+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:32.478665+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:55.357668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"231'25","reported_seq":551,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460798+0000","last_change":"2024-03-26T03:11:06.580747+0000","last_active":"2024-03-26T03:12:33.460798+0000","last_peered":"2024-03-26T03:12:33.460798+0000","last_clean":"2024-03-26T03:12:33.460798+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:12:33.460798+0000","last_undegraded":"2024-03-26T03:12:33.460798+0000","last_fullsized":"2024-03-26T03:12:33.460798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:11:06.580543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:06.580543+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138200+0000","last_change":"2024-03-26T03:11:00.578809+0000","last_active":"2024-03-26T03:12:34.138200+0000","last_peered":"2024-03-26T03:12:34.138200+0000","last_clean":"2024-03-26T03:12:34.138200+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:12:34.138200+0000","last_undegraded":"2024-03-26T03:12:34.138200+0000","last_fullsized":"2024-03-26T03:12:34.138200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:00.578704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:00.578704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":311,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461047+0000","last_change":"2024-03-26T03:11:48.965867+0000","last_active":"2024-03-26T03:12:33.461047+0000","last_peered":"2024-03-26T03:12:33.461047+0000","last_clean":"2024-03-26T03:12:33.461047+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:12:33.461047+0000","last_undegraded":"2024-03-26T03:12:33.461047+0000","last_fullsized":"2024-03-26T03:12:33.461047+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.965732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.965732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:12.010040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.210446+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:12:34.210446+0000","last_peered":"2024-03-26T03:12:34.210446+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:12:34.210446+0000","last_undegraded":"2024-03-26T03:12:34.210446+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:37.225029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":355,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.298787+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:12:34.298787+0000","last_peered":"2024-03-26T03:12:34.298787+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:12:34.298787+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":570,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.459751+0000","last_change":"2024-03-26T03:12:22.041053+0000","last_active":"2024-03-26T03:12:33.459751+0000","last_peered":"2024-03-26T03:12:33.459751+0000","last_clean":"2024-03-26T03:12:33.459751+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:12:33.459751+0000","last_undegraded":"2024-03-26T03:12:33.459751+0000","last_fullsized":"2024-03-26T03:12:33.459751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:12:22.040942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.040942+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:29.332833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":521,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.211014+0000","last_change":"2024-03-26T03:12:32.227477+0000","last_active":"2024-03-26T03:12:34.211014+0000","last_peered":"2024-03-26T03:12:34.211014+0000","last_clean":"2024-03-26T03:12:34.211014+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:12:34.211014+0000","last_undegraded":"2024-03-26T03:12:34.211014+0000","last_fullsized":"2024-03-26T03:12:34.211014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:32.227383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:12:32.227383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:47.486616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":501,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139137+0000","last_change":"2024-03-26T03:11:49.921518+0000","last_active":"2024-03-26T03:12:34.139137+0000","last_peered":"2024-03-26T03:12:34.139137+0000","last_clean":"2024-03-26T03:12:34.139137+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:12:34.139137+0000","last_undegraded":"2024-03-26T03:12:34.139137+0000","last_fullsized":"2024-03-26T03:12:34.139137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.921407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:05.885611+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":671,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.138899+0000","last_change":"2024-03-26T03:10:39.370375+0000","last_active":"2024-03-26T03:12:34.138899+0000","last_peered":"2024-03-26T03:12:34.138899+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:10:39.370375+0000","last_became_peered":"2024-03-26T03:10:39.370375+0000","last_unstale":"2024-03-26T03:12:34.138899+0000","last_undegraded":"2024-03-26T03:12:34.138899+0000","last_fullsized":"2024-03-26T03:10:39.363411+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016823+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":475,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.459866+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:12:33.459866+0000","last_peered":"2024-03-26T03:12:33.459866+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:12:33.459866+0000","last_undegraded":"2024-03-26T03:12:33.459866+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":316,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.138453+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:12:34.138453+0000","last_peered":"2024-03-26T03:12:34.138453+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:12:34.138453+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":515,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299532+0000","last_change":"2024-03-26T03:11:41.297116+0000","last_active":"2024-03-26T03:12:34.299532+0000","last_peered":"2024-03-26T03:12:34.299532+0000","last_clean":"2024-03-26T03:12:34.299532+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:12:34.299532+0000","last_undegraded":"2024-03-26T03:12:34.299532+0000","last_fullsized":"2024-03-26T03:12:34.299532+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:11:41.296291+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.296291+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:51.235662+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":471,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.460215+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:12:33.460215+0000","last_peered":"2024-03-26T03:12:33.460215+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:12:33.460215+0000","last_undegraded":"2024-03-26T03:12:33.460215+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":596,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460457+0000","last_change":"2024-03-26T03:10:51.609357+0000","last_active":"2024-03-26T03:12:33.460457+0000","last_peered":"2024-03-26T03:12:33.460457+0000","last_clean":"2024-03-26T03:12:33.460457+0000","last_became_active":"2024-03-26T03:10:39.368150+0000","last_became_peered":"2024-03-26T03:10:39.368150+0000","last_unstale":"2024-03-26T03:12:33.460457+0000","last_undegraded":"2024-03-26T03:12:33.460457+0000","last_fullsized":"2024-03-26T03:12:33.460457+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:51.609183+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.609183+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.211766+0000","last_change":"2024-03-26T03:12:34.211787+0000","last_active":"2024-03-26T03:12:34.211766+0000","last_peered":"2024-03-26T03:12:34.211766+0000","last_clean":"2024-03-26T03:12:34.211766+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:12:34.211766+0000","last_undegraded":"2024-03-26T03:12:34.211766+0000","last_fullsized":"2024-03-26T03:12:34.211766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.211706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.211706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.510026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":582,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.106195+0000","last_change":"2024-03-26T03:11:47.624747+0000","last_active":"2024-03-26T03:12:35.106195+0000","last_peered":"2024-03-26T03:12:35.106195+0000","last_clean":"2024-03-26T03:12:35.106195+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:12:35.106195+0000","last_undegraded":"2024-03-26T03:12:35.106195+0000","last_fullsized":"2024-03-26T03:12:35.106195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:11:47.624655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.624655+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:17.291201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":505,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139709+0000","last_change":"2024-03-26T03:10:54.596765+0000","last_active":"2024-03-26T03:12:34.139709+0000","last_peered":"2024-03-26T03:12:34.139709+0000","last_clean":"2024-03-26T03:12:34.139709+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:12:34.139709+0000","last_undegraded":"2024-03-26T03:12:34.139709+0000","last_fullsized":"2024-03-26T03:12:34.139709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.596513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.596513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":504,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.298932+0000","last_change":"2024-03-26T03:11:16.222874+0000","last_active":"2024-03-26T03:12:34.298932+0000","last_peered":"2024-03-26T03:12:34.298932+0000","last_clean":"2024-03-26T03:12:34.298932+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:12:34.298932+0000","last_undegraded":"2024-03-26T03:12:34.298932+0000","last_fullsized":"2024-03-26T03:12:34.298932+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.222744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.635224+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":570,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.210623+0000","last_change":"2024-03-26T03:10:39.372465+0000","last_active":"2024-03-26T03:12:34.210623+0000","last_peered":"2024-03-26T03:12:34.210623+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:10:39.372465+0000","last_became_peered":"2024-03-26T03:10:39.372465+0000","last_unstale":"2024-03-26T03:12:34.210623+0000","last_undegraded":"2024-03-26T03:12:34.210623+0000","last_fullsized":"2024-03-26T03:10:39.360974+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.264519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":289,"state":"creating+peering","last_fresh":"2024-03-26T03:12:35.101729+0000","last_change":"2024-03-26T03:12:35.101729+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.101729+0000","last_undegraded":"2024-03-26T03:12:35.101729+0000","last_fullsized":"2024-03-26T03:12:35.101729+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.635254+0000","last_change":"2024-03-26T03:11:13.563915+0000","last_active":"2024-03-26T03:12:33.635254+0000","last_peered":"2024-03-26T03:12:33.635254+0000","last_clean":"2024-03-26T03:12:33.635254+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:12:33.635254+0000","last_undegraded":"2024-03-26T03:12:33.635254+0000","last_fullsized":"2024-03-26T03:12:33.635254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.563699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.563699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.093265+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":561,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.105061+0000","last_change":"2024-03-26T03:11:48.575617+0000","last_active":"2024-03-26T03:12:35.105061+0000","last_peered":"2024-03-26T03:12:35.105061+0000","last_clean":"2024-03-26T03:12:35.105061+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:12:35.105061+0000","last_undegraded":"2024-03-26T03:12:35.105061+0000","last_fullsized":"2024-03-26T03:12:35.105061+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.575462+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:48.657881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":35,"reported_epoch":288,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:34.461856+0000","last_change":"2024-03-26T03:12:34.461962+0000","last_active":"2024-03-26T03:12:34.461856+0000","last_peered":"2024-03-26T03:12:34.461856+0000","last_clean":"2024-03-26T03:12:34.461856+0000","last_became_active":"2024-03-26T03:12:18.638374+0000","last_became_peered":"2024-03-26T03:12:18.638374+0000","last_unstale":"2024-03-26T03:12:34.461856+0000","last_undegraded":"2024-03-26T03:12:34.461856+0000","last_fullsized":"2024-03-26T03:12:34.461856+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.461667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.461667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:57.139669+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,7],"avail_no_missing":["1","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":501,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.104492+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:12:35.104492+0000","last_peered":"2024-03-26T03:12:35.104492+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:12:35.104492+0000","last_undegraded":"2024-03-26T03:12:35.104492+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517627+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":480,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.299588+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:12:34.299588+0000","last_peered":"2024-03-26T03:12:34.299588+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:12:34.299588+0000","last_undegraded":"2024-03-26T03:12:34.299588+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":604,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299071+0000","last_change":"2024-03-26T03:12:29.490041+0000","last_active":"2024-03-26T03:12:34.299071+0000","last_peered":"2024-03-26T03:12:34.299071+0000","last_clean":"2024-03-26T03:12:34.299071+0000","last_became_active":"2024-03-26T03:10:39.379213+0000","last_became_peered":"2024-03-26T03:10:39.379213+0000","last_unstale":"2024-03-26T03:12:34.299071+0000","last_undegraded":"2024-03-26T03:12:34.299071+0000","last_fullsized":"2024-03-26T03:12:34.299071+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:29.489954+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:29.489954+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:47.301969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":505,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139218+0000","last_change":"2024-03-26T03:11:47.792917+0000","last_active":"2024-03-26T03:12:34.139218+0000","last_peered":"2024-03-26T03:12:34.139218+0000","last_clean":"2024-03-26T03:12:34.139218+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:12:34.139218+0000","last_undegraded":"2024-03-26T03:12:34.139218+0000","last_fullsized":"2024-03-26T03:12:34.139218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:47.792700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.792700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:54.932511+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"231'28","reported_seq":721,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210730+0000","last_change":"2024-03-26T03:12:20.180262+0000","last_active":"2024-03-26T03:12:34.210730+0000","last_peered":"2024-03-26T03:12:34.210730+0000","last_clean":"2024-03-26T03:12:34.210730+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:12:34.210730+0000","last_undegraded":"2024-03-26T03:12:34.210730+0000","last_fullsized":"2024-03-26T03:12:34.210730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:12:20.180126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:12:20.180126+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:41.479448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":771,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138087+0000","last_change":"2024-03-26T03:10:47.568899+0000","last_active":"2024-03-26T03:12:34.138087+0000","last_peered":"2024-03-26T03:12:34.138087+0000","last_clean":"2024-03-26T03:12:34.138087+0000","last_became_active":"2024-03-26T03:10:39.378619+0000","last_became_peered":"2024-03-26T03:10:39.378619+0000","last_unstale":"2024-03-26T03:12:34.138087+0000","last_undegraded":"2024-03-26T03:12:34.138087+0000","last_fullsized":"2024-03-26T03:12:34.138087+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:47.568755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.568755+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":542,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.634548+0000","last_change":"2024-03-26T03:12:24.846998+0000","last_active":"2024-03-26T03:12:33.634548+0000","last_peered":"2024-03-26T03:12:33.634548+0000","last_clean":"2024-03-26T03:12:33.634548+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:12:33.634548+0000","last_undegraded":"2024-03-26T03:12:33.634548+0000","last_fullsized":"2024-03-26T03:12:33.634548+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:24.846895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:12:24.846895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:51.542935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":215,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.634684+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:12:33.634684+0000","last_peered":"2024-03-26T03:12:33.634684+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:12:33.634684+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":479,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.635266+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:12:33.635266+0000","last_peered":"2024-03-26T03:12:33.635266+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:12:33.635266+0000","last_undegraded":"2024-03-26T03:12:33.635266+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.462042+0000","last_change":"2024-03-26T03:11:15.024122+0000","last_active":"2024-03-26T03:12:33.462042+0000","last_peered":"2024-03-26T03:12:33.462042+0000","last_clean":"2024-03-26T03:12:33.462042+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:12:33.462042+0000","last_undegraded":"2024-03-26T03:12:33.462042+0000","last_fullsized":"2024-03-26T03:12:33.462042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.024002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.024002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:34.521104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":554,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.300282+0000","last_change":"2024-03-26T03:11:11.116895+0000","last_active":"2024-03-26T03:12:34.300282+0000","last_peered":"2024-03-26T03:12:34.300282+0000","last_clean":"2024-03-26T03:12:34.300282+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:12:34.300282+0000","last_undegraded":"2024-03-26T03:12:34.300282+0000","last_fullsized":"2024-03-26T03:12:34.300282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:11:11.116642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:11.116642+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.643762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":492,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.461201+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:12:33.461201+0000","last_peered":"2024-03-26T03:12:33.461201+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:12:33.461201+0000","last_undegraded":"2024-03-26T03:12:33.461201+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:33.900355+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1338,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299257+0000","last_change":"2024-03-26T03:11:48.252850+0000","last_active":"2024-03-26T03:12:34.299257+0000","last_peered":"2024-03-26T03:12:34.299257+0000","last_clean":"2024-03-26T03:12:34.299257+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:12:34.299257+0000","last_undegraded":"2024-03-26T03:12:34.299257+0000","last_fullsized":"2024-03-26T03:12:34.299257+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:11:48.252717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.252717+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:03.591867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":807,"num_read_kb":802,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":480,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.139555+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:12:34.139555+0000","last_peered":"2024-03-26T03:12:34.139555+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:12:34.139555+0000","last_undegraded":"2024-03-26T03:12:34.139555+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016936+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":778,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138549+0000","last_change":"2024-03-26T03:10:46.593971+0000","last_active":"2024-03-26T03:12:34.138549+0000","last_peered":"2024-03-26T03:12:34.138549+0000","last_clean":"2024-03-26T03:12:34.138549+0000","last_became_active":"2024-03-26T03:10:39.379191+0000","last_became_peered":"2024-03-26T03:10:39.379191+0000","last_unstale":"2024-03-26T03:12:34.138549+0000","last_undegraded":"2024-03-26T03:12:34.138549+0000","last_fullsized":"2024-03-26T03:12:34.138549+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:46.593847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:46.593847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":507,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.105034+0000","last_change":"2024-03-26T03:11:40.584646+0000","last_active":"2024-03-26T03:12:35.105034+0000","last_peered":"2024-03-26T03:12:35.105034+0000","last_clean":"2024-03-26T03:12:35.105034+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:12:35.105034+0000","last_undegraded":"2024-03-26T03:12:35.105034+0000","last_fullsized":"2024-03-26T03:12:35.105034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:40.584536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.584536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:49.955813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"231'58","reported_seq":742,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.298954+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:12:34.298954+0000","last_peered":"2024-03-26T03:12:34.298954+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:12:34.298954+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":314,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299284+0000","last_change":"2024-03-26T03:12:30.465375+0000","last_active":"2024-03-26T03:12:34.299284+0000","last_peered":"2024-03-26T03:12:34.299284+0000","last_clean":"2024-03-26T03:12:34.299284+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:12:34.299284+0000","last_undegraded":"2024-03-26T03:12:34.299284+0000","last_fullsized":"2024-03-26T03:12:34.299284+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:30.465292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.465292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:44.382195+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.461085+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:12:33.461085+0000","last_peered":"2024-03-26T03:12:33.461085+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:12:33.461085+0000","last_undegraded":"2024-03-26T03:12:33.461085+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":615,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.104393+0000","last_change":"2024-03-26T03:12:21.520856+0000","last_active":"2024-03-26T03:12:35.104393+0000","last_peered":"2024-03-26T03:12:35.104393+0000","last_clean":"2024-03-26T03:12:35.104393+0000","last_became_active":"2024-03-26T03:10:39.376563+0000","last_became_peered":"2024-03-26T03:10:39.376563+0000","last_unstale":"2024-03-26T03:12:35.104393+0000","last_undegraded":"2024-03-26T03:12:35.104393+0000","last_fullsized":"2024-03-26T03:12:35.104393+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:21.520759+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:12:21.520759+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:28.569089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":682,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.106643+0000","last_change":"2024-03-26T03:11:14.528793+0000","last_active":"2024-03-26T03:12:35.106643+0000","last_peered":"2024-03-26T03:12:35.106643+0000","last_clean":"2024-03-26T03:12:35.106643+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:12:35.106643+0000","last_undegraded":"2024-03-26T03:12:35.106643+0000","last_fullsized":"2024-03-26T03:12:35.106643+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:11:14.528661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:14.528661+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:38.412124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":560,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.300000+0000","last_change":"2024-03-26T03:12:28.539186+0000","last_active":"2024-03-26T03:12:34.300000+0000","last_peered":"2024-03-26T03:12:34.300000+0000","last_clean":"2024-03-26T03:12:34.300000+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:12:34.300000+0000","last_undegraded":"2024-03-26T03:12:34.300000+0000","last_fullsized":"2024-03-26T03:12:34.300000+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:12:28.539083+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.539083+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.950841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299369+0000","last_change":"2024-03-26T03:11:10.075730+0000","last_active":"2024-03-26T03:12:34.299369+0000","last_peered":"2024-03-26T03:12:34.299369+0000","last_clean":"2024-03-26T03:12:34.299369+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:12:34.299369+0000","last_undegraded":"2024-03-26T03:12:34.299369+0000","last_fullsized":"2024-03-26T03:12:34.299369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:10.075607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:10.075607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":482,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.102697+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:12:35.102697+0000","last_peered":"2024-03-26T03:12:35.102697+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:12:35.102697+0000","last_undegraded":"2024-03-26T03:12:35.102697+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517585+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":570,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.210838+0000","last_change":"2024-03-26T03:10:39.371201+0000","last_active":"2024-03-26T03:12:34.210838+0000","last_peered":"2024-03-26T03:12:34.210838+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:10:39.371201+0000","last_became_peered":"2024-03-26T03:10:39.371201+0000","last_unstale":"2024-03-26T03:12:34.210838+0000","last_undegraded":"2024-03-26T03:12:34.210838+0000","last_fullsized":"2024-03-26T03:10:39.361046+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.264533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:12:35.089656+0000","last_change":"2024-03-26T03:12:35.089656+0000","last_active":"2024-03-26T03:12:35.089656+0000","last_peered":"2024-03-26T03:12:35.089656+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:12:35.089656+0000","last_undegraded":"2024-03-26T03:12:35.089656+0000","last_fullsized":"2024-03-26T03:12:35.089656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":385,"reported_epoch":289,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:35.104666+0000","last_change":"2024-03-26T03:12:24.618622+0000","last_active":"2024-03-26T03:12:35.104666+0000","last_peered":"2024-03-26T03:12:35.104666+0000","last_clean":"2024-03-26T03:12:35.104666+0000","last_became_active":"2024-03-26T03:10:44.081411+0000","last_became_peered":"2024-03-26T03:10:44.081411+0000","last_unstale":"2024-03-26T03:12:35.104666+0000","last_undegraded":"2024-03-26T03:12:35.104666+0000","last_fullsized":"2024-03-26T03:12:35.104666+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:24.618461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:24.618461+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:34.569655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":521,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460944+0000","last_change":"2024-03-26T03:12:18.908516+0000","last_active":"2024-03-26T03:12:33.460944+0000","last_peered":"2024-03-26T03:12:33.460944+0000","last_clean":"2024-03-26T03:12:33.460944+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:12:33.460944+0000","last_undegraded":"2024-03-26T03:12:33.460944+0000","last_fullsized":"2024-03-26T03:12:33.460944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.908406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.908406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:25.724804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":557,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.459815+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:12:33.459815+0000","last_peered":"2024-03-26T03:12:33.459815+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:12:33.459815+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":552,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461066+0000","last_change":"2024-03-26T03:12:33.067623+0000","last_active":"2024-03-26T03:12:33.461066+0000","last_peered":"2024-03-26T03:12:33.461066+0000","last_clean":"2024-03-26T03:12:33.461066+0000","last_became_active":"2024-03-26T03:10:08.555235+0000","last_became_peered":"2024-03-26T03:10:08.555235+0000","last_unstale":"2024-03-26T03:12:33.461066+0000","last_undegraded":"2024-03-26T03:12:33.461066+0000","last_fullsized":"2024-03-26T03:12:33.461066+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:12:33.067512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:33.067512+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.040390+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.138824+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:12:34.138824+0000","last_peered":"2024-03-26T03:12:34.138824+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:12:34.138824+0000","last_undegraded":"2024-03-26T03:12:34.138824+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":607,"reported_epoch":288,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:34.298923+0000","last_change":"2024-03-26T03:12:22.342630+0000","last_active":"2024-03-26T03:12:34.298923+0000","last_peered":"2024-03-26T03:12:34.298923+0000","last_clean":"2024-03-26T03:12:34.298923+0000","last_became_active":"2024-03-26T03:10:42.404423+0000","last_became_peered":"2024-03-26T03:10:42.404423+0000","last_unstale":"2024-03-26T03:12:34.298923+0000","last_undegraded":"2024-03-26T03:12:34.298923+0000","last_fullsized":"2024-03-26T03:12:34.298923+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:22.342472+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.342472+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.417985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":538,"reported_epoch":289,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:35.105172+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:12:35.105172+0000","last_peered":"2024-03-26T03:12:35.105172+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:12:35.105172+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517619+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":534,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460779+0000","last_change":"2024-03-26T03:12:23.943984+0000","last_active":"2024-03-26T03:12:33.460779+0000","last_peered":"2024-03-26T03:12:33.460779+0000","last_clean":"2024-03-26T03:12:33.460779+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:12:33.460779+0000","last_undegraded":"2024-03-26T03:12:33.460779+0000","last_fullsized":"2024-03-26T03:12:33.460779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:12:23.943885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.943885+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:44.438967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.299719+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:12:34.299719+0000","last_peered":"2024-03-26T03:12:34.299719+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:12:34.299719+0000","last_undegraded":"2024-03-26T03:12:34.299719+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":508,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461429+0000","last_change":"2024-03-26T03:11:38.225205+0000","last_active":"2024-03-26T03:12:33.461429+0000","last_peered":"2024-03-26T03:12:33.461429+0000","last_clean":"2024-03-26T03:12:33.461429+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:12:33.461429+0000","last_undegraded":"2024-03-26T03:12:33.461429+0000","last_fullsized":"2024-03-26T03:12:33.461429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:38.225092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:11:38.225092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:46.994914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":383,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.634340+0000","last_change":"2024-03-26T03:12:16.847864+0000","last_active":"2024-03-26T03:12:33.634340+0000","last_peered":"2024-03-26T03:12:33.634340+0000","last_clean":"2024-03-26T03:12:33.634340+0000","last_became_active":"2024-03-26T03:10:44.085377+0000","last_became_peered":"2024-03-26T03:10:44.085377+0000","last_unstale":"2024-03-26T03:12:33.634340+0000","last_undegraded":"2024-03-26T03:12:33.634340+0000","last_fullsized":"2024-03-26T03:12:33.634340+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:16.847722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:16.847722+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:33.297125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":238,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.139851+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:12:34.139851+0000","last_peered":"2024-03-26T03:12:34.139851+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:12:34.139851+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":520,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210211+0000","last_change":"2024-03-26T03:12:18.094777+0000","last_active":"2024-03-26T03:12:34.210211+0000","last_peered":"2024-03-26T03:12:34.210211+0000","last_clean":"2024-03-26T03:12:34.210211+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:12:34.210211+0000","last_undegraded":"2024-03-26T03:12:34.210211+0000","last_fullsized":"2024-03-26T03:12:34.210211+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.094677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.094677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:29.032092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":482,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.634934+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:12:33.634934+0000","last_peered":"2024-03-26T03:12:33.634934+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:12:33.634934+0000","last_undegraded":"2024-03-26T03:12:33.634934+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":492,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139365+0000","last_change":"2024-03-26T03:11:45.964602+0000","last_active":"2024-03-26T03:12:34.139365+0000","last_peered":"2024-03-26T03:12:34.139365+0000","last_clean":"2024-03-26T03:12:34.139365+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:12:34.139365+0000","last_undegraded":"2024-03-26T03:12:34.139365+0000","last_fullsized":"2024-03-26T03:12:34.139365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:45.964365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:45.964365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:11.924887+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":615,"reported_epoch":289,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:35.104147+0000","last_change":"2024-03-26T03:12:23.572625+0000","last_active":"2024-03-26T03:12:35.104147+0000","last_peered":"2024-03-26T03:12:35.104147+0000","last_clean":"2024-03-26T03:12:35.104147+0000","last_became_active":"2024-03-26T03:10:42.404630+0000","last_became_peered":"2024-03-26T03:10:42.404630+0000","last_unstale":"2024-03-26T03:12:35.104147+0000","last_undegraded":"2024-03-26T03:12:35.104147+0000","last_fullsized":"2024-03-26T03:12:35.104147+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:23.572446+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.572446+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:26.325433+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":437,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.299456+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:12:34.299456+0000","last_peered":"2024-03-26T03:12:34.299456+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:12:34.299456+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":498,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.634504+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:12:33.634504+0000","last_peered":"2024-03-26T03:12:33.634504+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:12:33.634504+0000","last_undegraded":"2024-03-26T03:12:33.634504+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":684,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.634910+0000","last_change":"2024-03-26T03:11:38.754437+0000","last_active":"2024-03-26T03:12:33.634910+0000","last_peered":"2024-03-26T03:12:33.634910+0000","last_clean":"2024-03-26T03:12:33.634910+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:12:33.634910+0000","last_undegraded":"2024-03-26T03:12:33.634910+0000","last_fullsized":"2024-03-26T03:12:33.634910+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:11:38.754331+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:11:38.754331+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:43.119696+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":481,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.104857+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:12:35.104857+0000","last_peered":"2024-03-26T03:12:35.104857+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:12:35.104857+0000","last_undegraded":"2024-03-26T03:12:35.104857+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517577+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":778,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.137886+0000","last_change":"2024-03-26T03:10:49.610796+0000","last_active":"2024-03-26T03:12:34.137886+0000","last_peered":"2024-03-26T03:12:34.137886+0000","last_clean":"2024-03-26T03:12:34.137886+0000","last_became_active":"2024-03-26T03:10:44.085632+0000","last_became_peered":"2024-03-26T03:10:44.085632+0000","last_unstale":"2024-03-26T03:12:34.137886+0000","last_undegraded":"2024-03-26T03:12:34.137886+0000","last_fullsized":"2024-03-26T03:12:34.137886+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.610676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.610676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":332,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.139497+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:12:34.139497+0000","last_peered":"2024-03-26T03:12:34.139497+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:12:34.139497+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":485,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.106306+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:12:35.106306+0000","last_peered":"2024-03-26T03:12:35.106306+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:12:35.106306+0000","last_undegraded":"2024-03-26T03:12:35.106306+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517569+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":290,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.139223+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:12:34.139223+0000","last_peered":"2024-03-26T03:12:34.139223+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:12:34.139223+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":492,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.461167+0000","last_change":"2024-03-26T03:11:07.697991+0000","last_active":"2024-03-26T03:12:33.461167+0000","last_peered":"2024-03-26T03:12:33.461167+0000","last_clean":"2024-03-26T03:12:33.461167+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:12:33.461167+0000","last_undegraded":"2024-03-26T03:12:33.461167+0000","last_fullsized":"2024-03-26T03:12:33.461167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:11:07.697855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.697855+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":537,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299340+0000","last_change":"2024-03-26T03:11:46.227621+0000","last_active":"2024-03-26T03:12:34.299340+0000","last_peered":"2024-03-26T03:12:34.299340+0000","last_clean":"2024-03-26T03:12:34.299340+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:12:34.299340+0000","last_undegraded":"2024-03-26T03:12:34.299340+0000","last_fullsized":"2024-03-26T03:12:34.299340+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.227482+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:11.154098+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":491,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.210492+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:12:34.210492+0000","last_peered":"2024-03-26T03:12:34.210492+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:12:34.210492+0000","last_undegraded":"2024-03-26T03:12:34.210492+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:37.225054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":191,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.106466+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:12:35.106466+0000","last_peered":"2024-03-26T03:12:35.106466+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:12:35.106466+0000","last_undegraded":"2024-03-26T03:12:35.106466+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":505,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.102255+0000","last_change":"2024-03-26T03:11:13.575433+0000","last_active":"2024-03-26T03:12:35.102255+0000","last_peered":"2024-03-26T03:12:35.102255+0000","last_clean":"2024-03-26T03:12:35.102255+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:12:35.102255+0000","last_undegraded":"2024-03-26T03:12:35.102255+0000","last_fullsized":"2024-03-26T03:12:35.102255+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:13.575348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:13.575348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:36.771654+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":538,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.459973+0000","last_change":"2024-03-26T03:11:42.830091+0000","last_active":"2024-03-26T03:12:33.459973+0000","last_peered":"2024-03-26T03:12:33.459973+0000","last_clean":"2024-03-26T03:12:33.459973+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:12:33.459973+0000","last_undegraded":"2024-03-26T03:12:33.459973+0000","last_fullsized":"2024-03-26T03:12:33.459973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:11:42.829912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.829912+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:08.247461+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":520,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299805+0000","last_change":"2024-03-26T03:12:18.319272+0000","last_active":"2024-03-26T03:12:34.299805+0000","last_peered":"2024-03-26T03:12:34.299805+0000","last_clean":"2024-03-26T03:12:34.299805+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:12:34.299805+0000","last_undegraded":"2024-03-26T03:12:34.299805+0000","last_fullsized":"2024-03-26T03:12:34.299805+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.319189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.319189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:24.605611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":475,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.300092+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:12:34.300092+0000","last_peered":"2024-03-26T03:12:34.300092+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:12:34.300092+0000","last_undegraded":"2024-03-26T03:12:34.300092+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":191,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.104672+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:12:35.104672+0000","last_peered":"2024-03-26T03:12:35.104672+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:12:35.104672+0000","last_undegraded":"2024-03-26T03:12:35.104672+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":220,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.299379+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:12:34.299379+0000","last_peered":"2024-03-26T03:12:34.299379+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:12:34.299379+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":527,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.635098+0000","last_change":"2024-03-26T03:12:23.825797+0000","last_active":"2024-03-26T03:12:33.635098+0000","last_peered":"2024-03-26T03:12:33.635098+0000","last_clean":"2024-03-26T03:12:33.635098+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:12:33.635098+0000","last_undegraded":"2024-03-26T03:12:33.635098+0000","last_fullsized":"2024-03-26T03:12:33.635098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:23.825709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.825709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:32.106344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210108+0000","last_change":"2024-03-26T03:11:48.104641+0000","last_active":"2024-03-26T03:12:34.210108+0000","last_peered":"2024-03-26T03:12:34.210108+0000","last_clean":"2024-03-26T03:12:34.210108+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:12:34.210108+0000","last_undegraded":"2024-03-26T03:12:34.210108+0000","last_fullsized":"2024-03-26T03:12:34.210108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.104415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.104415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:48.998188+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":500,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138647+0000","last_change":"2024-03-26T03:11:01.548791+0000","last_active":"2024-03-26T03:12:34.138647+0000","last_peered":"2024-03-26T03:12:34.138647+0000","last_clean":"2024-03-26T03:12:34.138647+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:12:34.138647+0000","last_undegraded":"2024-03-26T03:12:34.138647+0000","last_fullsized":"2024-03-26T03:12:34.138647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.548673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.548673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":385,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460949+0000","last_change":"2024-03-26T03:12:21.953709+0000","last_active":"2024-03-26T03:12:33.460949+0000","last_peered":"2024-03-26T03:12:33.460949+0000","last_clean":"2024-03-26T03:12:33.460949+0000","last_became_active":"2024-03-26T03:10:44.079530+0000","last_became_peered":"2024-03-26T03:10:44.079530+0000","last_unstale":"2024-03-26T03:12:33.460949+0000","last_undegraded":"2024-03-26T03:12:33.460949+0000","last_fullsized":"2024-03-26T03:12:33.460949+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:21.953552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:21.953552+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:39.750935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":424,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.138266+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:12:34.138266+0000","last_peered":"2024-03-26T03:12:34.138266+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:12:34.138266+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210979+0000","last_change":"2024-03-26T03:11:42.129279+0000","last_active":"2024-03-26T03:12:34.210979+0000","last_peered":"2024-03-26T03:12:34.210979+0000","last_clean":"2024-03-26T03:12:34.210979+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:12:34.210979+0000","last_undegraded":"2024-03-26T03:12:34.210979+0000","last_fullsized":"2024-03-26T03:12:34.210979+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:42.129153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.129153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.397417+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138330+0000","last_change":"2024-03-26T03:12:30.019328+0000","last_active":"2024-03-26T03:12:34.138330+0000","last_peered":"2024-03-26T03:12:34.138330+0000","last_clean":"2024-03-26T03:12:34.138330+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:12:34.138330+0000","last_undegraded":"2024-03-26T03:12:34.138330+0000","last_fullsized":"2024-03-26T03:12:34.138330+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:30.019226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.019226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:31.273674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":191,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.105882+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:12:35.105882+0000","last_peered":"2024-03-26T03:12:35.105882+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:12:35.105882+0000","last_undegraded":"2024-03-26T03:12:35.105882+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":532,"reported_epoch":289,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:35.105631+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:12:35.105631+0000","last_peered":"2024-03-26T03:12:35.105631+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:12:35.105631+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":527,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.634262+0000","last_change":"2024-03-26T03:12:22.805078+0000","last_active":"2024-03-26T03:12:33.634262+0000","last_peered":"2024-03-26T03:12:33.634262+0000","last_clean":"2024-03-26T03:12:33.634262+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:12:33.634262+0000","last_undegraded":"2024-03-26T03:12:33.634262+0000","last_fullsized":"2024-03-26T03:12:33.634262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:22.804999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.804999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:32.842763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":680,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.211045+0000","last_change":"2024-03-26T03:10:51.299024+0000","last_active":"2024-03-26T03:12:34.211045+0000","last_peered":"2024-03-26T03:12:34.211045+0000","last_clean":"2024-03-26T03:12:34.211045+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:12:34.211045+0000","last_undegraded":"2024-03-26T03:12:34.211045+0000","last_fullsized":"2024-03-26T03:12:34.211045+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:10:51.298893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:51.298893+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":513,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.142022+0000","last_change":"2024-03-26T03:12:34.142076+0000","last_active":"2024-03-26T03:12:34.142022+0000","last_peered":"2024-03-26T03:12:34.142022+0000","last_clean":"2024-03-26T03:12:34.142022+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:12:34.142022+0000","last_undegraded":"2024-03-26T03:12:34.142022+0000","last_fullsized":"2024-03-26T03:12:34.142022+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.141917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:35.118061+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":552,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.211075+0000","last_change":"2024-03-26T03:11:40.118723+0000","last_active":"2024-03-26T03:12:34.211075+0000","last_peered":"2024-03-26T03:12:34.211075+0000","last_clean":"2024-03-26T03:12:34.211075+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:12:34.211075+0000","last_undegraded":"2024-03-26T03:12:34.211075+0000","last_fullsized":"2024-03-26T03:12:34.211075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:11:40.118572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.118572+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:56.770542+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":197,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.209999+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:12:34.209999+0000","last_peered":"2024-03-26T03:12:34.209999+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:12:34.209999+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:37.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":515,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.105843+0000","last_change":"2024-03-26T03:11:49.540875+0000","last_active":"2024-03-26T03:12:35.105843+0000","last_peered":"2024-03-26T03:12:35.105843+0000","last_clean":"2024-03-26T03:12:35.105843+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:12:35.105843+0000","last_undegraded":"2024-03-26T03:12:35.105843+0000","last_fullsized":"2024-03-26T03:12:35.105843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.540763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.540763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:52.443627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":492,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.299600+0000","last_change":"2024-03-26T03:10:57.204904+0000","last_active":"2024-03-26T03:12:34.299600+0000","last_peered":"2024-03-26T03:12:34.299600+0000","last_clean":"2024-03-26T03:12:34.299600+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:12:34.299600+0000","last_undegraded":"2024-03-26T03:12:34.299600+0000","last_fullsized":"2024-03-26T03:12:34.299600+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:57.204763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:10:57.204763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":530,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.461590+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:12:33.461590+0000","last_peered":"2024-03-26T03:12:33.461590+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:12:33.461590+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:33.900322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":520,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.210442+0000","last_change":"2024-03-26T03:11:16.283604+0000","last_active":"2024-03-26T03:12:34.210442+0000","last_peered":"2024-03-26T03:12:34.210442+0000","last_clean":"2024-03-26T03:12:34.210442+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:12:34.210442+0000","last_undegraded":"2024-03-26T03:12:34.210442+0000","last_fullsized":"2024-03-26T03:12:34.210442+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.283475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.283475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:45.688944+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":504,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.300252+0000","last_change":"2024-03-26T03:10:56.053025+0000","last_active":"2024-03-26T03:12:34.300252+0000","last_peered":"2024-03-26T03:12:34.300252+0000","last_clean":"2024-03-26T03:12:34.300252+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:12:34.300252+0000","last_undegraded":"2024-03-26T03:12:34.300252+0000","last_fullsized":"2024-03-26T03:12:34.300252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:56.052911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:10:56.052911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":500,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.460884+0000","last_change":"2024-03-26T03:11:15.795645+0000","last_active":"2024-03-26T03:12:33.460884+0000","last_peered":"2024-03-26T03:12:33.460884+0000","last_clean":"2024-03-26T03:12:33.460884+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:12:33.460884+0000","last_undegraded":"2024-03-26T03:12:33.460884+0000","last_fullsized":"2024-03-26T03:12:33.460884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.795433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.795433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":252,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:34.299900+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:12:34.299900+0000","last_peered":"2024-03-26T03:12:34.299900+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:12:34.299900+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.138893+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:12:34.138893+0000","last_peered":"2024-03-26T03:12:34.138893+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:12:34.138893+0000","last_undegraded":"2024-03-26T03:12:34.138893+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":474,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.460305+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:12:33.460305+0000","last_peered":"2024-03-26T03:12:33.460305+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:12:33.460305+0000","last_undegraded":"2024-03-26T03:12:33.460305+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.211108+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:12:34.211108+0000","last_peered":"2024-03-26T03:12:34.211108+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:12:34.211108+0000","last_undegraded":"2024-03-26T03:12:34.211108+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:37.225065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":541,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.459753+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:12:33.459753+0000","last_peered":"2024-03-26T03:12:33.459753+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:12:33.459753+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033807+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139036+0000","last_change":"2024-03-26T03:12:18.931056+0000","last_active":"2024-03-26T03:12:34.139036+0000","last_peered":"2024-03-26T03:12:34.139036+0000","last_clean":"2024-03-26T03:12:34.139036+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:12:34.139036+0000","last_undegraded":"2024-03-26T03:12:34.139036+0000","last_fullsized":"2024-03-26T03:12:34.139036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.930944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.930944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:26.735550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":511,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.634548+0000","last_change":"2024-03-26T03:11:16.659920+0000","last_active":"2024-03-26T03:12:33.634548+0000","last_peered":"2024-03-26T03:12:33.634548+0000","last_clean":"2024-03-26T03:12:33.634548+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:12:33.634548+0000","last_undegraded":"2024-03-26T03:12:33.634548+0000","last_fullsized":"2024-03-26T03:12:33.634548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.659717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.659717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":510,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.102666+0000","last_change":"2024-03-26T03:11:44.673836+0000","last_active":"2024-03-26T03:12:35.102666+0000","last_peered":"2024-03-26T03:12:35.102666+0000","last_clean":"2024-03-26T03:12:35.102666+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:12:35.102666+0000","last_undegraded":"2024-03-26T03:12:35.102666+0000","last_fullsized":"2024-03-26T03:12:35.102666+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:44.673348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:44.673348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.606713+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":503,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.460170+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:12:33.460170+0000","last_peered":"2024-03-26T03:12:33.460170+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:12:33.460170+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.033777+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":512,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.139463+0000","last_change":"2024-03-26T03:11:16.999880+0000","last_active":"2024-03-26T03:12:34.139463+0000","last_peered":"2024-03-26T03:12:34.139463+0000","last_clean":"2024-03-26T03:12:34.139463+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:12:34.139463+0000","last_undegraded":"2024-03-26T03:12:34.139463+0000","last_fullsized":"2024-03-26T03:12:34.139463+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.999769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.999769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":504,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.138473+0000","last_change":"2024-03-26T03:11:48.834864+0000","last_active":"2024-03-26T03:12:34.138473+0000","last_peered":"2024-03-26T03:12:34.138473+0000","last_clean":"2024-03-26T03:12:34.138473+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:12:34.138473+0000","last_undegraded":"2024-03-26T03:12:34.138473+0000","last_fullsized":"2024-03-26T03:12:34.138473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.834753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.834753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:51.805383+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":478,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:33.634410+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:12:33.634410+0000","last_peered":"2024-03-26T03:12:33.634410+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:12:33.634410+0000","last_undegraded":"2024-03-26T03:12:33.634410+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":544,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.459488+0000","last_change":"2024-03-26T03:11:41.830501+0000","last_active":"2024-03-26T03:12:33.459488+0000","last_peered":"2024-03-26T03:12:33.459488+0000","last_clean":"2024-03-26T03:12:33.459488+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:12:33.459488+0000","last_undegraded":"2024-03-26T03:12:33.459488+0000","last_fullsized":"2024-03-26T03:12:33.459488+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:11:41.830357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.830357+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:58.510180+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":192,"reported_epoch":289,"state":"active+undersized","last_fresh":"2024-03-26T03:12:35.105385+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:12:35.105385+0000","last_peered":"2024-03-26T03:12:35.105385+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:12:35.105385+0000","last_undegraded":"2024-03-26T03:12:35.105385+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.517593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":504,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:34.298785+0000","last_change":"2024-03-26T03:12:17.280984+0000","last_active":"2024-03-26T03:12:34.298785+0000","last_peered":"2024-03-26T03:12:34.298785+0000","last_clean":"2024-03-26T03:12:34.298785+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:12:34.298785+0000","last_undegraded":"2024-03-26T03:12:34.298785+0000","last_fullsized":"2024-03-26T03:12:34.298785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:17.280891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:17.280891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:35.555586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":499,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.459916+0000","last_change":"2024-03-26T03:11:46.936315+0000","last_active":"2024-03-26T03:12:33.459916+0000","last_peered":"2024-03-26T03:12:33.459916+0000","last_clean":"2024-03-26T03:12:33.459916+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:12:33.459916+0000","last_undegraded":"2024-03-26T03:12:33.459916+0000","last_fullsized":"2024-03-26T03:12:33.459916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.290695+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":300,"reported_epoch":288,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:33.460925+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:12:33.460925+0000","last_peered":"2024-03-26T03:12:33.460925+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:12:33.460925+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:33.900363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":523,"reported_epoch":289,"state":"active+clean","last_fresh":"2024-03-26T03:12:35.104965+0000","last_change":"2024-03-26T03:11:51.538045+0000","last_active":"2024-03-26T03:12:35.104965+0000","last_peered":"2024-03-26T03:12:35.104965+0000","last_clean":"2024-03-26T03:12:35.104965+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:12:35.104965+0000","last_undegraded":"2024-03-26T03:12:35.104965+0000","last_fullsized":"2024-03-26T03:12:35.104965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.537921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.537921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:21.217203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":503,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-03-26T03:12:33.459662+0000","last_change":"2024-03-26T03:10:52.646355+0000","last_active":"2024-03-26T03:12:33.459662+0000","last_peered":"2024-03-26T03:12:33.459662+0000","last_clean":"2024-03-26T03:12:33.459662+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:12:33.459662+0000","last_undegraded":"2024-03-26T03:12:33.459662+0000","last_fullsized":"2024-03-26T03:12:33.459662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:52.646244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:52.646244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":189,"reported_epoch":288,"state":"active+undersized","last_fresh":"2024-03-26T03:12:34.138312+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:12:34.138312+0000","last_peered":"2024-03-26T03:12:34.138312+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:12:34.138312+0000","last_undegraded":"2024-03-26T03:12:34.138312+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:35.016919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":608258,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":54,"acting":58,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":807,"num_read_kb":802,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1410,"num_read_kb":1135,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103336,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":650428,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93094724,"statfs":{"total":95995035648,"available":95328997376,"internally_reserved":0,"allocated":1495040,"data_stored":811540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":664533450},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430380,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393912,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93351240,"statfs":{"total":95995035648,"available":95591669760,"internally_reserved":0,"allocated":1814528,"data_stored":1138602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":628276,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93116876,"statfs":{"total":95995035648,"available":95351681024,"internally_reserved":0,"allocated":1548288,"data_stored":820104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":502228,"kb_used_data":1800,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93242924,"statfs":{"total":95995035648,"available":95480754176,"internally_reserved":0,"allocated":1843200,"data_stored":1153687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411519,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411912,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93333240,"statfs":{"total":95995035648,"available":95573237760,"internally_reserved":0,"allocated":1765376,"data_stored":1081501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986358,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":343640,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93401512,"statfs":{"total":95995035648,"available":95643148288,"internally_reserved":0,"allocated":1519616,"data_stored":818283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692018,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614924,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93130228,"statfs":{"total":95995035648,"available":95365353472,"internally_reserved":0,"allocated":2097152,"data_stored":1408202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:12:41.356 INFO:tasks.thrashosds.thrasher:Setting 6.7 to [0, 7, 6, 3, 1, 5] 2024-03-26T03:12:41.356 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.7', '0', '7', '6', '3', '1', '5'] 2024-03-26T03:12:41.357 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 6.7 0 7 6 3 1 5 2024-03-26T03:12:41.368 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:41.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:41.530 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:41.559 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:41.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:41.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: 1.f scrub starts 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: 1.f scrub ok 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: from='client.35513 v1:172.21.15.66:0/1165488965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: pgmap v606: 168 pgs: 3 creating+peering, 5 active+clean+remapped, 22 active+undersized+degraded, 42 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: from='client.35507 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[37920]: osdmap e293: 8 total, 7 up, 8 in 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: 1.f scrub starts 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: 1.f scrub ok 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: from='client.35513 v1:172.21.15.66:0/1165488965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:12:41.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: pgmap v606: 168 pgs: 3 creating+peering, 5 active+clean+remapped, 22 active+undersized+degraded, 42 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:41.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:41.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: from='client.35507 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:12:41.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:41 smithi066 ceph-mon[29666]: osdmap e293: 8 total, 7 up, 8 in 2024-03-26T03:12:41.661 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:41.691 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: 1.f scrub starts 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: 1.f scrub ok 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: from='client.35513 v1:172.21.15.66:0/1165488965' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: pgmap v606: 168 pgs: 3 creating+peering, 5 active+clean+remapped, 22 active+undersized+degraded, 42 active+undersized, 96 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 989 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: from='client.35507 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:12:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:41 smithi192 ceph-mon[33110]: osdmap e293: 8 total, 7 up, 8 in 2024-03-26T03:12:41.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:41.821 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:41.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:41.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:41.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:42.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:42.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:42.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:42.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:42.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:42.328 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:42.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:42.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:42.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:42.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:42.613 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:42.614 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:42.658 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:42.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:42.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:42.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:42.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:42.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:42.937 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:42.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:42.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:43.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:43.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:43.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:43.223 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:43.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:43.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:43.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:43.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:43.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:43.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:43.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:43.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:43 smithi192 ceph-mon[33110]: pgmap v608: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:43.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:43 smithi066 ceph-mon[29666]: pgmap v608: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:43.812 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:43 smithi066 ceph-mon[37920]: pgmap v608: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:43.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:43.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:43.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:43.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:43.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:44.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:44.084 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:12:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:12:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:12:44.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:44.205 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:44.235 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:44.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:44.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:44.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:44.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:44.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:44.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:44.651 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:12:44.660 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:12:44.660 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:12:44.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:44.742 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:44.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:44.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:44.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:44.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:44.999 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:45.102 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:45.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:45.232 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:45.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:45.345 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:45 smithi066 ceph-mon[29666]: osdmap e294: 8 total, 7 up, 8 in 2024-03-26T03:12:45.345 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:45 smithi066 ceph-mon[29666]: pgmap v610: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:45.345 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:45 smithi066 ceph-mon[37920]: osdmap e294: 8 total, 7 up, 8 in 2024-03-26T03:12:45.345 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:45 smithi066 ceph-mon[37920]: pgmap v610: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:45.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:45.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:45 smithi192 ceph-mon[33110]: osdmap e294: 8 total, 7 up, 8 in 2024-03-26T03:12:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:45 smithi192 ceph-mon[33110]: pgmap v610: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:45.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:45.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:45.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:45.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:45.727 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:12:45.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:45.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:45.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:45.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:45.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:45.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:46.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:46.122 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:46.149 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:46.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:46.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:46.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:46.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:46.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:46.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:46.633 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:46.661 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:46.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:46.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:46.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:46.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:46.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:47.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:47.044 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:47.146 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.175 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:47.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:47.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:47.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:47.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:47.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:47.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:47.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:47 smithi066 ceph-mon[29666]: pgmap v611: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:47.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:47 smithi066 ceph-mon[37920]: pgmap v611: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:47.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:47.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:47 smithi192 ceph-mon[33110]: pgmap v611: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:47.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:47.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:47.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:47.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:47.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:48.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:48.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:48.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:48.205 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:48.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:48.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:48.343 INFO:teuthology.orchestra.run.smithi066.stderr:set 6.7 pg_upmap_items mapping to [0->7,6->3,1->5] 2024-03-26T03:12:48.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:48.461 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:48.562 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.591 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:48.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:48 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3829185745' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [0, 7, 6, 3, 1, 5]}]: dispatch 2024-03-26T03:12:48.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:48 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3829185745' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [0, 7, 6, 3, 1, 5]}]: dispatch 2024-03-26T03:12:48.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:48.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:48 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3829185745' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [0, 7, 6, 3, 1, 5]}]: dispatch 2024-03-26T03:12:48.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:48.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:48.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:48.950 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:12:48.969 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:12:48.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:48.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:48.989 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:12:49.000 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:49.015 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:12:49.021 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:49.039 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:12:49.059 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:12:49.074 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:49.079 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:12:49.079 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:12:49.080 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:12:49.106 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-26T03:12:49.107 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:49.209 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:49.236 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:49.338 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.368 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:49.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:49.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:49.497 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:49.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:49.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: pgmap v612: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:49.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3829185745' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [0, 7, 6, 3, 1, 5]}]': finished 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: osdmap e295: 8 total, 7 up, 8 in 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/632157870' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: from='client.35525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2480723634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: from='client.35525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[29666]: osdmap e296: 8 total, 7 up, 8 in 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: pgmap v612: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3829185745' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [0, 7, 6, 3, 1, 5]}]': finished 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: osdmap e295: 8 total, 7 up, 8 in 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/632157870' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:12:49.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: from='client.35525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:12:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2480723634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: from='client.35525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:12:49.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:49 smithi066 ceph-mon[37920]: osdmap e296: 8 total, 7 up, 8 in 2024-03-26T03:12:49.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: pgmap v612: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3829185745' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [0, 7, 6, 3, 1, 5]}]': finished 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: osdmap e295: 8 total, 7 up, 8 in 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/632157870' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: from='client.35525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2480723634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: from='client.35525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:12:49.730 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-mon[33110]: osdmap e296: 8 total, 7 up, 8 in 2024-03-26T03:12:49.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:49.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:49.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:49.886 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:49.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:49.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:49.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:12:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:12:49.989 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:50.017 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:50.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:50.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:50.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:50.275 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:50.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:50.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:50.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:50.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:50.546 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:50 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:12:50.547 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:50 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:12:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:12:50.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:12:50.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.663 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:50.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:50 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:12:50.765 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.794 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:50.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:50.895 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:50.923 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:50.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:51.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:51.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:51.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:51.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:51.284 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:51.313 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:51.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:51.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:51.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:51.573 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:51.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:51.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:51 smithi192 ceph-mon[33110]: pgmap v615: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:51.796 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:51.796 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:51 smithi066 ceph-mon[37920]: pgmap v615: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:51.796 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:12:51.797 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:51 smithi066 ceph-mon[29666]: pgmap v615: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:51.804 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:51.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:51.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:51.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:51.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:52.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:52.093 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:52.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:52.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:52.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:52.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:52.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:52.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:52.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:12:52.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:52.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:52.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:52.828 DEBUG:teuthology.orchestra.run.smithi066:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-26T03:12:52.830 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:12:52.830 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:12:52.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:52.851 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:52.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:52.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:52.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:52.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:53.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:53.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:53.126 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:hints = 1 2024-03-26T03:12:53.126 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:12:53.126 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object prefix: benchmark_data_smithi066_148944 2024-03-26T03:12:53.127 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:12:53.127 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 0 3 3 0 0 0 - 0 2024-03-26T03:12:53.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:53.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:53.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:53.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:53.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:53.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:53.527 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:12:53.528 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:12:53.528 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 0 2024-03-26T03:12:53.528 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:12:53.550 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:53 smithi066 ceph-mon[29666]: pgmap v616: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:53.550 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:53 smithi066 ceph-mon[37920]: pgmap v616: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:53.611 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:53.656 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:53 smithi192 ceph-mon[33110]: pgmap v616: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 600 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 114/690 objects degraded (16.522%); 1/690 objects misplaced (0.145%) 2024-03-26T03:12:53.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:53.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:53.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:53.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:53.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:54.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:54.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:12:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:12:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:12:54.127 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 1 16 3774 3758 234.797 234.875 0.00269324 0.00424151 2024-03-26T03:12:54.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:54.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:54.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:54.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:54.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:54.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:54.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:54.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:54.663 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:12:54.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:12:54.663 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-26T03:12:54.663 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:12:54.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:54.795 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:54.897 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:54.945 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:54.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:55.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:55.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:55.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 2 16 7302 7286 227.636 220.5 0.00531915 0.00438341 2024-03-26T03:12:55.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:55.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:55.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:55.367 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:55.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:55.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:55.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.655 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:55.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:55 smithi192 ceph-mon[33110]: pgmap v617: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 34 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 4.2 MiB/s wr, 67 op/s; 114/2292 objects degraded (4.974%); 1/2292 objects misplaced (0.044%) 2024-03-26T03:12:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:55 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 114/2292 objects degraded (4.974%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:55.756 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.800 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:55.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:55.810 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[37920]: pgmap v617: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 34 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 4.2 MiB/s wr, 67 op/s; 114/2292 objects degraded (4.974%); 1/2292 objects misplaced (0.044%) 2024-03-26T03:12:55.810 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 114/2292 objects degraded (4.974%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:55.810 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:55.810 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:55.811 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[29666]: pgmap v617: 168 pgs: 5 active+clean+remapped, 22 active+undersized+degraded, 43 active+undersized, 98 active+clean; 34 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 4.2 MiB/s wr, 67 op/s; 114/2292 objects degraded (4.974%); 1/2292 objects misplaced (0.044%) 2024-03-26T03:12:55.811 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 114/2292 objects degraded (4.974%), 22 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:12:55.811 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:12:55.811 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:12:55.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:12:55.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:55.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:55.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:56.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:56.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:56.127 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 3 16 10674 10658 221.968 210.75 0.00363048 0.00449749 2024-03-26T03:12:56.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:56.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:56.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:56.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:56.413 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:12:56.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:56.493 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:56.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:56.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:56.697 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/307960808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:56.697 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:56 smithi066 ceph-mon[29666]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:56.698 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/307960808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:56.698 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:56 smithi066 ceph-mon[37920]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:56.734 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/307960808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:56 smithi192 ceph-mon[33110]: from='client.35546 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:12:56.778 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:56.880 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:56.919 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:56.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:57.021 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:57.064 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:57.128 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 4 15 13905 13890 216.972 202 0.00394587 0.00460254 2024-03-26T03:12:57.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:57.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:57.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:57.347 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:57.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:57.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:57.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:57.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:57.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:57 smithi192 ceph-mon[33110]: pgmap v618: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 364 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 727 op/s; 1997/18120 objects degraded (11.021%); 1/18120 objects misplaced (0.006%) 2024-03-26T03:12:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:57 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:57 smithi192 ceph-mon[33110]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:57 smithi192 ceph-mon[33110]: osdmap e297: 8 total, 7 up, 8 in 2024-03-26T03:12:57.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[29666]: pgmap v618: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 364 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 727 op/s; 1997/18120 objects degraded (11.021%); 1/18120 objects misplaced (0.006%) 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[29666]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[29666]: osdmap e297: 8 total, 7 up, 8 in 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[37920]: pgmap v618: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 364 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 727 op/s; 1997/18120 objects degraded (11.021%); 1/18120 objects misplaced (0.006%) 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[37920]: from='client.35546 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:12:57.771 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:57 smithi066 ceph-mon[37920]: osdmap e297: 8 total, 7 up, 8 in 2024-03-26T03:12:57.862 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:57.900 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:57.913 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:57.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:58.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:58.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:58.128 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 5 16 16941 16925 211.512 189.688 0.00370835 0.00472171 2024-03-26T03:12:58.152 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:58.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:58.313 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:12:58.361 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:12:58.463 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:12:58.508 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:58.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:12:58.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:58.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:58.647 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:12:58.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:58.768 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:12:58.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:58.809 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:12:58.838 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:12:58.859 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:12:58.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:12:58.896 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:12:58.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:58.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:12:58.940 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:12:58.980 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:12:58.980 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:12:58.980 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:12:58.980 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:12:58.981 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:12:58.981 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:12:58.981 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:12:58.981 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:12:58.981 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:12:59.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.069 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:59.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:59.128 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 6 16 19640 19624 204.373 168.688 0.00487012 0.00488712 2024-03-26T03:12:59.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:59.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:12:59.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:12:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:59.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:12:59.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:59.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:59.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:12:59.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:59 smithi066 ceph-mon[29666]: pgmap v620: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 364 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 633 op/s; 1997/18120 objects degraded (11.021%); 1/18120 objects misplaced (0.006%) 2024-03-26T03:12:59.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2809015155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:59.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:12:59 smithi066 ceph-mon[29666]: osdmap e298: 8 total, 7 up, 8 in 2024-03-26T03:12:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:59 smithi066 ceph-mon[37920]: pgmap v620: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 364 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 633 op/s; 1997/18120 objects degraded (11.021%); 1/18120 objects misplaced (0.006%) 2024-03-26T03:12:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2809015155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:12:59 smithi066 ceph-mon[37920]: osdmap e298: 8 total, 7 up, 8 in 2024-03-26T03:12:59.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:12:59.718 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:12:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:59 smithi192 ceph-mon[33110]: pgmap v620: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 364 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 633 op/s; 1997/18120 objects degraded (11.021%); 1/18120 objects misplaced (0.006%) 2024-03-26T03:12:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2809015155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:12:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:12:59 smithi192 ceph-mon[33110]: osdmap e298: 8 total, 7 up, 8 in 2024-03-26T03:12:59.761 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:12:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:12:59.830 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:12:59.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:12:59.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:12:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:12:59.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:00.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:00.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:00.129 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 7 16 22494 22478 200.638 178.375 0.00457007 0.00497924 2024-03-26T03:13:00.139 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:00.190 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:00.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:12:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:13:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:13:00.269 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:13:00.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:00.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:00.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:00.473 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:00.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:00.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:00.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:00 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 5875/56118 objects degraded (10.469%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:00.758 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:00.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:00 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 5875/56118 objects degraded (10.469%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:00.825 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:00 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 5875/56118 objects degraded (10.469%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:00.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:00.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:00.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:00.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:01.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:01.129 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 8 16 25134 25118 196.182 165 0.00670579 0.00509188 2024-03-26T03:13:01.137 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:01.183 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:01.285 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.317 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:01.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:01.418 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:01.462 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:01.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:01.611 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:01.713 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:01 smithi192 ceph-mon[33110]: pgmap v622: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.31k op/s; 5875/56118 objects degraded (10.469%); 1/56118 objects misplaced (0.002%) 2024-03-26T03:13:01.758 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:01 smithi066 ceph-mon[29666]: pgmap v622: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.31k op/s; 5875/56118 objects degraded (10.469%); 1/56118 objects misplaced (0.002%) 2024-03-26T03:13:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:01 smithi066 ceph-mon[37920]: pgmap v622: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 144 MiB/s wr, 2.31k op/s; 5875/56118 objects degraded (10.469%); 1/56118 objects misplaced (0.002%) 2024-03-26T03:13:01.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:01.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:01.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:01.999 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:02.045 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:02.109 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:13:02.112 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:13:02.129 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 9 15 27549 27534 191.161 151 0.00444545 0.00522567 2024-03-26T03:13:02.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:02.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:02.271 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:13:02.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:02.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:02.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:02.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:02.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:02.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:02.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:02.725 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:02.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:02.858 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:02.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:02.959 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:03.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:03.129 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 10 16 29508 29492 184.283 122.375 0.0042962 0.00541994 2024-03-26T03:13:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:03.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:03.240 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:03.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:03.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:03.425 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:03.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:03.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:03.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:03.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:03 smithi192 ceph-mon[33110]: from='client.15711 v1:172.21.15.66:0/3945367054' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:13:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:03 smithi192 ceph-mon[33110]: pgmap v623: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s; 6793/64482 objects degraded (10.535%); 1/64482 objects misplaced (0.002%) 2024-03-26T03:13:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:03 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:03 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:03.808 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:03.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[29666]: from='client.15711 v1:172.21.15.66:0/3945367054' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:13:03.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[29666]: pgmap v623: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s; 6793/64482 objects degraded (10.535%); 1/64482 objects misplaced (0.002%) 2024-03-26T03:13:03.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:03.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:03.834 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[37920]: from='client.15711 v1:172.21.15.66:0/3945367054' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:13:03.834 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[37920]: pgmap v623: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 162 MiB/s wr, 2.59k op/s; 6793/64482 objects degraded (10.535%); 1/64482 objects misplaced (0.002%) 2024-03-26T03:13:03.834 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:03.834 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:03.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:03.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:03.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:03.982 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2024-03-26T03:13:00.277402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1211457418,"num_objects":18706,"num_object_clones":0,"num_object_copies":56118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5875,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":18706,"num_whiteouts":0,"num_read":2272,"num_read_kb":1995,"num_write":19395,"num_write_kb":1182582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":54528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22201,"ondisk_log_size":22201,"up":434,"acting":439,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9242408,"kb_used_data":5314008,"kb_used_omap":12,"kb_used_meta":3928307,"kb_avail":740718808,"statfs":{"total":767960285184,"available":758496059392,"internally_reserved":0,"allocated":5441544192,"data_stored":5435978924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":4022586928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,5,9,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1210843136,"num_objects":18476,"num_object_clones":0,"num_object_copies":55428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18476,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18476,"num_write_kb":1182464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004013"},"pg_stats":[{"pgid":"10.6","version":"296'629","reported_seq":652,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:55.709272+0000","last_change":"2024-03-26T03:12:36.112086+0000","last_active":"2024-03-26T03:12:55.709272+0000","last_peered":"2024-03-26T03:12:55.709272+0000","last_clean":"2024-03-26T03:12:55.709272+0000","last_became_active":"2024-03-26T03:12:36.111737+0000","last_became_peered":"2024-03-26T03:12:36.111737+0000","last_unstale":"2024-03-26T03:12:55.709272+0000","last_undegraded":"2024-03-26T03:12:55.709272+0000","last_fullsized":"2024-03-26T03:12:55.709272+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:58.055331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41222144,"num_objects":629,"num_object_clones":0,"num_object_copies":1887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":629,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":398,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104710+0000","last_change":"2024-03-26T03:10:50.602175+0000","last_active":"2024-03-26T03:12:59.104710+0000","last_peered":"2024-03-26T03:12:59.104710+0000","last_clean":"2024-03-26T03:12:59.104710+0000","last_became_active":"2024-03-26T03:10:44.090472+0000","last_became_peered":"2024-03-26T03:10:44.090472+0000","last_unstale":"2024-03-26T03:12:59.104710+0000","last_undegraded":"2024-03-26T03:12:59.104710+0000","last_fullsized":"2024-03-26T03:12:59.104710+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:50.602062+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:10:50.602062+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017792+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"112'9","reported_seq":703,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.105059+0000","last_change":"2024-03-26T03:10:55.601026+0000","last_active":"2024-03-26T03:12:59.105059+0000","last_peered":"2024-03-26T03:12:59.105059+0000","last_clean":"2024-03-26T03:12:59.105059+0000","last_became_active":"2024-03-26T02:59:00.938505+0000","last_became_peered":"2024-03-26T02:59:00.938505+0000","last_unstale":"2024-03-26T03:12:59.105059+0000","last_undegraded":"2024-03-26T03:12:59.105059+0000","last_fullsized":"2024-03-26T03:12:59.105059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:10:55.600815+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:10:55.600815+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017842+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":611,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.111356+0000","last_change":"2024-03-26T03:11:39.721484+0000","last_active":"2024-03-26T03:12:59.111356+0000","last_peered":"2024-03-26T03:12:59.111356+0000","last_clean":"2024-03-26T03:12:59.111356+0000","last_became_active":"2024-03-26T02:58:58.945238+0000","last_became_peered":"2024-03-26T02:58:58.945238+0000","last_unstale":"2024-03-26T03:12:59.111356+0000","last_undegraded":"2024-03-26T03:12:59.111356+0000","last_fullsized":"2024-03-26T03:12:59.111356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:11:39.721355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:39.721355+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:06.244718+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":512,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.102186+0000","last_change":"2024-03-26T03:06:20.304574+0000","last_active":"2024-03-26T03:12:59.102186+0000","last_peered":"2024-03-26T03:12:59.102186+0000","last_clean":"2024-03-26T03:06:17.224420+0000","last_became_active":"2024-03-26T03:06:20.304574+0000","last_became_peered":"2024-03-26T03:06:20.304574+0000","last_unstale":"2024-03-26T03:12:59.102186+0000","last_undegraded":"2024-03-26T03:12:59.102186+0000","last_fullsized":"2024-03-26T03:06:20.299330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067612+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":342,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106999+0000","last_change":"2024-03-26T03:12:27.221535+0000","last_active":"2024-03-26T03:12:59.106999+0000","last_peered":"2024-03-26T03:12:59.106999+0000","last_clean":"2024-03-26T03:12:59.106999+0000","last_became_active":"2024-03-26T03:04:14.423233+0000","last_became_peered":"2024-03-26T03:04:14.423233+0000","last_unstale":"2024-03-26T03:12:59.106999+0000","last_undegraded":"2024-03-26T03:12:59.106999+0000","last_fullsized":"2024-03-26T03:12:59.106999+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:27.221439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:27.221439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:28.393425+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"298'1309","reported_seq":1334,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.610406+0000","last_change":"2024-03-26T03:12:53.130744+0000","last_active":"2024-03-26T03:12:59.610406+0000","last_peered":"2024-03-26T03:12:59.610406+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"2024-03-26T03:12:36.104612+0000","last_became_peered":"2024-03-26T03:12:36.104612+0000","last_unstale":"2024-03-26T03:12:59.610406+0000","last_undegraded":"2024-03-26T03:12:53.129775+0000","last_fullsized":"2024-03-26T03:12:36.099921+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:51.125212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85786624,"num_objects":1309,"num_object_clones":0,"num_object_copies":3927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1309,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":402,"reported_epoch":298,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:59.103084+0000","last_change":"2024-03-26T03:12:31.058456+0000","last_active":"2024-03-26T03:12:59.103084+0000","last_peered":"2024-03-26T03:12:59.103084+0000","last_clean":"2024-03-26T03:12:59.103084+0000","last_became_active":"2024-03-26T03:10:44.082186+0000","last_became_peered":"2024-03-26T03:10:44.082186+0000","last_unstale":"2024-03-26T03:12:59.103084+0000","last_undegraded":"2024-03-26T03:12:59.103084+0000","last_fullsized":"2024-03-26T03:12:59.103084+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:31.058290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:31.058290+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:32.950088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":575,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.099026+0000","last_change":"2024-03-26T03:12:30.521625+0000","last_active":"2024-03-26T03:12:49.099026+0000","last_peered":"2024-03-26T03:12:49.099026+0000","last_clean":"2024-03-26T03:12:49.099026+0000","last_became_active":"2024-03-26T02:58:58.934949+0000","last_became_peered":"2024-03-26T02:58:58.934949+0000","last_unstale":"2024-03-26T03:12:49.099026+0000","last_undegraded":"2024-03-26T03:12:49.099026+0000","last_fullsized":"2024-03-26T03:12:49.099026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:12:30.521522+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.521522+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:31.997671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":512,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.107463+0000","last_change":"2024-03-26T03:06:20.305162+0000","last_active":"2024-03-26T03:12:59.107463+0000","last_peered":"2024-03-26T03:12:59.107463+0000","last_clean":"2024-03-26T03:06:17.234110+0000","last_became_active":"2024-03-26T03:06:20.305162+0000","last_became_peered":"2024-03-26T03:06:20.305162+0000","last_unstale":"2024-03-26T03:12:59.107463+0000","last_undegraded":"2024-03-26T03:12:59.107463+0000","last_fullsized":"2024-03-26T03:06:20.299509+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:14.147006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:06:14.147006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":496,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.101985+0000","last_change":"2024-03-26T03:06:20.304944+0000","last_active":"2024-03-26T03:12:59.101985+0000","last_peered":"2024-03-26T03:12:59.101985+0000","last_clean":"2024-03-26T03:06:17.223268+0000","last_became_active":"2024-03-26T03:06:20.304944+0000","last_became_peered":"2024-03-26T03:06:20.304944+0000","last_unstale":"2024-03-26T03:12:59.101985+0000","last_undegraded":"2024-03-26T03:12:59.101985+0000","last_fullsized":"2024-03-26T03:06:20.263792+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":517,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.098261+0000","last_change":"2024-03-26T03:11:43.664485+0000","last_active":"2024-03-26T03:12:49.098261+0000","last_peered":"2024-03-26T03:12:49.098261+0000","last_clean":"2024-03-26T03:12:49.098261+0000","last_became_active":"2024-03-26T02:59:02.947942+0000","last_became_peered":"2024-03-26T02:59:02.947942+0000","last_unstale":"2024-03-26T03:12:49.098261+0000","last_undegraded":"2024-03-26T03:12:49.098261+0000","last_fullsized":"2024-03-26T03:12:49.098261+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:43.664357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:11:43.664357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:55.152927+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"296'603","reported_seq":624,"reported_epoch":296,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:55.720614+0000","last_change":"2024-03-26T03:12:53.133092+0000","last_active":"2024-03-26T03:12:55.720614+0000","last_peered":"2024-03-26T03:12:55.720614+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"2024-03-26T03:12:36.109075+0000","last_became_peered":"2024-03-26T03:12:36.109075+0000","last_unstale":"2024-03-26T03:12:55.720614+0000","last_undegraded":"2024-03-26T03:12:53.132067+0000","last_fullsized":"2024-03-26T03:12:36.101148+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:14:01.844534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":603,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":603,"num_write_kb":38592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":603}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":614,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107902+0000","last_change":"2024-03-26T03:12:40.141914+0000","last_active":"2024-03-26T03:12:59.107902+0000","last_peered":"2024-03-26T03:12:59.107902+0000","last_clean":"2024-03-26T03:12:59.107902+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:12:59.107902+0000","last_undegraded":"2024-03-26T03:12:59.107902+0000","last_fullsized":"2024-03-26T03:12:59.107902+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:40.141823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:40.141823+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.854915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":624,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107363+0000","last_change":"2024-03-26T03:11:12.308404+0000","last_active":"2024-03-26T03:12:59.107363+0000","last_peered":"2024-03-26T03:12:59.107363+0000","last_clean":"2024-03-26T03:12:59.107363+0000","last_became_active":"2024-03-26T02:59:30.868944+0000","last_became_peered":"2024-03-26T02:59:30.868944+0000","last_unstale":"2024-03-26T03:12:59.107363+0000","last_undegraded":"2024-03-26T03:12:59.107363+0000","last_fullsized":"2024-03-26T03:12:59.107363+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:11:12.308255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:12.308255+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":549,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102447+0000","last_change":"2024-03-26T03:11:51.169305+0000","last_active":"2024-03-26T03:12:59.102447+0000","last_peered":"2024-03-26T03:12:59.102447+0000","last_clean":"2024-03-26T03:12:59.102447+0000","last_became_active":"2024-03-26T02:58:56.920033+0000","last_became_peered":"2024-03-26T02:58:56.920033+0000","last_unstale":"2024-03-26T03:12:59.102447+0000","last_undegraded":"2024-03-26T03:12:59.102447+0000","last_fullsized":"2024-03-26T03:12:59.102447+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.169187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.169187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:15.669587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102033+0000","last_change":"2024-03-26T03:11:46.249753+0000","last_active":"2024-03-26T03:12:59.102033+0000","last_peered":"2024-03-26T03:12:59.102033+0000","last_clean":"2024-03-26T03:12:59.102033+0000","last_became_active":"2024-03-26T02:59:00.932780+0000","last_became_peered":"2024-03-26T02:59:00.932780+0000","last_unstale":"2024-03-26T03:12:59.102033+0000","last_undegraded":"2024-03-26T03:12:59.102033+0000","last_fullsized":"2024-03-26T03:12:59.102033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:46.249597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.249597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:08.763925+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":492,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104160+0000","last_change":"2024-03-26T03:06:20.303775+0000","last_active":"2024-03-26T03:12:59.104160+0000","last_peered":"2024-03-26T03:12:59.104160+0000","last_clean":"2024-03-26T03:06:17.230147+0000","last_became_active":"2024-03-26T03:06:20.303775+0000","last_became_peered":"2024-03-26T03:06:20.303775+0000","last_unstale":"2024-03-26T03:12:59.104160+0000","last_undegraded":"2024-03-26T03:12:59.104160+0000","last_fullsized":"2024-03-26T03:06:20.265355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3","version":"298'1316","reported_seq":1341,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.574155+0000","last_change":"2024-03-26T03:12:53.136099+0000","last_active":"2024-03-26T03:12:59.574155+0000","last_peered":"2024-03-26T03:12:59.574155+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"2024-03-26T03:12:36.104921+0000","last_became_peered":"2024-03-26T03:12:36.104921+0000","last_unstale":"2024-03-26T03:12:59.574155+0000","last_undegraded":"2024-03-26T03:12:53.133287+0000","last_fullsized":"2024-03-26T03:12:36.099922+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:47.777918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86245376,"num_objects":1316,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1316,"num_write_kb":84224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1316}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":592,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.103380+0000","last_change":"2024-03-26T03:12:30.037525+0000","last_active":"2024-03-26T03:12:59.103380+0000","last_peered":"2024-03-26T03:12:59.103380+0000","last_clean":"2024-03-26T03:12:59.103380+0000","last_became_active":"2024-03-26T02:58:58.920520+0000","last_became_peered":"2024-03-26T02:58:58.920520+0000","last_unstale":"2024-03-26T03:12:59.103380+0000","last_undegraded":"2024-03-26T03:12:59.103380+0000","last_fullsized":"2024-03-26T03:12:59.103380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:12:30.037403+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.037403+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.381867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104372+0000","last_change":"2024-03-26T03:11:41.872736+0000","last_active":"2024-03-26T03:12:59.104372+0000","last_peered":"2024-03-26T03:12:59.104372+0000","last_clean":"2024-03-26T03:12:59.104372+0000","last_became_active":"2024-03-26T02:58:56.922541+0000","last_became_peered":"2024-03-26T02:58:56.922541+0000","last_unstale":"2024-03-26T03:12:59.104372+0000","last_undegraded":"2024-03-26T03:12:59.104372+0000","last_fullsized":"2024-03-26T03:12:59.104372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:41.872620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.872620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017954+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":525,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106720+0000","last_change":"2024-03-26T03:12:23.358746+0000","last_active":"2024-03-26T03:12:59.106720+0000","last_peered":"2024-03-26T03:12:59.106720+0000","last_clean":"2024-03-26T03:12:59.106720+0000","last_became_active":"2024-03-26T02:59:00.936237+0000","last_became_peered":"2024-03-26T02:59:00.936237+0000","last_unstale":"2024-03-26T03:12:59.106720+0000","last_undegraded":"2024-03-26T03:12:59.106720+0000","last_fullsized":"2024-03-26T03:12:59.106720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:23.358630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.358630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:37.326581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":500,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.103067+0000","last_change":"2024-03-26T03:06:20.305272+0000","last_active":"2024-03-26T03:12:59.103067+0000","last_peered":"2024-03-26T03:12:59.103067+0000","last_clean":"2024-03-26T03:06:17.223209+0000","last_became_active":"2024-03-26T03:06:20.305272+0000","last_became_peered":"2024-03-26T03:06:20.305272+0000","last_unstale":"2024-03-26T03:12:59.103067+0000","last_undegraded":"2024-03-26T03:12:59.103067+0000","last_fullsized":"2024-03-26T03:06:20.264144+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067646+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":581,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.101536+0000","last_change":"2024-03-26T03:06:20.306144+0000","last_active":"2024-03-26T03:12:59.101536+0000","last_peered":"2024-03-26T03:12:59.101536+0000","last_clean":"2024-03-26T03:06:17.222359+0000","last_became_active":"2024-03-26T03:06:20.306144+0000","last_became_peered":"2024-03-26T03:06:20.306144+0000","last_unstale":"2024-03-26T03:12:59.101536+0000","last_undegraded":"2024-03-26T03:06:20.262550+0000","last_fullsized":"2024-03-26T03:06:20.262435+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"298'1393","reported_seq":1421,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.982978+0000","last_change":"2024-03-26T03:12:36.112758+0000","last_active":"2024-03-26T03:12:59.982978+0000","last_peered":"2024-03-26T03:12:59.982978+0000","last_clean":"2024-03-26T03:12:59.982978+0000","last_became_active":"2024-03-26T03:12:36.112377+0000","last_became_peered":"2024-03-26T03:12:36.112377+0000","last_unstale":"2024-03-26T03:12:59.982978+0000","last_undegraded":"2024-03-26T03:12:59.982978+0000","last_fullsized":"2024-03-26T03:12:59.982978+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1393,"log_dups_size":0,"ondisk_log_size":1393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:42.990662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":4179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1393,"num_write_kb":89152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":772,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104166+0000","last_change":"2024-03-26T03:06:20.303920+0000","last_active":"2024-03-26T03:12:59.104166+0000","last_peered":"2024-03-26T03:12:59.104166+0000","last_clean":"2024-03-26T03:06:17.230166+0000","last_became_active":"2024-03-26T03:06:20.303920+0000","last_became_peered":"2024-03-26T03:06:20.303920+0000","last_unstale":"2024-03-26T03:12:59.104166+0000","last_undegraded":"2024-03-26T03:12:59.104166+0000","last_fullsized":"2024-03-26T03:06:20.265637+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.630852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.630852+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":536,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.102914+0000","last_change":"2024-03-26T03:06:20.273541+0000","last_active":"2024-03-26T03:12:59.102914+0000","last_peered":"2024-03-26T03:12:59.102914+0000","last_clean":"2024-03-26T03:06:17.220025+0000","last_became_active":"2024-03-26T03:06:20.273541+0000","last_became_peered":"2024-03-26T03:06:20.273541+0000","last_unstale":"2024-03-26T03:12:59.102914+0000","last_undegraded":"2024-03-26T03:06:20.262608+0000","last_fullsized":"2024-03-26T03:06:20.262480+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255078+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":504,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.106944+0000","last_change":"2024-03-26T03:06:20.311932+0000","last_active":"2024-03-26T03:12:59.106944+0000","last_peered":"2024-03-26T03:12:59.106944+0000","last_clean":"2024-03-26T03:06:17.249719+0000","last_became_active":"2024-03-26T03:06:20.311932+0000","last_became_peered":"2024-03-26T03:06:20.311932+0000","last_unstale":"2024-03-26T03:12:59.106944+0000","last_undegraded":"2024-03-26T03:12:59.106944+0000","last_fullsized":"2024-03-26T03:06:20.300995+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.659265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.659265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":719,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102579+0000","last_change":"2024-03-26T03:12:28.903130+0000","last_active":"2024-03-26T03:12:59.102579+0000","last_peered":"2024-03-26T03:12:59.102579+0000","last_clean":"2024-03-26T03:12:59.102579+0000","last_became_active":"2024-03-26T02:59:00.938793+0000","last_became_peered":"2024-03-26T02:59:00.938793+0000","last_unstale":"2024-03-26T03:12:59.102579+0000","last_undegraded":"2024-03-26T03:12:59.102579+0000","last_fullsized":"2024-03-26T03:12:59.102579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:12:28.903038+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.903038+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:42.184864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.103498+0000","last_change":"2024-03-26T03:12:25.078525+0000","last_active":"2024-03-26T03:12:59.103498+0000","last_peered":"2024-03-26T03:12:59.103498+0000","last_clean":"2024-03-26T03:12:59.103498+0000","last_became_active":"2024-03-26T02:59:02.944934+0000","last_became_peered":"2024-03-26T02:59:02.944934+0000","last_unstale":"2024-03-26T03:12:59.103498+0000","last_undegraded":"2024-03-26T03:12:59.103498+0000","last_fullsized":"2024-03-26T03:12:59.103498+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:25.078426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:25.078426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:37.085323+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"298'1271","reported_seq":1298,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.230911+0000","last_change":"2024-03-26T03:12:36.108639+0000","last_active":"2024-03-26T03:12:59.230911+0000","last_peered":"2024-03-26T03:12:59.230911+0000","last_clean":"2024-03-26T03:12:59.230911+0000","last_became_active":"2024-03-26T03:12:36.108301+0000","last_became_peered":"2024-03-26T03:12:36.108301+0000","last_unstale":"2024-03-26T03:12:59.230911+0000","last_undegraded":"2024-03-26T03:12:59.230911+0000","last_fullsized":"2024-03-26T03:12:59.230911+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1271,"log_dups_size":0,"ondisk_log_size":1271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.335079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83296256,"num_objects":1271,"num_object_clones":0,"num_object_copies":3813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1271,"num_write_kb":81344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":619,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.101832+0000","last_change":"2024-03-26T03:11:52.158484+0000","last_active":"2024-03-26T03:12:59.101832+0000","last_peered":"2024-03-26T03:12:59.101832+0000","last_clean":"2024-03-26T03:12:59.101832+0000","last_became_active":"2024-03-26T03:04:41.967233+0000","last_became_peered":"2024-03-26T03:04:41.967233+0000","last_unstale":"2024-03-26T03:12:59.101832+0000","last_undegraded":"2024-03-26T03:12:59.101832+0000","last_fullsized":"2024-03-26T03:12:59.101832+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:11:52.158163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:11:52.158163+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:07.642900+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":551,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.107509+0000","last_change":"2024-03-26T03:06:20.312453+0000","last_active":"2024-03-26T03:12:59.107509+0000","last_peered":"2024-03-26T03:12:59.107509+0000","last_clean":"2024-03-26T03:06:17.234444+0000","last_became_active":"2024-03-26T03:06:20.312453+0000","last_became_peered":"2024-03-26T03:06:20.312453+0000","last_unstale":"2024-03-26T03:12:59.107509+0000","last_undegraded":"2024-03-26T03:06:20.304943+0000","last_fullsized":"2024-03-26T03:06:20.304823+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:06:17.147821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.147821+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":504,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.106180+0000","last_change":"2024-03-26T03:06:20.308706+0000","last_active":"2024-03-26T03:12:59.106180+0000","last_peered":"2024-03-26T03:12:59.106180+0000","last_clean":"2024-03-26T03:06:17.232567+0000","last_became_active":"2024-03-26T03:06:20.308706+0000","last_became_peered":"2024-03-26T03:06:20.308706+0000","last_unstale":"2024-03-26T03:12:59.106180+0000","last_undegraded":"2024-03-26T03:12:59.106180+0000","last_fullsized":"2024-03-26T03:06:20.303884+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:12.538928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:12.538928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"71'8","reported_seq":700,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.103473+0000","last_change":"2024-03-26T03:11:40.792357+0000","last_active":"2024-03-26T03:12:59.103473+0000","last_peered":"2024-03-26T03:12:59.103473+0000","last_clean":"2024-03-26T03:12:59.103473+0000","last_became_active":"2024-03-26T02:59:00.934467+0000","last_became_peered":"2024-03-26T02:59:00.934467+0000","last_unstale":"2024-03-26T03:12:59.103473+0000","last_undegraded":"2024-03-26T03:12:59.103473+0000","last_fullsized":"2024-03-26T03:12:59.103473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:11:40.791990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.791990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255213+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":325,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107120+0000","last_change":"2024-03-26T03:11:01.239298+0000","last_active":"2024-03-26T03:12:59.107120+0000","last_peered":"2024-03-26T03:12:59.107120+0000","last_clean":"2024-03-26T03:12:59.107120+0000","last_became_active":"2024-03-26T03:04:14.431302+0000","last_became_peered":"2024-03-26T03:04:14.431302+0000","last_unstale":"2024-03-26T03:12:59.107120+0000","last_undegraded":"2024-03-26T03:12:59.107120+0000","last_fullsized":"2024-03-26T03:12:59.107120+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.239012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.239012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"298'1240","reported_seq":1268,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.352256+0000","last_change":"2024-03-26T03:12:36.111085+0000","last_active":"2024-03-26T03:12:59.352256+0000","last_peered":"2024-03-26T03:12:59.352256+0000","last_clean":"2024-03-26T03:12:59.352256+0000","last_became_active":"2024-03-26T03:12:36.110775+0000","last_became_peered":"2024-03-26T03:12:36.110775+0000","last_unstale":"2024-03-26T03:12:59.352256+0000","last_undegraded":"2024-03-26T03:12:59.352256+0000","last_fullsized":"2024-03-26T03:12:59.352256+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:45.793717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":79360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":632,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106599+0000","last_change":"2024-03-26T03:12:32.478786+0000","last_active":"2024-03-26T03:12:59.106599+0000","last_peered":"2024-03-26T03:12:59.106599+0000","last_clean":"2024-03-26T03:12:59.106599+0000","last_became_active":"2024-03-26T03:10:44.079636+0000","last_became_peered":"2024-03-26T03:10:44.079636+0000","last_unstale":"2024-03-26T03:12:59.106599+0000","last_undegraded":"2024-03-26T03:12:59.106599+0000","last_fullsized":"2024-03-26T03:12:59.106599+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:32.478665+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:32.478665+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:55.357668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"291'31","reported_seq":584,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.103011+0000","last_change":"2024-03-26T03:12:37.144959+0000","last_active":"2024-03-26T03:12:59.103011+0000","last_peered":"2024-03-26T03:12:59.103011+0000","last_clean":"2024-03-26T03:12:59.103011+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:12:59.103011+0000","last_undegraded":"2024-03-26T03:12:59.103011+0000","last_fullsized":"2024-03-26T03:12:59.103011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:12:37.144848+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:14:02.201527+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104462+0000","last_change":"2024-03-26T03:11:00.578809+0000","last_active":"2024-03-26T03:12:59.104462+0000","last_peered":"2024-03-26T03:12:59.104462+0000","last_clean":"2024-03-26T03:12:59.104462+0000","last_became_active":"2024-03-26T02:58:56.925603+0000","last_became_peered":"2024-03-26T02:58:56.925603+0000","last_unstale":"2024-03-26T03:12:59.104462+0000","last_undegraded":"2024-03-26T03:12:59.104462+0000","last_fullsized":"2024-03-26T03:12:59.104462+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:00.578704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:00.578704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017811+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":330,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102698+0000","last_change":"2024-03-26T03:11:48.965867+0000","last_active":"2024-03-26T03:12:59.102698+0000","last_peered":"2024-03-26T03:12:59.102698+0000","last_clean":"2024-03-26T03:12:59.102698+0000","last_became_active":"2024-03-26T03:05:32.342438+0000","last_became_peered":"2024-03-26T03:05:32.342438+0000","last_unstale":"2024-03-26T03:12:59.102698+0000","last_undegraded":"2024-03-26T03:12:59.102698+0000","last_fullsized":"2024-03-26T03:12:59.102698+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.965732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.965732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:12.010040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":209,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.107109+0000","last_change":"2024-03-26T03:06:20.276196+0000","last_active":"2024-03-26T03:12:59.107109+0000","last_peered":"2024-03-26T03:12:59.107109+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276196+0000","last_became_peered":"2024-03-26T03:06:20.276196+0000","last_unstale":"2024-03-26T03:12:59.107109+0000","last_undegraded":"2024-03-26T03:12:59.107109+0000","last_fullsized":"2024-03-26T03:06:20.269438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"298'1269","reported_seq":1294,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.341399+0000","last_change":"2024-03-26T03:12:53.137668+0000","last_active":"2024-03-26T03:12:59.341399+0000","last_peered":"2024-03-26T03:12:59.341399+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"2024-03-26T03:12:36.105799+0000","last_became_peered":"2024-03-26T03:12:36.105799+0000","last_unstale":"2024-03-26T03:12:59.341399+0000","last_undegraded":"2024-03-26T03:12:53.136975+0000","last_fullsized":"2024-03-26T03:12:36.101200+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1269,"log_dups_size":0,"ondisk_log_size":1269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:14:00.995614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83165184,"num_objects":1269,"num_object_clones":0,"num_object_copies":3807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1269,"num_write_kb":81216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":1269}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"199'10","reported_seq":380,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.106046+0000","last_change":"2024-03-26T03:06:20.300687+0000","last_active":"2024-03-26T03:12:59.106046+0000","last_peered":"2024-03-26T03:12:59.106046+0000","last_clean":"2024-03-26T03:00:37.455731+0000","last_became_active":"2024-03-26T03:06:20.300687+0000","last_became_peered":"2024-03-26T03:06:20.300687+0000","last_unstale":"2024-03-26T03:12:59.106046+0000","last_undegraded":"2024-03-26T03:06:20.262544+0000","last_fullsized":"2024-03-26T03:06:20.262407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":589,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.101898+0000","last_change":"2024-03-26T03:12:22.041053+0000","last_active":"2024-03-26T03:12:59.101898+0000","last_peered":"2024-03-26T03:12:59.101898+0000","last_clean":"2024-03-26T03:12:59.101898+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:12:59.101898+0000","last_undegraded":"2024-03-26T03:12:59.101898+0000","last_fullsized":"2024-03-26T03:12:59.101898+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:12:22.040942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.040942+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:29.332833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":541,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.108245+0000","last_change":"2024-03-26T03:12:32.227477+0000","last_active":"2024-03-26T03:12:59.108245+0000","last_peered":"2024-03-26T03:12:59.108245+0000","last_clean":"2024-03-26T03:12:59.108245+0000","last_became_active":"2024-03-26T02:58:56.922603+0000","last_became_peered":"2024-03-26T02:58:56.922603+0000","last_unstale":"2024-03-26T03:12:59.108245+0000","last_undegraded":"2024-03-26T03:12:59.108245+0000","last_fullsized":"2024-03-26T03:12:59.108245+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:32.227383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:12:32.227383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:47.486616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":521,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.105196+0000","last_change":"2024-03-26T03:11:49.921518+0000","last_active":"2024-03-26T03:12:59.105196+0000","last_peered":"2024-03-26T03:12:59.105196+0000","last_clean":"2024-03-26T03:12:59.105196+0000","last_became_active":"2024-03-26T02:59:03.207398+0000","last_became_peered":"2024-03-26T02:59:03.207398+0000","last_unstale":"2024-03-26T03:12:59.105196+0000","last_undegraded":"2024-03-26T03:12:59.105196+0000","last_fullsized":"2024-03-26T03:12:59.105196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.921407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:05.885611+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":692,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104499+0000","last_change":"2024-03-26T03:10:39.370375+0000","last_active":"2024-03-26T03:12:59.104499+0000","last_peered":"2024-03-26T03:12:59.104499+0000","last_clean":"2024-03-26T03:04:37.120815+0000","last_became_active":"2024-03-26T03:10:39.370375+0000","last_became_peered":"2024-03-26T03:10:39.370375+0000","last_unstale":"2024-03-26T03:12:59.104499+0000","last_undegraded":"2024-03-26T03:12:59.104499+0000","last_fullsized":"2024-03-26T03:10:39.363411+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:03.056124+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"298'1228","reported_seq":1255,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.239486+0000","last_change":"2024-03-26T03:12:36.109906+0000","last_active":"2024-03-26T03:12:59.239486+0000","last_peered":"2024-03-26T03:12:59.239486+0000","last_clean":"2024-03-26T03:12:59.239486+0000","last_became_active":"2024-03-26T03:12:36.109323+0000","last_became_peered":"2024-03-26T03:12:36.109323+0000","last_unstale":"2024-03-26T03:12:59.239486+0000","last_undegraded":"2024-03-26T03:12:59.239486+0000","last_fullsized":"2024-03-26T03:12:59.239486+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1228,"log_dups_size":0,"ondisk_log_size":1228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:51.471972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80478208,"num_objects":1228,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1228,"num_write_kb":78592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":494,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.102477+0000","last_change":"2024-03-26T03:06:20.271379+0000","last_active":"2024-03-26T03:12:59.102477+0000","last_peered":"2024-03-26T03:12:59.102477+0000","last_clean":"2024-03-26T03:06:17.219212+0000","last_became_active":"2024-03-26T03:06:20.271379+0000","last_became_peered":"2024-03-26T03:06:20.271379+0000","last_unstale":"2024-03-26T03:12:59.102477+0000","last_undegraded":"2024-03-26T03:12:59.102477+0000","last_fullsized":"2024-03-26T03:06:20.262385+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":336,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.104780+0000","last_change":"2024-03-26T03:06:20.301603+0000","last_active":"2024-03-26T03:12:59.104780+0000","last_peered":"2024-03-26T03:12:59.104780+0000","last_clean":"2024-03-26T03:00:08.766589+0000","last_became_active":"2024-03-26T03:06:20.301603+0000","last_became_peered":"2024-03-26T03:06:20.301603+0000","last_unstale":"2024-03-26T03:12:59.104780+0000","last_undegraded":"2024-03-26T03:06:20.266569+0000","last_fullsized":"2024-03-26T03:06:20.266458+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":535,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107194+0000","last_change":"2024-03-26T03:11:41.297116+0000","last_active":"2024-03-26T03:12:59.107194+0000","last_peered":"2024-03-26T03:12:59.107194+0000","last_clean":"2024-03-26T03:12:59.107194+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:12:59.107194+0000","last_undegraded":"2024-03-26T03:12:59.107194+0000","last_fullsized":"2024-03-26T03:12:59.107194+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:11:41.296291+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.296291+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":490,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.102210+0000","last_change":"2024-03-26T03:06:20.306471+0000","last_active":"2024-03-26T03:12:59.102210+0000","last_peered":"2024-03-26T03:12:59.102210+0000","last_clean":"2024-03-26T03:06:17.219927+0000","last_became_active":"2024-03-26T03:06:20.306471+0000","last_became_peered":"2024-03-26T03:06:20.306471+0000","last_unstale":"2024-03-26T03:12:59.102210+0000","last_undegraded":"2024-03-26T03:12:59.102210+0000","last_fullsized":"2024-03-26T03:06:20.264003+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255048+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":623,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102840+0000","last_change":"2024-03-26T03:12:36.136920+0000","last_active":"2024-03-26T03:12:59.102840+0000","last_peered":"2024-03-26T03:12:59.102840+0000","last_clean":"2024-03-26T03:12:59.102840+0000","last_became_active":"2024-03-26T03:10:39.368150+0000","last_became_peered":"2024-03-26T03:10:39.368150+0000","last_unstale":"2024-03-26T03:12:59.102840+0000","last_undegraded":"2024-03-26T03:12:59.102840+0000","last_fullsized":"2024-03-26T03:12:59.102840+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:12:36.136805+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:49.835152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"298'1277","reported_seq":1305,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.353203+0000","last_change":"2024-03-26T03:12:36.108726+0000","last_active":"2024-03-26T03:12:59.353203+0000","last_peered":"2024-03-26T03:12:59.353203+0000","last_clean":"2024-03-26T03:12:59.353203+0000","last_became_active":"2024-03-26T03:12:36.108383+0000","last_became_peered":"2024-03-26T03:12:36.108383+0000","last_unstale":"2024-03-26T03:12:59.353203+0000","last_undegraded":"2024-03-26T03:12:59.353203+0000","last_fullsized":"2024-03-26T03:12:59.353203+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:58.697294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83689472,"num_objects":1277,"num_object_clones":0,"num_object_copies":3831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1277,"num_write_kb":81728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107528+0000","last_change":"2024-03-26T03:12:34.211787+0000","last_active":"2024-03-26T03:12:59.107528+0000","last_peered":"2024-03-26T03:12:59.107528+0000","last_clean":"2024-03-26T03:12:59.107528+0000","last_became_active":"2024-03-26T02:59:00.939698+0000","last_became_peered":"2024-03-26T02:59:00.939698+0000","last_unstale":"2024-03-26T03:12:59.107528+0000","last_undegraded":"2024-03-26T03:12:59.107528+0000","last_fullsized":"2024-03-26T03:12:59.107528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.211706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.211706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.510026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":596,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.098706+0000","last_change":"2024-03-26T03:11:47.624747+0000","last_active":"2024-03-26T03:12:49.098706+0000","last_peered":"2024-03-26T03:12:49.098706+0000","last_clean":"2024-03-26T03:12:49.098706+0000","last_became_active":"2024-03-26T02:58:58.945453+0000","last_became_peered":"2024-03-26T02:58:58.945453+0000","last_unstale":"2024-03-26T03:12:49.098706+0000","last_undegraded":"2024-03-26T03:12:49.098706+0000","last_fullsized":"2024-03-26T03:12:49.098706+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:11:47.624655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.624655+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:17.291201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":525,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.105154+0000","last_change":"2024-03-26T03:10:54.596765+0000","last_active":"2024-03-26T03:12:59.105154+0000","last_peered":"2024-03-26T03:12:59.105154+0000","last_clean":"2024-03-26T03:12:59.105154+0000","last_became_active":"2024-03-26T02:58:56.922199+0000","last_became_peered":"2024-03-26T02:58:56.922199+0000","last_unstale":"2024-03-26T03:12:59.105154+0000","last_undegraded":"2024-03-26T03:12:59.105154+0000","last_fullsized":"2024-03-26T03:12:59.105154+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:54.596513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:10:54.596513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017870+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":524,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106373+0000","last_change":"2024-03-26T03:11:16.222874+0000","last_active":"2024-03-26T03:12:59.106373+0000","last_peered":"2024-03-26T03:12:59.106373+0000","last_clean":"2024-03-26T03:12:59.106373+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:12:59.106373+0000","last_undegraded":"2024-03-26T03:12:59.106373+0000","last_fullsized":"2024-03-26T03:12:59.106373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.222744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":590,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.107677+0000","last_change":"2024-03-26T03:10:39.372465+0000","last_active":"2024-03-26T03:12:59.107677+0000","last_peered":"2024-03-26T03:12:59.107677+0000","last_clean":"2024-03-26T03:06:17.226066+0000","last_became_active":"2024-03-26T03:10:39.372465+0000","last_became_peered":"2024-03-26T03:10:39.372465+0000","last_unstale":"2024-03-26T03:12:59.107677+0000","last_undegraded":"2024-03-26T03:12:59.107677+0000","last_fullsized":"2024-03-26T03:10:39.360974+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"296'570","reported_seq":594,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:55.720292+0000","last_change":"2024-03-26T03:12:36.113138+0000","last_active":"2024-03-26T03:12:55.720292+0000","last_peered":"2024-03-26T03:12:55.720292+0000","last_clean":"2024-03-26T03:12:55.720292+0000","last_became_active":"2024-03-26T03:12:36.112023+0000","last_became_peered":"2024-03-26T03:12:36.112023+0000","last_unstale":"2024-03-26T03:12:55.720292+0000","last_undegraded":"2024-03-26T03:12:55.720292+0000","last_fullsized":"2024-03-26T03:12:55.720292+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:38.125757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":36480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":540,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.112656+0000","last_change":"2024-03-26T03:12:37.662737+0000","last_active":"2024-03-26T03:12:59.112656+0000","last_peered":"2024-03-26T03:12:59.112656+0000","last_clean":"2024-03-26T03:12:59.112656+0000","last_became_active":"2024-03-26T02:59:00.934518+0000","last_became_peered":"2024-03-26T02:59:00.934518+0000","last_unstale":"2024-03-26T03:12:59.112656+0000","last_undegraded":"2024-03-26T03:12:59.112656+0000","last_fullsized":"2024-03-26T03:12:59.112656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:37.662593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:37.662593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:14:05.940317+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":575,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.098911+0000","last_change":"2024-03-26T03:11:48.575617+0000","last_active":"2024-03-26T03:12:49.098911+0000","last_peered":"2024-03-26T03:12:49.098911+0000","last_clean":"2024-03-26T03:12:49.098911+0000","last_became_active":"2024-03-26T02:58:58.934732+0000","last_became_peered":"2024-03-26T02:58:58.934732+0000","last_unstale":"2024-03-26T03:12:49.098911+0000","last_undegraded":"2024-03-26T03:12:49.098911+0000","last_fullsized":"2024-03-26T03:12:49.098911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.575462+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":298,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:59.106175+0000","last_change":"2024-03-26T03:12:34.461962+0000","last_active":"2024-03-26T03:12:59.106175+0000","last_peered":"2024-03-26T03:12:59.106175+0000","last_clean":"2024-03-26T03:12:59.106175+0000","last_became_active":"2024-03-26T03:12:18.638374+0000","last_became_peered":"2024-03-26T03:12:18.638374+0000","last_unstale":"2024-03-26T03:12:59.106175+0000","last_undegraded":"2024-03-26T03:12:59.106175+0000","last_fullsized":"2024-03-26T03:12:59.106175+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.461667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.461667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:57.139669+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0,7],"avail_no_missing":["1","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":515,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.097761+0000","last_change":"2024-03-26T03:06:20.303400+0000","last_active":"2024-03-26T03:12:49.097761+0000","last_peered":"2024-03-26T03:12:49.097761+0000","last_clean":"2024-03-26T03:06:18.665793+0000","last_became_active":"2024-03-26T03:06:20.303400+0000","last_became_peered":"2024-03-26T03:06:20.303400+0000","last_unstale":"2024-03-26T03:12:49.097761+0000","last_undegraded":"2024-03-26T03:12:49.097761+0000","last_fullsized":"2024-03-26T03:06:20.264727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416960+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":500,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.106773+0000","last_change":"2024-03-26T03:06:20.310769+0000","last_active":"2024-03-26T03:12:59.106773+0000","last_peered":"2024-03-26T03:12:59.106773+0000","last_clean":"2024-03-26T03:06:19.187144+0000","last_became_active":"2024-03-26T03:06:20.310769+0000","last_became_peered":"2024-03-26T03:06:20.310769+0000","last_unstale":"2024-03-26T03:12:59.106773+0000","last_undegraded":"2024-03-26T03:12:59.106773+0000","last_fullsized":"2024-03-26T03:06:20.298862+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.187030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.187030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":624,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106707+0000","last_change":"2024-03-26T03:12:29.490041+0000","last_active":"2024-03-26T03:12:59.106707+0000","last_peered":"2024-03-26T03:12:59.106707+0000","last_clean":"2024-03-26T03:12:59.106707+0000","last_became_active":"2024-03-26T03:10:39.379213+0000","last_became_peered":"2024-03-26T03:10:39.379213+0000","last_unstale":"2024-03-26T03:12:59.106707+0000","last_undegraded":"2024-03-26T03:12:59.106707+0000","last_fullsized":"2024-03-26T03:12:59.106707+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:29.489954+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:29.489954+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:47.301969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"298'1260","reported_seq":1288,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.337762+0000","last_change":"2024-03-26T03:12:36.111288+0000","last_active":"2024-03-26T03:12:59.337762+0000","last_peered":"2024-03-26T03:12:59.337762+0000","last_clean":"2024-03-26T03:12:59.337762+0000","last_became_active":"2024-03-26T03:12:36.110985+0000","last_became_peered":"2024-03-26T03:12:36.110985+0000","last_unstale":"2024-03-26T03:12:59.337762+0000","last_undegraded":"2024-03-26T03:12:59.337762+0000","last_fullsized":"2024-03-26T03:12:59.337762+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1260,"log_dups_size":0,"ondisk_log_size":1260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:14:01.052950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82575360,"num_objects":1260,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":80640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":525,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104878+0000","last_change":"2024-03-26T03:11:47.792917+0000","last_active":"2024-03-26T03:12:59.104878+0000","last_peered":"2024-03-26T03:12:59.104878+0000","last_clean":"2024-03-26T03:12:59.104878+0000","last_became_active":"2024-03-26T02:59:00.934545+0000","last_became_peered":"2024-03-26T02:59:00.934545+0000","last_unstale":"2024-03-26T03:12:59.104878+0000","last_undegraded":"2024-03-26T03:12:59.104878+0000","last_fullsized":"2024-03-26T03:12:59.104878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:47.792700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.792700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017997+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"231'28","reported_seq":746,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107956+0000","last_change":"2024-03-26T03:12:20.180262+0000","last_active":"2024-03-26T03:12:59.107956+0000","last_peered":"2024-03-26T03:12:59.107956+0000","last_clean":"2024-03-26T03:12:59.107956+0000","last_became_active":"2024-03-26T02:58:58.945349+0000","last_became_peered":"2024-03-26T02:58:58.945349+0000","last_unstale":"2024-03-26T03:12:59.107956+0000","last_undegraded":"2024-03-26T03:12:59.107956+0000","last_fullsized":"2024-03-26T03:12:59.107956+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:12:20.180126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:12:20.180126+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:41.479448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":792,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104357+0000","last_change":"2024-03-26T03:10:47.568899+0000","last_active":"2024-03-26T03:12:59.104357+0000","last_peered":"2024-03-26T03:12:59.104357+0000","last_clean":"2024-03-26T03:12:59.104357+0000","last_became_active":"2024-03-26T03:10:39.378619+0000","last_became_peered":"2024-03-26T03:10:39.378619+0000","last_unstale":"2024-03-26T03:12:59.104357+0000","last_undegraded":"2024-03-26T03:12:59.104357+0000","last_fullsized":"2024-03-26T03:12:59.104357+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:47.568755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:47.568755+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":562,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.111749+0000","last_change":"2024-03-26T03:12:24.846998+0000","last_active":"2024-03-26T03:12:59.111749+0000","last_peered":"2024-03-26T03:12:59.111749+0000","last_clean":"2024-03-26T03:12:59.111749+0000","last_became_active":"2024-03-26T02:59:30.867079+0000","last_became_peered":"2024-03-26T02:59:30.867079+0000","last_unstale":"2024-03-26T03:12:59.111749+0000","last_undegraded":"2024-03-26T03:12:59.111749+0000","last_fullsized":"2024-03-26T03:12:59.111749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:24.846895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:12:24.846895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:51.542935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":236,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.111875+0000","last_change":"2024-03-26T03:06:20.279803+0000","last_active":"2024-03-26T03:12:59.111875+0000","last_peered":"2024-03-26T03:12:59.111875+0000","last_clean":"2024-03-26T02:59:00.924687+0000","last_became_active":"2024-03-26T03:06:20.279803+0000","last_became_peered":"2024-03-26T03:06:20.279803+0000","last_unstale":"2024-03-26T03:12:59.111875+0000","last_undegraded":"2024-03-26T03:06:20.268128+0000","last_fullsized":"2024-03-26T03:06:20.268006+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":499,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.112676+0000","last_change":"2024-03-26T03:06:20.278182+0000","last_active":"2024-03-26T03:12:59.112676+0000","last_peered":"2024-03-26T03:12:59.112676+0000","last_clean":"2024-03-26T03:06:17.223227+0000","last_became_active":"2024-03-26T03:06:20.278182+0000","last_became_peered":"2024-03-26T03:06:20.278182+0000","last_unstale":"2024-03-26T03:12:59.112676+0000","last_undegraded":"2024-03-26T03:12:59.112676+0000","last_fullsized":"2024-03-26T03:06:20.266864+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.275045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.275045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"298'1290","reported_seq":1318,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.577768+0000","last_change":"2024-03-26T03:12:36.109267+0000","last_active":"2024-03-26T03:12:59.577768+0000","last_peered":"2024-03-26T03:12:59.577768+0000","last_clean":"2024-03-26T03:12:59.577768+0000","last_became_active":"2024-03-26T03:12:36.108830+0000","last_became_peered":"2024-03-26T03:12:36.108830+0000","last_unstale":"2024-03-26T03:12:59.577768+0000","last_undegraded":"2024-03-26T03:12:59.577768+0000","last_fullsized":"2024-03-26T03:12:59.577768+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:46.094589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84541440,"num_objects":1290,"num_object_clones":0,"num_object_copies":3870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1290,"num_write_kb":82560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102808+0000","last_change":"2024-03-26T03:11:15.024122+0000","last_active":"2024-03-26T03:12:59.102808+0000","last_peered":"2024-03-26T03:12:59.102808+0000","last_clean":"2024-03-26T03:12:59.102808+0000","last_became_active":"2024-03-26T02:59:00.934915+0000","last_became_peered":"2024-03-26T02:59:00.934915+0000","last_unstale":"2024-03-26T03:12:59.102808+0000","last_undegraded":"2024-03-26T03:12:59.102808+0000","last_fullsized":"2024-03-26T03:12:59.102808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.024002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.024002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":574,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107961+0000","last_change":"2024-03-26T03:11:11.116895+0000","last_active":"2024-03-26T03:12:59.107961+0000","last_peered":"2024-03-26T03:12:59.107961+0000","last_clean":"2024-03-26T03:12:59.107961+0000","last_became_active":"2024-03-26T02:58:58.929267+0000","last_became_peered":"2024-03-26T02:58:58.929267+0000","last_unstale":"2024-03-26T03:12:59.107961+0000","last_undegraded":"2024-03-26T03:12:59.107961+0000","last_fullsized":"2024-03-26T03:12:59.107961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:11:11.116642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:11.116642+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":512,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.102595+0000","last_change":"2024-03-26T03:06:20.306819+0000","last_active":"2024-03-26T03:12:59.102595+0000","last_peered":"2024-03-26T03:12:59.102595+0000","last_clean":"2024-03-26T03:06:17.222701+0000","last_became_active":"2024-03-26T03:06:20.306819+0000","last_became_peered":"2024-03-26T03:06:20.306819+0000","last_unstale":"2024-03-26T03:12:59.102595+0000","last_undegraded":"2024-03-26T03:12:59.102595+0000","last_fullsized":"2024-03-26T03:06:20.263412+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067662+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1384,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.254495+0000","last_change":"2024-03-26T03:11:48.252850+0000","last_active":"2024-03-26T03:12:59.254495+0000","last_peered":"2024-03-26T03:12:59.254495+0000","last_clean":"2024-03-26T03:12:59.254495+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:12:59.254495+0000","last_undegraded":"2024-03-26T03:12:59.254495+0000","last_fullsized":"2024-03-26T03:12:59.254495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:11:48.252717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.252717+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:03.591867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":832,"num_read_kb":827,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":500,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104988+0000","last_change":"2024-03-26T03:06:20.274488+0000","last_active":"2024-03-26T03:12:59.104988+0000","last_peered":"2024-03-26T03:12:59.104988+0000","last_clean":"2024-03-26T03:06:17.230895+0000","last_became_active":"2024-03-26T03:06:20.274488+0000","last_became_peered":"2024-03-26T03:06:20.274488+0000","last_unstale":"2024-03-26T03:12:59.104988+0000","last_undegraded":"2024-03-26T03:12:59.104988+0000","last_fullsized":"2024-03-26T03:06:20.269658+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017676+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":799,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104656+0000","last_change":"2024-03-26T03:10:46.593971+0000","last_active":"2024-03-26T03:12:59.104656+0000","last_peered":"2024-03-26T03:12:59.104656+0000","last_clean":"2024-03-26T03:12:59.104656+0000","last_became_active":"2024-03-26T03:10:39.379191+0000","last_became_peered":"2024-03-26T03:10:39.379191+0000","last_unstale":"2024-03-26T03:12:59.104656+0000","last_undegraded":"2024-03-26T03:12:59.104656+0000","last_fullsized":"2024-03-26T03:12:59.104656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:46.593847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:46.593847+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017714+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.a","version":"298'1238","reported_seq":1266,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.351985+0000","last_change":"2024-03-26T03:12:36.108858+0000","last_active":"2024-03-26T03:12:59.351985+0000","last_peered":"2024-03-26T03:12:59.351985+0000","last_clean":"2024-03-26T03:12:59.351985+0000","last_became_active":"2024-03-26T03:12:36.108555+0000","last_became_peered":"2024-03-26T03:12:36.108555+0000","last_unstale":"2024-03-26T03:12:59.351985+0000","last_undegraded":"2024-03-26T03:12:59.351985+0000","last_fullsized":"2024-03-26T03:12:59.351985+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:51.995213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81133568,"num_objects":1238,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":521,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.096266+0000","last_change":"2024-03-26T03:11:40.584646+0000","last_active":"2024-03-26T03:12:49.096266+0000","last_peered":"2024-03-26T03:12:49.096266+0000","last_clean":"2024-03-26T03:12:49.096266+0000","last_became_active":"2024-03-26T02:59:00.939897+0000","last_became_peered":"2024-03-26T02:59:00.939897+0000","last_unstale":"2024-03-26T03:12:49.096266+0000","last_undegraded":"2024-03-26T03:12:49.096266+0000","last_fullsized":"2024-03-26T03:12:49.096266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:40.584536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.584536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:49.955813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"231'58","reported_seq":767,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.106370+0000","last_change":"2024-03-26T03:06:20.312183+0000","last_active":"2024-03-26T03:12:59.106370+0000","last_peered":"2024-03-26T03:12:59.106370+0000","last_clean":"2024-03-26T03:06:17.232471+0000","last_became_active":"2024-03-26T03:06:20.312183+0000","last_became_peered":"2024-03-26T03:06:20.312183+0000","last_unstale":"2024-03-26T03:12:59.106370+0000","last_undegraded":"2024-03-26T03:06:20.298543+0000","last_fullsized":"2024-03-26T03:06:20.263409+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'45","last_scrub_stamp":"2024-03-26T03:05:25.716449+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.716449+0000","objects_scrubbed":11,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":334,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106674+0000","last_change":"2024-03-26T03:12:30.465375+0000","last_active":"2024-03-26T03:12:59.106674+0000","last_peered":"2024-03-26T03:12:59.106674+0000","last_clean":"2024-03-26T03:12:59.106674+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:12:59.106674+0000","last_undegraded":"2024-03-26T03:12:59.106674+0000","last_fullsized":"2024-03-26T03:12:59.106674+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:30.465292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.465292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:44.382195+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":209,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.103572+0000","last_change":"2024-03-26T03:06:20.271712+0000","last_active":"2024-03-26T03:12:59.103572+0000","last_peered":"2024-03-26T03:12:59.103572+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.271712+0000","last_became_peered":"2024-03-26T03:06:20.271712+0000","last_unstale":"2024-03-26T03:12:59.103572+0000","last_undegraded":"2024-03-26T03:12:59.103572+0000","last_fullsized":"2024-03-26T03:06:20.263095+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":629,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.097619+0000","last_change":"2024-03-26T03:12:21.520856+0000","last_active":"2024-03-26T03:12:49.097619+0000","last_peered":"2024-03-26T03:12:49.097619+0000","last_clean":"2024-03-26T03:12:49.097619+0000","last_became_active":"2024-03-26T03:10:39.376563+0000","last_became_peered":"2024-03-26T03:10:39.376563+0000","last_unstale":"2024-03-26T03:12:49.097619+0000","last_undegraded":"2024-03-26T03:12:49.097619+0000","last_fullsized":"2024-03-26T03:12:49.097619+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:21.520759+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:12:21.520759+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:28.569089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"298'1319","reported_seq":1347,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.605620+0000","last_change":"2024-03-26T03:12:36.112266+0000","last_active":"2024-03-26T03:12:59.605620+0000","last_peered":"2024-03-26T03:12:59.605620+0000","last_clean":"2024-03-26T03:12:59.605620+0000","last_became_active":"2024-03-26T03:12:36.111818+0000","last_became_peered":"2024-03-26T03:12:36.111818+0000","last_unstale":"2024-03-26T03:12:59.605620+0000","last_undegraded":"2024-03-26T03:12:59.605620+0000","last_fullsized":"2024-03-26T03:12:59.605620+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:54.454865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":84416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"71'8","reported_seq":708,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.099196+0000","last_change":"2024-03-26T03:12:39.462297+0000","last_active":"2024-03-26T03:12:49.099196+0000","last_peered":"2024-03-26T03:12:49.099196+0000","last_clean":"2024-03-26T03:12:49.099196+0000","last_became_active":"2024-03-26T02:59:00.939203+0000","last_became_peered":"2024-03-26T02:59:00.939203+0000","last_unstale":"2024-03-26T03:12:49.099196+0000","last_undegraded":"2024-03-26T03:12:49.099196+0000","last_fullsized":"2024-03-26T03:12:49.099196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:12:39.462209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:39.462209+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.415104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":580,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107259+0000","last_change":"2024-03-26T03:12:28.539186+0000","last_active":"2024-03-26T03:12:59.107259+0000","last_peered":"2024-03-26T03:12:59.107259+0000","last_clean":"2024-03-26T03:12:59.107259+0000","last_became_active":"2024-03-26T02:58:58.925713+0000","last_became_peered":"2024-03-26T02:58:58.925713+0000","last_unstale":"2024-03-26T03:12:59.107259+0000","last_undegraded":"2024-03-26T03:12:59.107259+0000","last_fullsized":"2024-03-26T03:12:59.107259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:12:28.539083+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.539083+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.950841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.106514+0000","last_change":"2024-03-26T03:11:10.075730+0000","last_active":"2024-03-26T03:12:59.106514+0000","last_peered":"2024-03-26T03:12:59.106514+0000","last_clean":"2024-03-26T03:12:59.106514+0000","last_became_active":"2024-03-26T02:58:56.929352+0000","last_became_peered":"2024-03-26T02:58:56.929352+0000","last_unstale":"2024-03-26T03:12:59.106514+0000","last_undegraded":"2024-03-26T03:12:59.106514+0000","last_fullsized":"2024-03-26T03:12:59.106514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:10.075607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:10.075607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":496,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.096661+0000","last_change":"2024-03-26T03:06:20.307556+0000","last_active":"2024-03-26T03:12:49.096661+0000","last_peered":"2024-03-26T03:12:49.096661+0000","last_clean":"2024-03-26T03:06:17.228050+0000","last_became_active":"2024-03-26T03:06:20.307556+0000","last_became_peered":"2024-03-26T03:06:20.307556+0000","last_unstale":"2024-03-26T03:12:49.096661+0000","last_undegraded":"2024-03-26T03:12:49.096661+0000","last_fullsized":"2024-03-26T03:06:20.264125+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416872+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":590,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.108025+0000","last_change":"2024-03-26T03:10:39.371201+0000","last_active":"2024-03-26T03:12:59.108025+0000","last_peered":"2024-03-26T03:12:59.108025+0000","last_clean":"2024-03-26T03:06:17.226273+0000","last_became_active":"2024-03-26T03:10:39.371201+0000","last_became_peered":"2024-03-26T03:10:39.371201+0000","last_unstale":"2024-03-26T03:12:59.108025+0000","last_undegraded":"2024-03-26T03:12:59.108025+0000","last_fullsized":"2024-03-26T03:10:39.361046+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"298'1264","reported_seq":1289,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.355383+0000","last_change":"2024-03-26T03:12:53.131517+0000","last_active":"2024-03-26T03:12:59.355383+0000","last_peered":"2024-03-26T03:12:59.355383+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"2024-03-26T03:12:36.106277+0000","last_became_peered":"2024-03-26T03:12:36.106277+0000","last_unstale":"2024-03-26T03:12:59.355383+0000","last_undegraded":"2024-03-26T03:12:53.130568+0000","last_fullsized":"2024-03-26T03:12:36.101554+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:51.444373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82837504,"num_objects":1264,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":80896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1264}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":400,"reported_epoch":296,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:49.097696+0000","last_change":"2024-03-26T03:12:24.618622+0000","last_active":"2024-03-26T03:12:49.097696+0000","last_peered":"2024-03-26T03:12:49.097696+0000","last_clean":"2024-03-26T03:12:49.097696+0000","last_became_active":"2024-03-26T03:10:44.081411+0000","last_became_peered":"2024-03-26T03:10:44.081411+0000","last_unstale":"2024-03-26T03:12:49.097696+0000","last_undegraded":"2024-03-26T03:12:49.097696+0000","last_fullsized":"2024-03-26T03:12:49.097696+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:24.618461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:24.618461+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:34.569655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,5],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":541,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.101722+0000","last_change":"2024-03-26T03:12:18.908516+0000","last_active":"2024-03-26T03:12:59.101722+0000","last_peered":"2024-03-26T03:12:59.101722+0000","last_clean":"2024-03-26T03:12:59.101722+0000","last_became_active":"2024-03-26T02:59:00.934105+0000","last_became_peered":"2024-03-26T02:59:00.934105+0000","last_unstale":"2024-03-26T03:12:59.101722+0000","last_undegraded":"2024-03-26T03:12:59.101722+0000","last_fullsized":"2024-03-26T03:12:59.101722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.908406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.908406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:25.724804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":576,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.102211+0000","last_change":"2024-03-26T03:06:20.280064+0000","last_active":"2024-03-26T03:12:59.102211+0000","last_peered":"2024-03-26T03:12:59.102211+0000","last_clean":"2024-03-26T03:06:17.219171+0000","last_became_active":"2024-03-26T03:06:20.280064+0000","last_became_peered":"2024-03-26T03:06:20.280064+0000","last_unstale":"2024-03-26T03:12:59.102211+0000","last_undegraded":"2024-03-26T03:06:20.262252+0000","last_fullsized":"2024-03-26T03:06:20.262100+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'24","last_scrub_stamp":"2024-03-26T03:03:50.379947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.379947+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.254953+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":571,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102603+0000","last_change":"2024-03-26T03:12:33.067623+0000","last_active":"2024-03-26T03:12:59.102603+0000","last_peered":"2024-03-26T03:12:59.102603+0000","last_clean":"2024-03-26T03:12:59.102603+0000","last_became_active":"2024-03-26T03:10:08.555235+0000","last_became_peered":"2024-03-26T03:10:08.555235+0000","last_unstale":"2024-03-26T03:12:59.102603+0000","last_undegraded":"2024-03-26T03:12:59.102603+0000","last_fullsized":"2024-03-26T03:12:59.102603+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:12:33.067512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:33.067512+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.040390+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":209,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104488+0000","last_change":"2024-03-26T03:06:20.305845+0000","last_active":"2024-03-26T03:12:59.104488+0000","last_peered":"2024-03-26T03:12:59.104488+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.305845+0000","last_became_peered":"2024-03-26T03:06:20.305845+0000","last_unstale":"2024-03-26T03:12:59.104488+0000","last_undegraded":"2024-03-26T03:12:59.104488+0000","last_fullsized":"2024-03-26T03:06:20.265764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":628,"reported_epoch":298,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:59.106698+0000","last_change":"2024-03-26T03:12:22.342630+0000","last_active":"2024-03-26T03:12:59.106698+0000","last_peered":"2024-03-26T03:12:59.106698+0000","last_clean":"2024-03-26T03:12:59.106698+0000","last_became_active":"2024-03-26T03:10:42.404423+0000","last_became_peered":"2024-03-26T03:10:42.404423+0000","last_unstale":"2024-03-26T03:12:59.106698+0000","last_undegraded":"2024-03-26T03:12:59.106698+0000","last_fullsized":"2024-03-26T03:12:59.106698+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:22.342472+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.342472+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.417985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":552,"reported_epoch":296,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:49.099026+0000","last_change":"2024-03-26T03:06:20.309586+0000","last_active":"2024-03-26T03:12:49.099026+0000","last_peered":"2024-03-26T03:12:49.099026+0000","last_clean":"2024-03-26T03:06:17.233575+0000","last_became_active":"2024-03-26T03:06:20.309586+0000","last_became_peered":"2024-03-26T03:06:20.309586+0000","last_unstale":"2024-03-26T03:12:49.099026+0000","last_undegraded":"2024-03-26T03:06:20.301993+0000","last_fullsized":"2024-03-26T03:06:20.301860+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416946+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":555,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.101650+0000","last_change":"2024-03-26T03:12:23.943984+0000","last_active":"2024-03-26T03:12:59.101650+0000","last_peered":"2024-03-26T03:12:59.101650+0000","last_clean":"2024-03-26T03:12:59.101650+0000","last_became_active":"2024-03-26T02:58:56.924767+0000","last_became_peered":"2024-03-26T02:58:56.924767+0000","last_unstale":"2024-03-26T03:12:59.101650+0000","last_undegraded":"2024-03-26T03:12:59.101650+0000","last_fullsized":"2024-03-26T03:12:59.101650+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:12:23.943885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.943885+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:44.438967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":209,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.107571+0000","last_change":"2024-03-26T03:06:20.313845+0000","last_active":"2024-03-26T03:12:59.107571+0000","last_peered":"2024-03-26T03:12:59.107571+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.313845+0000","last_became_peered":"2024-03-26T03:06:20.313845+0000","last_unstale":"2024-03-26T03:12:59.107571+0000","last_undegraded":"2024-03-26T03:12:59.107571+0000","last_fullsized":"2024-03-26T03:06:20.306115+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":528,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102271+0000","last_change":"2024-03-26T03:11:38.225205+0000","last_active":"2024-03-26T03:12:59.102271+0000","last_peered":"2024-03-26T03:12:59.102271+0000","last_clean":"2024-03-26T03:12:59.102271+0000","last_became_active":"2024-03-26T02:59:02.942456+0000","last_became_peered":"2024-03-26T02:59:02.942456+0000","last_unstale":"2024-03-26T03:12:59.102271+0000","last_undegraded":"2024-03-26T03:12:59.102271+0000","last_fullsized":"2024-03-26T03:12:59.102271+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:38.225092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:11:38.225092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":403,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.111475+0000","last_change":"2024-03-26T03:12:16.847864+0000","last_active":"2024-03-26T03:12:59.111475+0000","last_peered":"2024-03-26T03:12:59.111475+0000","last_clean":"2024-03-26T03:12:59.111475+0000","last_became_active":"2024-03-26T03:10:44.085377+0000","last_became_peered":"2024-03-26T03:10:44.085377+0000","last_unstale":"2024-03-26T03:12:59.111475+0000","last_undegraded":"2024-03-26T03:12:59.111475+0000","last_fullsized":"2024-03-26T03:12:59.111475+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:16.847722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:16.847722+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:33.297125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":258,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.105442+0000","last_change":"2024-03-26T03:06:20.303853+0000","last_active":"2024-03-26T03:12:59.105442+0000","last_peered":"2024-03-26T03:12:59.105442+0000","last_clean":"2024-03-26T02:59:04.312282+0000","last_became_active":"2024-03-26T03:06:20.303853+0000","last_became_peered":"2024-03-26T03:06:20.303853+0000","last_unstale":"2024-03-26T03:12:59.105442+0000","last_undegraded":"2024-03-26T03:06:20.268138+0000","last_fullsized":"2024-03-26T03:06:20.268024+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":541,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107220+0000","last_change":"2024-03-26T03:12:18.094777+0000","last_active":"2024-03-26T03:12:59.107220+0000","last_peered":"2024-03-26T03:12:59.107220+0000","last_clean":"2024-03-26T03:12:59.107220+0000","last_became_active":"2024-03-26T02:58:56.926489+0000","last_became_peered":"2024-03-26T02:58:56.926489+0000","last_unstale":"2024-03-26T03:12:59.107220+0000","last_undegraded":"2024-03-26T03:12:59.107220+0000","last_fullsized":"2024-03-26T03:12:59.107220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.094677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.094677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:29.032092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":502,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.112287+0000","last_change":"2024-03-26T03:06:20.305029+0000","last_active":"2024-03-26T03:12:59.112287+0000","last_peered":"2024-03-26T03:12:59.112287+0000","last_clean":"2024-03-26T03:06:19.011627+0000","last_became_active":"2024-03-26T03:06:20.305029+0000","last_became_peered":"2024-03-26T03:06:20.305029+0000","last_unstale":"2024-03-26T03:12:59.112287+0000","last_undegraded":"2024-03-26T03:12:59.112287+0000","last_fullsized":"2024-03-26T03:06:20.266454+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.507239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.507239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":512,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104769+0000","last_change":"2024-03-26T03:11:45.964602+0000","last_active":"2024-03-26T03:12:59.104769+0000","last_peered":"2024-03-26T03:12:59.104769+0000","last_clean":"2024-03-26T03:12:59.104769+0000","last_became_active":"2024-03-26T02:59:03.208213+0000","last_became_peered":"2024-03-26T02:59:03.208213+0000","last_unstale":"2024-03-26T03:12:59.104769+0000","last_undegraded":"2024-03-26T03:12:59.104769+0000","last_fullsized":"2024-03-26T03:12:59.104769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:45.964365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:45.964365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:11.924887+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":629,"reported_epoch":296,"state":"active+clean+remapped","last_fresh":"2024-03-26T03:12:49.097524+0000","last_change":"2024-03-26T03:12:23.572625+0000","last_active":"2024-03-26T03:12:49.097524+0000","last_peered":"2024-03-26T03:12:49.097524+0000","last_clean":"2024-03-26T03:12:49.097524+0000","last_became_active":"2024-03-26T03:10:42.404630+0000","last_became_peered":"2024-03-26T03:10:42.404630+0000","last_unstale":"2024-03-26T03:12:49.097524+0000","last_undegraded":"2024-03-26T03:12:49.097524+0000","last_fullsized":"2024-03-26T03:12:49.097524+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:23.572446+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.572446+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:26.325433+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,2],"avail_no_missing":["0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":457,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.106607+0000","last_change":"2024-03-26T03:06:20.306239+0000","last_active":"2024-03-26T03:12:59.106607+0000","last_peered":"2024-03-26T03:12:59.106607+0000","last_clean":"2024-03-26T03:03:52.386088+0000","last_became_active":"2024-03-26T03:06:20.306239+0000","last_became_peered":"2024-03-26T03:06:20.306239+0000","last_unstale":"2024-03-26T03:12:59.106607+0000","last_undegraded":"2024-03-26T03:06:20.298748+0000","last_fullsized":"2024-03-26T03:06:20.298635+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.111957+0000","last_change":"2024-03-26T03:06:20.278192+0000","last_active":"2024-03-26T03:12:59.111957+0000","last_peered":"2024-03-26T03:12:59.111957+0000","last_clean":"2024-03-26T03:06:17.583134+0000","last_became_active":"2024-03-26T03:06:20.278192+0000","last_became_peered":"2024-03-26T03:06:20.278192+0000","last_unstale":"2024-03-26T03:12:59.111957+0000","last_undegraded":"2024-03-26T03:12:59.111957+0000","last_fullsized":"2024-03-26T03:06:20.267494+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"201'9","reported_seq":709,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.111930+0000","last_change":"2024-03-26T03:11:38.754437+0000","last_active":"2024-03-26T03:12:59.111930+0000","last_peered":"2024-03-26T03:12:59.111930+0000","last_clean":"2024-03-26T03:12:59.111930+0000","last_became_active":"2024-03-26T02:59:00.933917+0000","last_became_peered":"2024-03-26T02:59:00.933917+0000","last_unstale":"2024-03-26T03:12:59.111930+0000","last_undegraded":"2024-03-26T03:12:59.111930+0000","last_fullsized":"2024-03-26T03:12:59.111930+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:11:38.754331+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:11:38.754331+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.666167+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":495,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.098707+0000","last_change":"2024-03-26T03:06:20.307922+0000","last_active":"2024-03-26T03:12:49.098707+0000","last_peered":"2024-03-26T03:12:49.098707+0000","last_clean":"2024-03-26T03:06:17.233386+0000","last_became_active":"2024-03-26T03:06:20.307922+0000","last_became_peered":"2024-03-26T03:06:20.307922+0000","last_unstale":"2024-03-26T03:12:49.098707+0000","last_undegraded":"2024-03-26T03:12:49.098707+0000","last_fullsized":"2024-03-26T03:06:20.298142+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416857+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":799,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104128+0000","last_change":"2024-03-26T03:10:49.610796+0000","last_active":"2024-03-26T03:12:59.104128+0000","last_peered":"2024-03-26T03:12:59.104128+0000","last_clean":"2024-03-26T03:12:59.104128+0000","last_became_active":"2024-03-26T03:10:44.085632+0000","last_became_peered":"2024-03-26T03:10:44.085632+0000","last_unstale":"2024-03-26T03:12:59.104128+0000","last_undegraded":"2024-03-26T03:12:59.104128+0000","last_fullsized":"2024-03-26T03:12:59.104128+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:10:49.610676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:10:49.610676+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017775+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":352,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.104883+0000","last_change":"2024-03-26T03:06:20.278295+0000","last_active":"2024-03-26T03:12:59.104883+0000","last_peered":"2024-03-26T03:12:59.104883+0000","last_clean":"2024-03-26T03:06:17.230804+0000","last_became_active":"2024-03-26T03:06:20.278295+0000","last_became_peered":"2024-03-26T03:06:20.278295+0000","last_unstale":"2024-03-26T03:12:59.104883+0000","last_undegraded":"2024-03-26T03:06:20.270115+0000","last_fullsized":"2024-03-26T03:06:20.269425+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":499,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.098814+0000","last_change":"2024-03-26T03:06:20.303007+0000","last_active":"2024-03-26T03:12:49.098814+0000","last_peered":"2024-03-26T03:12:49.098814+0000","last_clean":"2024-03-26T03:06:17.229113+0000","last_became_active":"2024-03-26T03:06:20.303007+0000","last_became_peered":"2024-03-26T03:06:20.303007+0000","last_unstale":"2024-03-26T03:12:49.098814+0000","last_undegraded":"2024-03-26T03:12:49.098814+0000","last_fullsized":"2024-03-26T03:06:20.264303+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416843+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"199'9","reported_seq":315,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.104986+0000","last_change":"2024-03-26T03:06:20.312626+0000","last_active":"2024-03-26T03:12:59.104986+0000","last_peered":"2024-03-26T03:12:59.104986+0000","last_clean":"2024-03-26T02:59:04.144092+0000","last_became_active":"2024-03-26T03:06:20.312626+0000","last_became_peered":"2024-03-26T03:06:20.312626+0000","last_unstale":"2024-03-26T03:12:59.104986+0000","last_undegraded":"2024-03-26T03:06:20.268320+0000","last_fullsized":"2024-03-26T03:06:20.268196+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":511,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102802+0000","last_change":"2024-03-26T03:11:07.697991+0000","last_active":"2024-03-26T03:12:59.102802+0000","last_peered":"2024-03-26T03:12:59.102802+0000","last_clean":"2024-03-26T03:12:59.102802+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:12:59.102802+0000","last_undegraded":"2024-03-26T03:12:59.102802+0000","last_fullsized":"2024-03-26T03:12:59.102802+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:11:07.697855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:07.697855+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255151+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":558,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107195+0000","last_change":"2024-03-26T03:11:46.227621+0000","last_active":"2024-03-26T03:12:59.107195+0000","last_peered":"2024-03-26T03:12:59.107195+0000","last_clean":"2024-03-26T03:12:59.107195+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:12:59.107195+0000","last_undegraded":"2024-03-26T03:12:59.107195+0000","last_fullsized":"2024-03-26T03:12:59.107195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.227482+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:11.154098+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":511,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.107214+0000","last_change":"2024-03-26T03:06:20.276441+0000","last_active":"2024-03-26T03:12:59.107214+0000","last_peered":"2024-03-26T03:12:59.107214+0000","last_clean":"2024-03-26T03:06:17.225787+0000","last_became_active":"2024-03-26T03:06:20.276441+0000","last_became_peered":"2024-03-26T03:06:20.276441+0000","last_unstale":"2024-03-26T03:12:59.107214+0000","last_undegraded":"2024-03-26T03:12:59.107214+0000","last_fullsized":"2024-03-26T03:06:20.269771+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":205,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.099144+0000","last_change":"2024-03-26T03:06:20.320547+0000","last_active":"2024-03-26T03:12:49.099144+0000","last_peered":"2024-03-26T03:12:49.099144+0000","last_clean":"2024-03-26T02:58:59.914157+0000","last_became_active":"2024-03-26T03:06:20.320547+0000","last_became_peered":"2024-03-26T03:06:20.320547+0000","last_unstale":"2024-03-26T03:12:49.099144+0000","last_undegraded":"2024-03-26T03:12:49.099144+0000","last_fullsized":"2024-03-26T03:06:20.298525+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":528,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.096611+0000","last_change":"2024-03-26T03:12:38.484145+0000","last_active":"2024-03-26T03:12:49.096611+0000","last_peered":"2024-03-26T03:12:49.096611+0000","last_clean":"2024-03-26T03:12:49.096611+0000","last_became_active":"2024-03-26T02:59:02.946806+0000","last_became_peered":"2024-03-26T02:59:02.946806+0000","last_unstale":"2024-03-26T03:12:49.096611+0000","last_undegraded":"2024-03-26T03:12:49.096611+0000","last_fullsized":"2024-03-26T03:12:49.096611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:38.484038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:38.484038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:41.026400+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":557,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.101845+0000","last_change":"2024-03-26T03:11:42.830091+0000","last_active":"2024-03-26T03:12:59.101845+0000","last_peered":"2024-03-26T03:12:59.101845+0000","last_clean":"2024-03-26T03:12:59.101845+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:12:59.101845+0000","last_undegraded":"2024-03-26T03:12:59.101845+0000","last_fullsized":"2024-03-26T03:12:59.101845+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:11:42.829912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.829912+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:08.247461+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":540,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107305+0000","last_change":"2024-03-26T03:12:18.319272+0000","last_active":"2024-03-26T03:12:59.107305+0000","last_peered":"2024-03-26T03:12:59.107305+0000","last_clean":"2024-03-26T03:12:59.107305+0000","last_became_active":"2024-03-26T02:58:56.929038+0000","last_became_peered":"2024-03-26T02:58:56.929038+0000","last_unstale":"2024-03-26T03:12:59.107305+0000","last_undegraded":"2024-03-26T03:12:59.107305+0000","last_fullsized":"2024-03-26T03:12:59.107305+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.319189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.319189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:24.605611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":495,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.107368+0000","last_change":"2024-03-26T03:06:20.312469+0000","last_active":"2024-03-26T03:12:59.107368+0000","last_peered":"2024-03-26T03:12:59.107368+0000","last_clean":"2024-03-26T03:06:17.250052+0000","last_became_active":"2024-03-26T03:06:20.312469+0000","last_became_peered":"2024-03-26T03:06:20.312469+0000","last_unstale":"2024-03-26T03:12:59.107368+0000","last_undegraded":"2024-03-26T03:12:59.107368+0000","last_fullsized":"2024-03-26T03:06:20.304941+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:30.742238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:30.742238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":205,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.098388+0000","last_change":"2024-03-26T03:06:20.303766+0000","last_active":"2024-03-26T03:12:49.098388+0000","last_peered":"2024-03-26T03:12:49.098388+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.303766+0000","last_became_peered":"2024-03-26T03:06:20.303766+0000","last_unstale":"2024-03-26T03:12:49.098388+0000","last_undegraded":"2024-03-26T03:12:49.098388+0000","last_fullsized":"2024-03-26T03:06:20.298942+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:54.412346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:03:54.412346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":240,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.106417+0000","last_change":"2024-03-26T03:06:20.306787+0000","last_active":"2024-03-26T03:12:59.106417+0000","last_peered":"2024-03-26T03:12:59.106417+0000","last_clean":"2024-03-26T02:59:04.147254+0000","last_became_active":"2024-03-26T03:06:20.306787+0000","last_became_peered":"2024-03-26T03:06:20.306787+0000","last_unstale":"2024-03-26T03:12:59.106417+0000","last_undegraded":"2024-03-26T03:06:20.263501+0000","last_fullsized":"2024-03-26T03:06:20.263336+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":547,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.112456+0000","last_change":"2024-03-26T03:12:23.825797+0000","last_active":"2024-03-26T03:12:59.112456+0000","last_peered":"2024-03-26T03:12:59.112456+0000","last_clean":"2024-03-26T03:12:59.112456+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:12:59.112456+0000","last_undegraded":"2024-03-26T03:12:59.112456+0000","last_fullsized":"2024-03-26T03:12:59.112456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:23.825709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.825709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:32.106344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107092+0000","last_change":"2024-03-26T03:11:48.104641+0000","last_active":"2024-03-26T03:12:59.107092+0000","last_peered":"2024-03-26T03:12:59.107092+0000","last_clean":"2024-03-26T03:12:59.107092+0000","last_became_active":"2024-03-26T02:59:00.940152+0000","last_became_peered":"2024-03-26T02:59:00.940152+0000","last_unstale":"2024-03-26T03:12:59.107092+0000","last_undegraded":"2024-03-26T03:12:59.107092+0000","last_fullsized":"2024-03-26T03:12:59.107092+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.104415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:19.233223+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.104415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":520,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104708+0000","last_change":"2024-03-26T03:11:01.548791+0000","last_active":"2024-03-26T03:12:59.104708+0000","last_peered":"2024-03-26T03:12:59.104708+0000","last_clean":"2024-03-26T03:12:59.104708+0000","last_became_active":"2024-03-26T02:59:02.940834+0000","last_became_peered":"2024-03-26T02:59:02.940834+0000","last_unstale":"2024-03-26T03:12:59.104708+0000","last_undegraded":"2024-03-26T03:12:59.104708+0000","last_fullsized":"2024-03-26T03:12:59.104708+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.548673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.548673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":405,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102068+0000","last_change":"2024-03-26T03:12:21.953709+0000","last_active":"2024-03-26T03:12:59.102068+0000","last_peered":"2024-03-26T03:12:59.102068+0000","last_clean":"2024-03-26T03:12:59.102068+0000","last_became_active":"2024-03-26T03:10:44.079530+0000","last_became_peered":"2024-03-26T03:10:44.079530+0000","last_unstale":"2024-03-26T03:12:59.102068+0000","last_undegraded":"2024-03-26T03:12:59.102068+0000","last_fullsized":"2024-03-26T03:12:59.102068+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:21.953552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:21.953552+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:39.750935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":444,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.104620+0000","last_change":"2024-03-26T03:06:20.275080+0000","last_active":"2024-03-26T03:12:59.104620+0000","last_peered":"2024-03-26T03:12:59.104620+0000","last_clean":"2024-03-26T03:03:50.321804+0000","last_became_active":"2024-03-26T03:06:20.275080+0000","last_became_peered":"2024-03-26T03:06:20.275080+0000","last_unstale":"2024-03-26T03:12:59.104620+0000","last_undegraded":"2024-03-26T03:06:20.266256+0000","last_fullsized":"2024-03-26T03:06:20.266141+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.108114+0000","last_change":"2024-03-26T03:11:42.129279+0000","last_active":"2024-03-26T03:12:59.108114+0000","last_peered":"2024-03-26T03:12:59.108114+0000","last_clean":"2024-03-26T03:12:59.108114+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:12:59.108114+0000","last_undegraded":"2024-03-26T03:12:59.108114+0000","last_fullsized":"2024-03-26T03:12:59.108114+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:42.129153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.129153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104672+0000","last_change":"2024-03-26T03:12:30.019328+0000","last_active":"2024-03-26T03:12:59.104672+0000","last_peered":"2024-03-26T03:12:59.104672+0000","last_clean":"2024-03-26T03:12:59.104672+0000","last_became_active":"2024-03-26T02:59:00.935458+0000","last_became_peered":"2024-03-26T02:59:00.935458+0000","last_unstale":"2024-03-26T03:12:59.104672+0000","last_undegraded":"2024-03-26T03:12:59.104672+0000","last_fullsized":"2024-03-26T03:12:59.104672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:30.019226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.019226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:31.273674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":205,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.098061+0000","last_change":"2024-03-26T03:06:20.301091+0000","last_active":"2024-03-26T03:12:49.098061+0000","last_peered":"2024-03-26T03:12:49.098061+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.301091+0000","last_became_peered":"2024-03-26T03:06:20.301091+0000","last_unstale":"2024-03-26T03:12:49.098061+0000","last_undegraded":"2024-03-26T03:12:49.098061+0000","last_fullsized":"2024-03-26T03:06:20.264654+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":546,"reported_epoch":296,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:49.099310+0000","last_change":"2024-03-26T03:06:20.309962+0000","last_active":"2024-03-26T03:12:49.099310+0000","last_peered":"2024-03-26T03:12:49.099310+0000","last_clean":"2024-03-26T03:06:17.233775+0000","last_became_active":"2024-03-26T03:06:20.309962+0000","last_became_peered":"2024-03-26T03:06:20.309962+0000","last_unstale":"2024-03-26T03:12:49.099310+0000","last_undegraded":"2024-03-26T03:06:20.302780+0000","last_fullsized":"2024-03-26T03:06:20.302331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":547,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.111295+0000","last_change":"2024-03-26T03:12:22.805078+0000","last_active":"2024-03-26T03:12:59.111295+0000","last_peered":"2024-03-26T03:12:59.111295+0000","last_clean":"2024-03-26T03:12:59.111295+0000","last_became_active":"2024-03-26T02:58:56.924610+0000","last_became_peered":"2024-03-26T02:58:56.924610+0000","last_unstale":"2024-03-26T03:12:59.111295+0000","last_undegraded":"2024-03-26T03:12:59.111295+0000","last_fullsized":"2024-03-26T03:12:59.111295+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:22.804999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.804999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:32.842763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"71'8","reported_seq":713,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.108294+0000","last_change":"2024-03-26T03:12:38.101128+0000","last_active":"2024-03-26T03:12:59.108294+0000","last_peered":"2024-03-26T03:12:59.108294+0000","last_clean":"2024-03-26T03:12:59.108294+0000","last_became_active":"2024-03-26T02:59:00.938662+0000","last_became_peered":"2024-03-26T02:59:00.938662+0000","last_unstale":"2024-03-26T03:12:59.108294+0000","last_undegraded":"2024-03-26T03:12:59.108294+0000","last_fullsized":"2024-03-26T03:12:59.108294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:12:38.100984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:38.100984+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:59.544665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.105576+0000","last_change":"2024-03-26T03:12:34.142076+0000","last_active":"2024-03-26T03:12:59.105576+0000","last_peered":"2024-03-26T03:12:59.105576+0000","last_clean":"2024-03-26T03:12:59.105576+0000","last_became_active":"2024-03-26T02:59:02.943361+0000","last_became_peered":"2024-03-26T02:59:02.943361+0000","last_unstale":"2024-03-26T03:12:59.105576+0000","last_undegraded":"2024-03-26T03:12:59.105576+0000","last_fullsized":"2024-03-26T03:12:59.105576+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.141917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:35.118061+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":572,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.108346+0000","last_change":"2024-03-26T03:11:40.118723+0000","last_active":"2024-03-26T03:12:59.108346+0000","last_peered":"2024-03-26T03:12:59.108346+0000","last_clean":"2024-03-26T03:12:59.108346+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:12:59.108346+0000","last_undegraded":"2024-03-26T03:12:59.108346+0000","last_fullsized":"2024-03-26T03:12:59.108346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:11:40.118572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:40.118572+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":217,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.106855+0000","last_change":"2024-03-26T03:06:20.302609+0000","last_active":"2024-03-26T03:12:59.106855+0000","last_peered":"2024-03-26T03:12:59.106855+0000","last_clean":"2024-03-26T02:58:56.925782+0000","last_became_active":"2024-03-26T03:06:20.302609+0000","last_became_peered":"2024-03-26T03:06:20.302609+0000","last_unstale":"2024-03-26T03:12:59.106855+0000","last_undegraded":"2024-03-26T03:06:20.272722+0000","last_fullsized":"2024-03-26T03:06:20.272608+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":529,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.099339+0000","last_change":"2024-03-26T03:11:49.540875+0000","last_active":"2024-03-26T03:12:49.099339+0000","last_peered":"2024-03-26T03:12:49.099339+0000","last_clean":"2024-03-26T03:12:49.099339+0000","last_became_active":"2024-03-26T02:59:00.936358+0000","last_became_peered":"2024-03-26T02:59:00.936358+0000","last_unstale":"2024-03-26T03:12:49.099339+0000","last_undegraded":"2024-03-26T03:12:49.099339+0000","last_fullsized":"2024-03-26T03:12:49.099339+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.540763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.540763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:52.443627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":520,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107139+0000","last_change":"2024-03-26T03:12:36.651831+0000","last_active":"2024-03-26T03:12:59.107139+0000","last_peered":"2024-03-26T03:12:59.107139+0000","last_clean":"2024-03-26T03:12:59.107139+0000","last_became_active":"2024-03-26T02:59:02.938408+0000","last_became_peered":"2024-03-26T02:59:02.938408+0000","last_unstale":"2024-03-26T03:12:59.107139+0000","last_undegraded":"2024-03-26T03:12:59.107139+0000","last_fullsized":"2024-03-26T03:12:59.107139+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:36.651720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:36.651720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:52.701462+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":550,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.102571+0000","last_change":"2024-03-26T03:06:20.316914+0000","last_active":"2024-03-26T03:12:59.102571+0000","last_peered":"2024-03-26T03:12:59.102571+0000","last_clean":"2024-03-26T03:06:17.226622+0000","last_became_active":"2024-03-26T03:06:20.316914+0000","last_became_peered":"2024-03-26T03:06:20.316914+0000","last_unstale":"2024-03-26T03:12:59.102571+0000","last_undegraded":"2024-03-26T03:06:20.304159+0000","last_fullsized":"2024-03-26T03:06:20.304028+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":540,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107318+0000","last_change":"2024-03-26T03:11:16.283604+0000","last_active":"2024-03-26T03:12:59.107318+0000","last_peered":"2024-03-26T03:12:59.107318+0000","last_clean":"2024-03-26T03:12:59.107318+0000","last_became_active":"2024-03-26T02:58:56.921783+0000","last_became_peered":"2024-03-26T02:58:56.921783+0000","last_unstale":"2024-03-26T03:12:59.107318+0000","last_undegraded":"2024-03-26T03:12:59.107318+0000","last_fullsized":"2024-03-26T03:12:59.107318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.283475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.283475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107815+0000","last_change":"2024-03-26T03:12:35.484444+0000","last_active":"2024-03-26T03:12:59.107815+0000","last_peered":"2024-03-26T03:12:59.107815+0000","last_clean":"2024-03-26T03:12:59.107815+0000","last_became_active":"2024-03-26T02:59:00.935399+0000","last_became_peered":"2024-03-26T02:59:00.935399+0000","last_unstale":"2024-03-26T03:12:59.107815+0000","last_undegraded":"2024-03-26T03:12:59.107815+0000","last_fullsized":"2024-03-26T03:12:59.107815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.484358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.484358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:36.767951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.103262+0000","last_change":"2024-03-26T03:11:15.795645+0000","last_active":"2024-03-26T03:12:59.103262+0000","last_peered":"2024-03-26T03:12:59.103262+0000","last_clean":"2024-03-26T03:12:59.103262+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:12:59.103262+0000","last_undegraded":"2024-03-26T03:12:59.103262+0000","last_fullsized":"2024-03-26T03:12:59.103262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:15.795433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:15.795433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":272,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.107658+0000","last_change":"2024-03-26T03:06:20.305753+0000","last_active":"2024-03-26T03:12:59.107658+0000","last_peered":"2024-03-26T03:12:59.107658+0000","last_clean":"2024-03-26T02:59:04.360178+0000","last_became_active":"2024-03-26T03:06:20.305753+0000","last_became_peered":"2024-03-26T03:06:20.305753+0000","last_unstale":"2024-03-26T03:12:59.107658+0000","last_undegraded":"2024-03-26T03:06:20.299917+0000","last_fullsized":"2024-03-26T03:06:20.299798+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":209,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104991+0000","last_change":"2024-03-26T03:06:20.274265+0000","last_active":"2024-03-26T03:12:59.104991+0000","last_peered":"2024-03-26T03:12:59.104991+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.274265+0000","last_became_peered":"2024-03-26T03:06:20.274265+0000","last_unstale":"2024-03-26T03:12:59.104991+0000","last_undegraded":"2024-03-26T03:12:59.104991+0000","last_fullsized":"2024-03-26T03:06:20.267002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":493,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.102774+0000","last_change":"2024-03-26T03:06:20.308466+0000","last_active":"2024-03-26T03:12:59.102774+0000","last_peered":"2024-03-26T03:12:59.102774+0000","last_clean":"2024-03-26T03:06:17.220725+0000","last_became_active":"2024-03-26T03:06:20.308466+0000","last_became_peered":"2024-03-26T03:06:20.308466+0000","last_unstale":"2024-03-26T03:12:59.102774+0000","last_undegraded":"2024-03-26T03:12:59.102774+0000","last_fullsized":"2024-03-26T03:06:20.264401+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:53.357766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:53.357766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.254987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":210,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.108443+0000","last_change":"2024-03-26T03:06:20.276371+0000","last_active":"2024-03-26T03:12:59.108443+0000","last_peered":"2024-03-26T03:12:59.108443+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.276371+0000","last_became_peered":"2024-03-26T03:06:20.276371+0000","last_unstale":"2024-03-26T03:12:59.108443+0000","last_undegraded":"2024-03-26T03:12:59.108443+0000","last_fullsized":"2024-03-26T03:06:20.269650+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":560,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.101926+0000","last_change":"2024-03-26T03:06:20.309125+0000","last_active":"2024-03-26T03:12:59.101926+0000","last_peered":"2024-03-26T03:12:59.101926+0000","last_clean":"2024-03-26T03:06:17.219533+0000","last_became_active":"2024-03-26T03:06:20.309125+0000","last_became_peered":"2024-03-26T03:06:20.309125+0000","last_unstale":"2024-03-26T03:12:59.101926+0000","last_undegraded":"2024-03-26T03:06:20.263784+0000","last_fullsized":"2024-03-26T03:06:20.262465+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255033+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104674+0000","last_change":"2024-03-26T03:12:18.931056+0000","last_active":"2024-03-26T03:12:59.104674+0000","last_peered":"2024-03-26T03:12:59.104674+0000","last_clean":"2024-03-26T03:12:59.104674+0000","last_became_active":"2024-03-26T02:58:56.922445+0000","last_became_peered":"2024-03-26T02:58:56.922445+0000","last_unstale":"2024-03-26T03:12:59.104674+0000","last_undegraded":"2024-03-26T03:12:59.104674+0000","last_fullsized":"2024-03-26T03:12:59.104674+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.930944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.930944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:26.735550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":539,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.111881+0000","last_change":"2024-03-26T03:12:34.667196+0000","last_active":"2024-03-26T03:12:59.111881+0000","last_peered":"2024-03-26T03:12:59.111881+0000","last_clean":"2024-03-26T03:12:59.111881+0000","last_became_active":"2024-03-26T02:59:00.933674+0000","last_became_peered":"2024-03-26T02:59:00.933674+0000","last_unstale":"2024-03-26T03:12:59.111881+0000","last_undegraded":"2024-03-26T03:12:59.111881+0000","last_fullsized":"2024-03-26T03:12:59.111881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.667060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.667060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:48.238134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":524,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.097321+0000","last_change":"2024-03-26T03:11:44.673836+0000","last_active":"2024-03-26T03:12:49.097321+0000","last_peered":"2024-03-26T03:12:49.097321+0000","last_clean":"2024-03-26T03:12:49.097321+0000","last_became_active":"2024-03-26T02:59:02.945047+0000","last_became_peered":"2024-03-26T02:59:02.945047+0000","last_unstale":"2024-03-26T03:12:49.097321+0000","last_undegraded":"2024-03-26T03:12:49.097321+0000","last_fullsized":"2024-03-26T03:12:49.097321+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:44.673348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:44.673348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.606713+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":522,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.102065+0000","last_change":"2024-03-26T03:06:20.306623+0000","last_active":"2024-03-26T03:12:59.102065+0000","last_peered":"2024-03-26T03:12:59.102065+0000","last_clean":"2024-03-26T03:06:17.219768+0000","last_became_active":"2024-03-26T03:06:20.306623+0000","last_became_peered":"2024-03-26T03:06:20.306623+0000","last_unstale":"2024-03-26T03:12:59.102065+0000","last_undegraded":"2024-03-26T03:06:20.263364+0000","last_fullsized":"2024-03-26T03:06:20.263065+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255004+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":532,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104776+0000","last_change":"2024-03-26T03:11:16.999880+0000","last_active":"2024-03-26T03:12:59.104776+0000","last_peered":"2024-03-26T03:12:59.104776+0000","last_clean":"2024-03-26T03:12:59.104776+0000","last_became_active":"2024-03-26T02:58:56.924009+0000","last_became_peered":"2024-03-26T02:58:56.924009+0000","last_unstale":"2024-03-26T03:12:59.104776+0000","last_undegraded":"2024-03-26T03:12:59.104776+0000","last_fullsized":"2024-03-26T03:12:59.104776+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.999769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.999769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017898+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":524,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.104464+0000","last_change":"2024-03-26T03:11:48.834864+0000","last_active":"2024-03-26T03:12:59.104464+0000","last_peered":"2024-03-26T03:12:59.104464+0000","last_clean":"2024-03-26T03:12:59.104464+0000","last_became_active":"2024-03-26T02:59:00.936400+0000","last_became_peered":"2024-03-26T02:59:00.936400+0000","last_unstale":"2024-03-26T03:12:59.104464+0000","last_undegraded":"2024-03-26T03:12:59.104464+0000","last_fullsized":"2024-03-26T03:12:59.104464+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.834753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.834753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":498,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.111777+0000","last_change":"2024-03-26T03:06:20.313334+0000","last_active":"2024-03-26T03:12:59.111777+0000","last_peered":"2024-03-26T03:12:59.111777+0000","last_clean":"2024-03-26T03:06:17.225026+0000","last_became_active":"2024-03-26T03:06:20.313334+0000","last_became_peered":"2024-03-26T03:06:20.313334+0000","last_unstale":"2024-03-26T03:12:59.111777+0000","last_undegraded":"2024-03-26T03:12:59.111777+0000","last_fullsized":"2024-03-26T03:06:20.267029+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.598813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.598813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":563,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102072+0000","last_change":"2024-03-26T03:11:41.830501+0000","last_active":"2024-03-26T03:12:59.102072+0000","last_peered":"2024-03-26T03:12:59.102072+0000","last_clean":"2024-03-26T03:12:59.102072+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:12:59.102072+0000","last_undegraded":"2024-03-26T03:12:59.102072+0000","last_fullsized":"2024-03-26T03:12:59.102072+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:11:41.830357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.830357+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":206,"reported_epoch":296,"state":"active+undersized","last_fresh":"2024-03-26T03:12:49.099125+0000","last_change":"2024-03-26T03:06:20.309640+0000","last_active":"2024-03-26T03:12:49.099125+0000","last_peered":"2024-03-26T03:12:49.099125+0000","last_clean":"2024-03-26T02:58:55.892771+0000","last_became_active":"2024-03-26T03:06:20.309640+0000","last_became_peered":"2024-03-26T03:06:20.309640+0000","last_unstale":"2024-03-26T03:12:49.099125+0000","last_undegraded":"2024-03-26T03:12:49.099125+0000","last_fullsized":"2024-03-26T03:06:20.306060+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:12:53.416897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":524,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.107709+0000","last_change":"2024-03-26T03:12:17.280984+0000","last_active":"2024-03-26T03:12:59.107709+0000","last_peered":"2024-03-26T03:12:59.107709+0000","last_clean":"2024-03-26T03:12:59.107709+0000","last_became_active":"2024-03-26T02:59:00.931493+0000","last_became_peered":"2024-03-26T02:59:00.931493+0000","last_unstale":"2024-03-26T03:12:59.107709+0000","last_undegraded":"2024-03-26T03:12:59.107709+0000","last_fullsized":"2024-03-26T03:12:59.107709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:17.280891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:17.280891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:35.555586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":518,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.101906+0000","last_change":"2024-03-26T03:11:46.936315+0000","last_active":"2024-03-26T03:12:59.101906+0000","last_peered":"2024-03-26T03:12:59.101906+0000","last_clean":"2024-03-26T03:12:59.101906+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:12:59.101906+0000","last_undegraded":"2024-03-26T03:12:59.101906+0000","last_fullsized":"2024-03-26T03:12:59.101906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:11:46.936200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":320,"reported_epoch":298,"state":"active+undersized+degraded","last_fresh":"2024-03-26T03:12:59.102355+0000","last_change":"2024-03-26T03:06:20.306252+0000","last_active":"2024-03-26T03:12:59.102355+0000","last_peered":"2024-03-26T03:12:59.102355+0000","last_clean":"2024-03-26T02:59:04.357969+0000","last_became_active":"2024-03-26T03:06:20.306252+0000","last_became_peered":"2024-03-26T03:06:20.306252+0000","last_unstale":"2024-03-26T03:12:59.102355+0000","last_undegraded":"2024-03-26T03:06:20.263251+0000","last_fullsized":"2024-03-26T03:06:20.263153+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:01.067700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":537,"reported_epoch":296,"state":"active+clean","last_fresh":"2024-03-26T03:12:49.095664+0000","last_change":"2024-03-26T03:11:51.538045+0000","last_active":"2024-03-26T03:12:49.095664+0000","last_peered":"2024-03-26T03:12:49.095664+0000","last_clean":"2024-03-26T03:12:49.095664+0000","last_became_active":"2024-03-26T02:58:56.920085+0000","last_became_peered":"2024-03-26T02:58:56.920085+0000","last_unstale":"2024-03-26T03:12:49.095664+0000","last_undegraded":"2024-03-26T03:12:49.095664+0000","last_fullsized":"2024-03-26T03:12:49.095664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.537921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.537921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:21.217203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":522,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-03-26T03:12:59.102319+0000","last_change":"2024-03-26T03:10:52.646355+0000","last_active":"2024-03-26T03:12:59.102319+0000","last_peered":"2024-03-26T03:12:59.102319+0000","last_clean":"2024-03-26T03:12:59.102319+0000","last_became_active":"2024-03-26T02:59:00.943437+0000","last_became_peered":"2024-03-26T02:59:00.943437+0000","last_unstale":"2024-03-26T03:12:59.102319+0000","last_undegraded":"2024-03-26T03:12:59.102319+0000","last_fullsized":"2024-03-26T03:12:59.102319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:10:52.646244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:10:52.646244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:00.255111+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":209,"reported_epoch":298,"state":"active+undersized","last_fresh":"2024-03-26T03:12:59.104368+0000","last_change":"2024-03-26T03:06:20.272368+0000","last_active":"2024-03-26T03:12:59.104368+0000","last_peered":"2024-03-26T03:12:59.104368+0000","last_clean":"2024-03-26T02:59:01.918211+0000","last_became_active":"2024-03-26T03:06:20.272368+0000","last_became_peered":"2024-03-26T03:06:20.272368+0000","last_unstale":"2024-03-26T03:12:59.104368+0000","last_undegraded":"2024-03-26T03:12:59.104368+0000","last_fullsized":"2024-03-26T03:06:20.265330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:13:02.017648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1210843136,"num_objects":18476,"num_object_clones":0,"num_object_copies":55428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18476,"num_write_kb":1182464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3175612416,"data_stored":3175612416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18476,"ondisk_log_size":18476,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":608258,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":54,"acting":58,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":832,"num_read_kb":827,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":7,"acting":7,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1410,"num_read_kb":1135,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":950,"ondisk_log_size":950,"up":79,"acting":79,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":80,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103341,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964852,"kb_used_data":311724,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92780300,"statfs":{"total":95995035648,"available":95007027200,"internally_reserved":0,"allocated":319205376,"data_stored":318471292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":668793290},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430385,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972732,"kb_used_data":548656,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92772420,"statfs":{"total":95995035648,"available":94998958080,"internally_reserved":0,"allocated":561823744,"data_stored":561092095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900012,"kb_used_data":255200,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92845140,"statfs":{"total":95995035648,"available":95073423360,"internally_reserved":0,"allocated":261324800,"data_stored":260546347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738574,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830500,"kb_used_data":334236,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92914652,"statfs":{"total":95995035648,"available":95144603648,"internally_reserved":0,"allocated":342257664,"data_stored":341532940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411524,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928576,"kb_used_data":469812,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":92816576,"statfs":{"total":95995035648,"available":95044173824,"internally_reserved":0,"allocated":481087488,"data_stored":480353344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":469760453},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754608,"kb_used_data":2462440,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":90990544,"statfs":{"total":95995035648,"available":93174317056,"internally_reserved":0,"allocated":2521538560,"data_stored":2521156889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986363,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897540,"kb_used_data":559544,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92847612,"statfs":{"total":95995035648,"available":95075954688,"internally_reserved":0,"allocated":572973056,"data_stored":572226593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692023,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993588,"kb_used_data":372396,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":92751564,"statfs":{"total":95995035648,"available":94977601536,"internally_reserved":0,"allocated":381333504,"data_stored":380599424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383582208,"data_stored":383582208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267714560,"data_stored":267714560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":515375104,"data_stored":515375104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":658440192,"data_stored":658440192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":429129728,"data_stored":429129728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":576323584,"data_stored":576323584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":345047040,"data_stored":345047040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:13:03.989 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T03:13:03.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pg_num 31 2024-03-26T03:13:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.003 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:04.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:04.104 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:13:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:13:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:13:04.129 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 11 16 31893 31877 181.072 149.062 0.00468359 0.00551556 2024-03-26T03:13:04.156 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:04.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:04.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:04.396 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.429 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:04.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:04.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:04.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:04.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:04.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:04.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:04.841 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:04.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:04 smithi066 ceph-mon[29666]: osdmap e299: 8 total, 7 up, 8 in 2024-03-26T03:13:04.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:04 smithi066 ceph-mon[29666]: osdmap e300: 8 total, 7 up, 8 in 2024-03-26T03:13:04.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:04 smithi066 ceph-mon[37920]: osdmap e299: 8 total, 7 up, 8 in 2024-03-26T03:13:04.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:04 smithi066 ceph-mon[37920]: osdmap e300: 8 total, 7 up, 8 in 2024-03-26T03:13:04.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:04.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:04.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:04 smithi192 ceph-mon[33110]: osdmap e299: 8 total, 7 up, 8 in 2024-03-26T03:13:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:04 smithi192 ceph-mon[33110]: osdmap e300: 8 total, 7 up, 8 in 2024-03-26T03:13:05.075 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.121 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:05.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:05.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 12 15 34123 34108 177.599 139.438 0.00713692 0.00562579 2024-03-26T03:13:05.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:05.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:05.360 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:05.404 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:05.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:05.536 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:05.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:05.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:05.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:05.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:05.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:05 smithi066 ceph-mon[29666]: pgmap v626: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 177 MiB/s wr, 2.83k op/s; 6793/69105 objects degraded (9.830%); 1/69105 objects misplaced (0.001%) 2024-03-26T03:13:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:05 smithi066 ceph-mon[37920]: pgmap v626: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 177 MiB/s wr, 2.83k op/s; 6793/69105 objects degraded (9.830%); 1/69105 objects misplaced (0.001%) 2024-03-26T03:13:05.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:05.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:05.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:05 smithi192 ceph-mon[33110]: pgmap v626: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 177 MiB/s wr, 2.83k op/s; 6793/69105 objects degraded (9.830%); 1/69105 objects misplaced (0.001%) 2024-03-26T03:13:06.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:06.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:06.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 13 16 36276 36260 174.283 134.5 0.0039912 0.00573249 2024-03-26T03:13:06.196 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:06.239 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:06.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:06.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:06.477 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:06.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:06.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:06.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:06.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:06.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:06.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:06.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:06 smithi066 ceph-mon[29666]: pgmap v627: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 238 MiB/s wr, 3.81k op/s; 10559/100182 objects degraded (10.540%); 1/100182 objects misplaced (0.001%) 2024-03-26T03:13:06.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:06 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 10559/100182 objects degraded (10.540%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:06.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:06 smithi066 ceph-mon[37920]: pgmap v627: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 238 MiB/s wr, 3.81k op/s; 10559/100182 objects degraded (10.540%); 1/100182 objects misplaced (0.001%) 2024-03-26T03:13:06.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:06 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 10559/100182 objects degraded (10.540%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:06.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:06.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:06.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:06.923 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:06 smithi192 ceph-mon[33110]: pgmap v627: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 238 MiB/s wr, 3.81k op/s; 10559/100182 objects degraded (10.540%); 1/100182 objects misplaced (0.001%) 2024-03-26T03:13:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:06 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 10559/100182 objects degraded (10.540%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:07.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:07.065 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:07.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 14 15 38053 38038 169.772 111.125 0.0071237 0.00588487 2024-03-26T03:13:07.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.215 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:07.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:07.316 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:07.360 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:07.461 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:07.513 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:07.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:07.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:07.754 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:07.800 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:07.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:07.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:07.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:08.041 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:08.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:08.118 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:13:08.118 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:13:08.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 15 15 40162 40147 167.241 131.812 0.00687812 0.00597434 2024-03-26T03:13:08.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:08.226 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:08.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:08.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:08.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:08.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:08.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:08.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:08.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:08.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:08.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:08.907 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:13:08.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:08.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:09.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:09.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:09.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 16 16 42265 42249 164.997 131.375 0.00982885 0.00605538 2024-03-26T03:13:09.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:09.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:09.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:09.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:09.331 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:09.339 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:09.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:09.473 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:09.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:09 smithi066 ceph-mon[29666]: pgmap v628: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.83k op/s; 10559/100182 objects degraded (10.540%); 1/100182 objects misplaced (0.001%) 2024-03-26T03:13:09.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:09.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:09.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:09 smithi066 ceph-mon[37920]: pgmap v628: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.83k op/s; 10559/100182 objects degraded (10.540%); 1/100182 objects misplaced (0.001%) 2024-03-26T03:13:09.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:09.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:09.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:09.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:09.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:09 smithi192 ceph-mon[33110]: pgmap v628: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.83k op/s; 10559/100182 objects degraded (10.540%); 1/100182 objects misplaced (0.001%) 2024-03-26T03:13:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:09.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:09.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:09.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:09.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:09.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:10.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:10.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:10.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 17 15 44234 44219 162.533 123.125 0.00390154 0.0061472 2024-03-26T03:13:10.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:10.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:13:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:13:10.265 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:10.310 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:10.412 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:10.456 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:10.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:10.603 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:10 smithi066 ceph-mon[29666]: osdmap e301: 8 total, 7 up, 8 in 2024-03-26T03:13:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:10.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:10 smithi066 ceph-mon[37920]: osdmap e301: 8 total, 7 up, 8 in 2024-03-26T03:13:10.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:10.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:10 smithi192 ceph-mon[33110]: osdmap e301: 8 total, 7 up, 8 in 2024-03-26T03:13:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.749 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:10.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:10.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:10.890 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:10.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:10.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:11.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:11.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:11.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 18 16 46308 46292 160.701 129.562 0.0108446 0.0062178 2024-03-26T03:13:11.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:11.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:11.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:11.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:11.450 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:11.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:11.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:11 smithi192 ceph-mon[33110]: pgmap v630: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 186 MiB/s wr, 2.97k op/s; 13081/125013 objects degraded (10.464%); 1/125013 objects misplaced (0.001%) 2024-03-26T03:13:11.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:11.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:11.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:11.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:11.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:11 smithi066 ceph-mon[29666]: pgmap v630: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 186 MiB/s wr, 2.97k op/s; 13081/125013 objects degraded (10.464%); 1/125013 objects misplaced (0.001%) 2024-03-26T03:13:11.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:11.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:11 smithi066 ceph-mon[37920]: pgmap v630: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 186 MiB/s wr, 2.97k op/s; 13081/125013 objects degraded (10.464%); 1/125013 objects misplaced (0.001%) 2024-03-26T03:13:11.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:11.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:11.916 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:11.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:12.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:12.110 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:12.130 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 19 16 48274 48258 158.71 122.875 0.00957209 0.00629459 2024-03-26T03:13:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.173 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:12.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:12.274 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.318 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:12.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:12.419 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.498 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:12.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:12.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:12.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:12.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:12.679 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:12 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 13709/130911 objects degraded (10.472%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:12.680 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:12 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 13709/130911 objects degraded (10.472%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:12.736 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:12 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 13709/130911 objects degraded (10.472%), 27 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.786 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:12.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:12.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:12.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:12.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:13.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.072 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:13.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:13.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:13:13.131129+0000 min lat: 0.00101075 max lat: 0.162256 avg lat: 0.00637946 2024-03-26T03:13:13.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:13:13.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 20 16 50140 50124 156.605 116.625 0.0079856 0.00637946 2024-03-26T03:13:13.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:13.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:13.311 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.356 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:13.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:13.456 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:13.468 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 1 pg_num to 31 2024-03-26T03:13:13.506 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:13.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:13.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:13 smithi192 ceph-mon[33110]: pgmap v631: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 169 MiB/s wr, 2.71k op/s; 13709/130911 objects degraded (10.472%); 1/130911 objects misplaced (0.001%) 2024-03-26T03:13:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:13 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/14370506' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-26T03:13:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:13 smithi192 ceph-mon[33110]: from='client.35561 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-26T03:13:13.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:13.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:13.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-mon[29666]: pgmap v631: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 169 MiB/s wr, 2.71k op/s; 13709/130911 objects degraded (10.472%); 1/130911 objects misplaced (0.001%) 2024-03-26T03:13:13.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/14370506' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-26T03:13:13.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-mon[29666]: from='client.35561 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-26T03:13:13.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-mon[37920]: pgmap v631: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 169 MiB/s wr, 2.71k op/s; 13709/130911 objects degraded (10.472%); 1/130911 objects misplaced (0.001%) 2024-03-26T03:13:13.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/14370506' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-26T03:13:13.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-mon[37920]: from='client.35561 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-26T03:13:13.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:13.943 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:13.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:14.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:14.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:13:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:13:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:13:14.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 21 16 51933 51917 154.484 112.062 0.0111341 0.0064676 2024-03-26T03:13:14.180 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:14.221 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:14.323 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.368 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:14.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:14.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.520 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:14.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:14.621 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:14.666 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:14 smithi192 ceph-mon[33110]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-26T03:13:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:14 smithi192 ceph-mon[33110]: osdmap e302: 8 total, 7 up, 8 in 2024-03-26T03:13:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:14 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/896210116' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:13:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:14 smithi192 ceph-mon[33110]: from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:13:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:14 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T03:13:14.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:14.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:14.842 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[29666]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-26T03:13:14.842 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[29666]: osdmap e302: 8 total, 7 up, 8 in 2024-03-26T03:13:14.842 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/896210116' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:13:14.842 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[29666]: from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:13:14.843 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T03:13:14.843 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[37920]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-26T03:13:14.843 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[37920]: osdmap e302: 8 total, 7 up, 8 in 2024-03-26T03:13:14.843 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/896210116' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:13:14.843 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[37920]: from='client.25546 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:13:14.843 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:14 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T03:13:14.905 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:14.945 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:14.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:15.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:15.087 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:15.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 22 16 53760 53744 152.652 114.188 0.00551896 0.00654572 2024-03-26T03:13:15.189 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:15.231 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:15.292 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:13:15.333 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:15.374 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:15.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.521 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:15.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:15.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:15.659 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: pgmap v633: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.43k op/s; 13709/134565 objects degraded (10.188%); 1/134565 objects misplaced (0.001%) 2024-03-26T03:13:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: osdmap e303: 8 total, 7 up, 8 in 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:13:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:15 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:13:15.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:15.803 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: pgmap v633: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.43k op/s; 13709/134565 objects degraded (10.188%); 1/134565 objects misplaced (0.001%) 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: osdmap e303: 8 total, 7 up, 8 in 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:13:15.815 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: pgmap v633: 168 pgs: 5 active+clean+remapped, 27 active+undersized+degraded, 38 active+undersized, 98 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.43k op/s; 13709/134565 objects degraded (10.188%); 1/134565 objects misplaced (0.001%) 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: from='client.25546 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: osdmap e303: 8 total, 7 up, 8 in 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:13:15.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:15 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:13:15.906 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:15.949 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:15.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:16.051 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:16.056 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:13:16.095 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:16.096 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:13:16.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 23 16 55531 55515 150.827 110.688 0.00456745 0.00662538 2024-03-26T03:13:16.132 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:13:16.175 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:13:16.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:16.212 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:13:16.240 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:16.255 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:13:16.296 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:13:16.296 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:13:16.296 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:13:16.297 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:13:16.297 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:13:16.297 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:13:16.297 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:13:16.297 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:13:16.297 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:13:16.341 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:16.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:16.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:16.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:13:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:13:16.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: osdmap e304: 8 total, 7 up, 8 in 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.b scrub starts 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.b scrub ok 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1009809256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.d scrub starts 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.d scrub ok 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.5 scrub starts 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: 1.5 scrub ok 2024-03-26T03:13:16.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: osdmap e304: 8 total, 7 up, 8 in 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.b scrub starts 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.b scrub ok 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1009809256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.d scrub starts 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.d scrub ok 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.5 scrub starts 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: 1.5 scrub ok 2024-03-26T03:13:16.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:16 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T03:13:16.630 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:16.673 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: osdmap e304: 8 total, 7 up, 8 in 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.b scrub starts 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.b scrub ok 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1009809256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.d scrub starts 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.d scrub ok 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.5 scrub starts 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: 1.5 scrub ok 2024-03-26T03:13:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:16 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T03:13:16.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:16.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:16.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:16.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:16.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:17.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:13:17.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:13:17.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:17.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 24 16 57242 57226 148.998 106.938 0.00387867 0.00670655 2024-03-26T03:13:17.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:17.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:17.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:17.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:17.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:17.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:17.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: pgmap v636: 178 pgs: 10 unknown, 5 active+clean+remapped, 5 peering, 26 active+undersized+degraded, 37 active+undersized, 95 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 120 MiB/s wr, 1.92k op/s; 16715/159630 objects degraded (10.471%) 2024-03-26T03:13:17.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: 1.c scrub starts 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: 1.c scrub ok 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: osdmap e305: 8 total, 7 up, 8 in 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: 1.e scrub starts 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[29666]: 1.e scrub ok 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: pgmap v636: 178 pgs: 10 unknown, 5 active+clean+remapped, 5 peering, 26 active+undersized+degraded, 37 active+undersized, 95 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 120 MiB/s wr, 1.92k op/s; 16715/159630 objects degraded (10.471%) 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: 1.c scrub starts 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: 1.c scrub ok 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T03:13:17.563 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: osdmap e305: 8 total, 7 up, 8 in 2024-03-26T03:13:17.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:13:17.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:13:17.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: 1.e scrub starts 2024-03-26T03:13:17.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:17 smithi066 ceph-mon[37920]: 1.e scrub ok 2024-03-26T03:13:17.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:17.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:17.672 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:13:17.672 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:13:17.672 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:32363 8.1 8:87fc8ad4:::benchmark_data_smithi066_100612_object32362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:20.391847+0000", 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 477.26537113199998, 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19202301199999999, 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.673 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 32363 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391847+0000", 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391847+0000", 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.674 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391848+0000", 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3179999999999999e-06 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391872+0000", 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4352e-05 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391883+0000", 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0710000000000001e-05 2024-03-26T03:13:17.675 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.391903+0000", 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0075e-05 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.393374+0000", 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014705110000000001 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.393492+0000", 2024-03-26T03:13:17.676 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118849 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.393622+0000", 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000130094 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.394813+0000", 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001190565 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.395867+0000", 2024-03-26T03:13:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001053663 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.583775+0000", 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.18790796400000001 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.583829+0000", 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4098000000000003e-05 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.678 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:20.583870+0000", 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0812999999999997e-05 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:33124 8.1 8:8362333d:::benchmark_data_smithi066_100612_object33123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.033247+0000", 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 476.62397100099997, 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14529449799999999, 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.679 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 33124 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033247+0000", 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.680 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033247+0000", 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033249+0000", 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9259999999999999e-06 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.033271+0000", 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2568000000000001e-05 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.681 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154308+0000", 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.12103662900000001 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154328+0000", 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0445000000000002e-05 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154565+0000", 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000236266 2024-03-26T03:13:17.682 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154771+0000", 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020644899999999999 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.154944+0000", 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017347099999999999 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.156718+0000", 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017739450000000001 2024-03-26T03:13:17.683 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.159798+0000", 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030792340000000001 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.178423+0000", 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.018625701000000001 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.684 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.178491+0000", 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8093e-05 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.178541+0000", 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9771000000000002e-05 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34158 8.7 8:e390f89f:::benchmark_data_smithi066_100612_object34157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.704399+0000", 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.95281886599997, 2024-03-26T03:13:17.685 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21413618700000001, 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34158 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704399+0000", 2024-03-26T03:13:17.686 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704399+0000", 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704401+0000", 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.328e-06 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704429+0000", 2024-03-26T03:13:17.687 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.8170999999999998e-05 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704441+0000", 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1956e-05 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704450+0000", 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6119999999999999e-06 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704606+0000", 2024-03-26T03:13:17.688 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015650700000000001 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704731+0000", 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012487800000000001 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.704864+0000", 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000132325 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.689 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706325+0000", 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001461838 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707781+0000", 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014558590000000001 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708401+0000", 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00062018800000000003 2024-03-26T03:13:17.690 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918486+0000", 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21008421499999999 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918535+0000", 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9310000000000001e-05 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.691 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34164 8.a 8:5912da51:::benchmark_data_smithi066_100612_object34163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.706561+0000", 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.95065629300001, 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21261882400000001, 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34164 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.692 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706561+0000", 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706561+0000", 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.693 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706563+0000", 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.812e-06 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706587+0000", 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3898999999999999e-05 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706598+0000", 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1399e-05 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.694 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706607+0000", 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3480000000000004e-06 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706665+0000", 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8217999999999997e-05 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706833+0000", 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016762299999999999 2024-03-26T03:13:17.695 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.706981+0000", 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014804900000000001 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919035+0000", 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21205386500000001 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919048+0000", 2024-03-26T03:13:17.696 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3811e-05 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919086+0000", 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7258999999999997e-05 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919135+0000", 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9626000000000002e-05 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.697 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919180+0000", 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4914999999999999e-05 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34166 8.3 8:ca06eeeb:::benchmark_data_smithi066_100612_object34165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.707740+0000", 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.949477559, 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.216887831, 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.698 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34166 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707740+0000", 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.699 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707740+0000", 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707742+0000", 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5990000000000001e-06 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707765+0000", 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3765e-05 2024-03-26T03:13:17.700 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707775+0000", 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4800000000000007e-06 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707783+0000", 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7430000000000002e-06 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707839+0000", 2024-03-26T03:13:17.701 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6372999999999997e-05 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.707985+0000", 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000145494 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708097+0000", 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011292 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.702 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919114+0000", 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21101609800000001 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919150+0000", 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6705000000000002e-05 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924521+0000", 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053703479999999996 2024-03-26T03:13:17.703 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924585+0000", 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4406000000000006e-05 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924628+0000", 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2899999999999999e-05 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.704 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.24961.0:34158 8.7 e183/172 ondisk, result = 0)", 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.708205+0000", 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.94901273699998, 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.210338894, 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708205+0000", 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.705 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708205+0000", 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708206+0000", 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2299999999999999e-07 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.706 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708210+0000", 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7519999999999998e-06 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708218+0000", 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7159999999999992e-06 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708225+0000", 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3890000000000002e-06 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.707 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708389+0000", 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016390900000000001 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708396+0000", 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5340000000000002e-06 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918544+0000", 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21014767100000001 2024-03-26T03:13:17.708 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34168 8.a 8:59dd35f8:::benchmark_data_smithi066_100612_object34167:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.708754+0000", 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.94846356699998, 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.21735549000000001, 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.709 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34168 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708754+0000", 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708754+0000", 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.710 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708755+0000", 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2249999999999999e-06 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708775+0000", 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0165000000000001e-05 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918607+0000", 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20983168899999999 2024-03-26T03:13:17.711 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918619+0000", 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2215000000000001e-05 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919213+0000", 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00059358900000000005 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919358+0000", 2024-03-26T03:13:17.712 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014458599999999999 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919512+0000", 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000154382 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.923710+0000", 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041979529999999999 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.713 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.923752+0000", 2024-03-26T03:13:17.714 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2240000000000002e-05 2024-03-26T03:13:17.714 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.714 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.714 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.925962+0000", 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022098700000000001 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.926065+0000", 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010254700000000001 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.715 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.926110+0000", 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5028999999999999e-05 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.24961.0:34164 8.a e183/172 ondisk, result = 0)", 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.708786+0000", 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.94843197900002, 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.210263589, 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.716 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708786+0000", 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708786+0000", 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.717 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708787+0000", 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.297e-06 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.708792+0000", 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5510000000000001e-06 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918393+0000", 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.209601015 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.718 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918409+0000", 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6861e-05 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919020+0000", 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00061044400000000003 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919028+0000", 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2320000000000001e-06 2024-03-26T03:13:17.719 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.919049+0000", 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1189e-05 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.24961.0:34169 8.b e183/172 8:dfcca531:::benchmark_data_smithi066_100612_object34168:head v 183'2156, mlcod=183'2154)", 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:21.709192+0000", 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.94802538200003, 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.215641534, 2024-03-26T03:13:17.720 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709192+0000", 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709192+0000", 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.721 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709194+0000", 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2270000000000001e-06 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.709217+0000", 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.387e-05 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918416+0000", 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20919812300000001 2024-03-26T03:13:17.722 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918427+0000", 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1219999999999999e-05 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918621+0000", 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00019475800000000001 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.918640+0000", 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8970999999999998e-05 2024-03-26T03:13:17.723 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924640+0000", 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0059992149999999996 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:21.924834+0000", 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00019415 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.724 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34572 8.7 8:e0d7c130:::benchmark_data_smithi066_100612_object34571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.123979+0000", 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.533238412, 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.204696821, 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34572 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.725 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.123979+0000", 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.123979+0000", 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.123981+0000", 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5570000000000001e-06 2024-03-26T03:13:17.726 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.124014+0000", 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2842999999999997e-05 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.124026+0000", 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2119999999999999e-05 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.124035+0000", 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9889999999999998e-06 2024-03-26T03:13:17.727 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.125033+0000", 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00099777900000000007 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.125154+0000", 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012128100000000001 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.728 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.125285+0000", 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000131321 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127808+0000", 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0025223950000000002 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.130018+0000", 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022102939999999998 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.729 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328567+0000", 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19854961700000001 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328624+0000", 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6298000000000002e-05 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328676+0000", 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2327000000000002e-05 2024-03-26T03:13:17.730 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34578 8.7 8:e32f7176:::benchmark_data_smithi066_100612_object34577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.127436+0000", 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.52978124100002, 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20137244400000001, 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.731 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34578 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127436+0000", 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127436+0000", 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.732 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127437+0000", 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3600000000000002e-07 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127461+0000", 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.391e-05 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127470+0000", 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8929999999999994e-06 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.733 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.127477+0000", 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1369999999999996e-06 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.128109+0000", 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00063122000000000004 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.128234+0000", 2024-03-26T03:13:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000125704 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.128368+0000", 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000133843 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.130056+0000", 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001687926 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131128+0000", 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001072143 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328724+0000", 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19759611599999999 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328771+0000", 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6535000000000001e-05 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.736 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328809+0000", 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8080999999999997e-05 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34584 8.7 8:e3ec79c0:::benchmark_data_smithi066_100612_object34583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.131279+0000", 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.525938844, 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20215682099999999, 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.737 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34584 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131279+0000", 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.738 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131279+0000", 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131281+0000", 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8390000000000001e-06 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131303+0000", 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.226e-05 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.739 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131314+0000", 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0624e-05 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131322+0000", 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0350000000000001e-06 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131403+0000", 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1310000000000006e-05 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.740 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131521+0000", 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118009 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131654+0000", 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013324300000000001 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330270+0000", 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.198616031 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.741 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330309+0000", 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8905000000000001e-05 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333277+0000", 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0029678460000000001 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333374+0000", 2024-03-26T03:13:17.742 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7459000000000003e-05 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333436+0000", 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1260000000000006e-05 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34585 8.3 8:c22d9924:::benchmark_data_smithi066_100612_object34584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.131968+0000", 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.52524987200002, 2024-03-26T03:13:17.743 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20611238700000001, 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34585 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131968+0000", 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.744 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131968+0000", 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.131969+0000", 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.282e-06 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132008+0000", 2024-03-26T03:13:17.745 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9110999999999999e-05 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132018+0000", 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6919999999999997e-06 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132026+0000", 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8660000000000006e-06 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.746 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132126+0000", 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100704 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330081+0000", 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.197954138 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330927+0000", 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00084675600000000005 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.747 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.335100+0000", 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041724580000000004 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.335144+0000", 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4280999999999998e-05 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.337992+0000", 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002848176 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.748 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.338040+0000", 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7685000000000002e-05 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.338080+0000", 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0238000000000003e-05 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34588 8.1 8:8ba52af6:::benchmark_data_smithi066_100612_object34587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.749 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.132044+0000", 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.52517361600002, 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20180536399999999, 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34588 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.750 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132044+0000", 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132044+0000", 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132045+0000", 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0400000000000005e-07 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.751 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132068+0000", 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3337e-05 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132076+0000", 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5129999999999999e-06 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132082+0000", 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5350000000000001e-06 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.752 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132187+0000", 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000104338 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328422+0000", 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19623518100000001 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328587+0000", 2024-03-26T03:13:17.753 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016529199999999999 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.332138+0000", 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003551397 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.332175+0000", 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6486999999999998e-05 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.754 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333748+0000", 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001573292 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333807+0000", 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8996000000000002e-05 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.333849+0000", 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2191999999999999e-05 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.755 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop_reply(client.24961.0:34572 8.7 e183/172 ondisk, result = 0)", 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.132197+0000", 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.525021133, 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19648653499999999, 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.756 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132197+0000", 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132197+0000", 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132198+0000", 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3909999999999999e-06 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.757 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132203+0000", 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8930000000000004e-06 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328492+0000", 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19628907800000001 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328502+0000", 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0404999999999999e-05 2024-03-26T03:13:17.758 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328550+0000", 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8109999999999998e-05 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328561+0000", 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0276e-05 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.759 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328683+0000", 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012238200000000001 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.24961.0:34594 8.7 8:e90646ed:::benchmark_data_smithi066_100612_object34593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:05:22.132813+0000", 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 475.52440447200001, 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20418934899999999, 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.760 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.24961", 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3766986948", 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 34594 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132813+0000", 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132813+0000", 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.761 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132815+0000", 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3069999999999999e-06 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.132837+0000", 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2121e-05 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.762 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328282+0000", 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19544577399999999 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.328295+0000", 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2354999999999999e-05 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330364+0000", 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020697240000000002 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.763 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.330510+0000", 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014537299999999999 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.331832+0000", 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013220879999999999 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336747+0000", 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049151799999999999 2024-03-26T03:13:17.764 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336817+0000", 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9748999999999999e-05 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336910+0000", 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3179999999999999e-05 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.336961+0000", 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1149000000000001e-05 2024-03-26T03:13:17.765 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:05:22.337003+0000", 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1349e-05 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:283803 9.a e278/225 9:5f6e1312:::benchmark_data_smithi066_125102_object137502:head v 278'17692, mlcod=278'17690)", 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:04.997554+0000", 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 72.659663723999998, 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.15112352400000001, 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.766 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.997554+0000", 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.997554+0000", 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.767 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.997555+0000", 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2190000000000001e-06 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.997565+0000", 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7030000000000001e-06 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.997575+0000", 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9720000000000004e-06 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.768 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.997582+0000", 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9970000000000001e-06 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999266+0000", 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0016845569999999999 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999286+0000", 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9364e-05 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.769 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148627+0000", 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14934112299999999 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148677+0000", 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0589e-05 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.771 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:283805 9.d e278/225 9:b29b54ea:::benchmark_data_smithi066_125102_object137504:head v 278'17887, mlcod=278'17884)", 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:04.999048+0000", 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 72.658169545000007, 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.149696406, 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.772 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999048+0000", 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999048+0000", 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999049+0000", 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4600000000000003e-07 2024-03-26T03:13:17.773 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999057+0000", 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0839999999999992e-06 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999065+0000", 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0309999999999998e-06 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999072+0000", 2024-03-26T03:13:17.774 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3990000000000001e-06 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999123+0000", 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1181999999999999e-05 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:04.999143+0000", 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9933999999999999e-05 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.775 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148699+0000", 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14955643299999999 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148745+0000", 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5396999999999999e-05 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:283809 9.3 e278/225 9:c67a7481:::benchmark_data_smithi066_125102_object137508:head v 278'17596, mlcod=278'17594)", 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:05.000694+0000", 2024-03-26T03:13:17.776 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 72.656523831000001, 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.15251658200000001, 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.000694+0000", 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.000694+0000", 2024-03-26T03:13:17.777 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.000695+0000", 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3799999999999996e-07 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.000701+0000", 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.764e-06 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.778 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.000708+0000", 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5710000000000003e-06 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.000713+0000", 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8250000000000004e-06 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148640+0000", 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14792709800000001 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.779 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148680+0000", 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9714e-05 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.153184+0000", 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0045045479999999997 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.153210+0000", 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6224000000000002e-05 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.780 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15360.0:283807 9.f e278/225 9:f34c1168:::benchmark_data_smithi066_125102_object137506:head v 278'17648, mlcod=278'17646)", 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:05.002146+0000", 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 72.655071980000002, 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.15865411600000001, 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.002146+0000", 2024-03-26T03:13:17.781 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.002146+0000", 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.002147+0000", 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9900000000000009e-07 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:17.782 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.002152+0000", 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3890000000000004e-06 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148606+0000", 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14645422899999999 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148622+0000", 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5458999999999999e-05 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.783 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148845+0000", 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00022321400000000001 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.148869+0000", 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.425e-05 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.160767+0000", 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011898124 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:17.784 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.160800+0000", 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2452e-05 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:17.785 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:13:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:17.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:17.909 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:17.952 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:17.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: pgmap v636: 178 pgs: 10 unknown, 5 active+clean+remapped, 5 peering, 26 active+undersized+degraded, 37 active+undersized, 95 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 120 MiB/s wr, 1.92k op/s; 16715/159630 objects degraded (10.471%) 2024-03-26T03:13:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: 1.c scrub starts 2024-03-26T03:13:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: 1.c scrub ok 2024-03-26T03:13:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T03:13:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: osdmap e305: 8 total, 7 up, 8 in 2024-03-26T03:13:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:13:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:13:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: 1.e scrub starts 2024-03-26T03:13:17.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:17 smithi192 ceph-mon[33110]: 1.e scrub ok 2024-03-26T03:13:18.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:18.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:18.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 25 16 58903 58887 147.19 103.812 0.00505052 0.00678915 2024-03-26T03:13:18.189 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:18.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:18.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:18.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:18.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:13:18.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:18.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:18.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:18.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:18.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:18.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: osdmap e306: 8 total, 7 up, 8 in 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 1.10 scrub starts 2024-03-26T03:13:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[29666]: 1.10 scrub ok 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: osdmap e306: 8 total, 7 up, 8 in 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 1.10 scrub starts 2024-03-26T03:13:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:18 smithi066 ceph-mon[37920]: 1.10 scrub ok 2024-03-26T03:13:18.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:18.930 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:18.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:18.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: osdmap e306: 8 total, 7 up, 8 in 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 1.10 scrub starts 2024-03-26T03:13:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:18 smithi192 ceph-mon[33110]: 1.10 scrub ok 2024-03-26T03:13:19.031 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:19.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:19.075 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:13:19.131 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 26 16 60483 60467 145.326 98.75 0.0187936 0.0068744 2024-03-26T03:13:19.175 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:19.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:19.313 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:19.358 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:19.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:19.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:19.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:19.622 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:19.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:19.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: pgmap v639: 178 pgs: 10 unknown, 5 active+clean+remapped, 5 peering, 26 active+undersized+degraded, 37 active+undersized, 95 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s; 16715/159630 objects degraded (10.471%) 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: osdmap e307: 8 total, 7 up, 8 in 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: 1.1d scrub starts 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: 1.1d scrub ok 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 16715/159630 objects degraded (10.471%), 26 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[29666]: osdmap e308: 8 total, 7 up, 8 in 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: pgmap v639: 178 pgs: 10 unknown, 5 active+clean+remapped, 5 peering, 26 active+undersized+degraded, 37 active+undersized, 95 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s; 16715/159630 objects degraded (10.471%) 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: osdmap e307: 8 total, 7 up, 8 in 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: 1.1d scrub starts 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: 1.1d scrub ok 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 16715/159630 objects degraded (10.471%), 26 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:19 smithi066 ceph-mon[37920]: osdmap e308: 8 total, 7 up, 8 in 2024-03-26T03:13:19.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:19.894 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:13:19.895 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:13:19.895 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:13:19.895 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd out 0 2024-03-26T03:13:19.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:19.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:19.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:13:19.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: pgmap v639: 178 pgs: 10 unknown, 5 active+clean+remapped, 5 peering, 26 active+undersized+degraded, 37 active+undersized, 95 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s; 16715/159630 objects degraded (10.471%) 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: osdmap e307: 8 total, 7 up, 8 in 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: 1.1d scrub starts 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: 1.1d scrub ok 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 16715/159630 objects degraded (10.471%), 26 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:19.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:19 smithi192 ceph-mon[33110]: osdmap e308: 8 total, 7 up, 8 in 2024-03-26T03:13:20.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.055 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:20.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:20.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 27 16 62009 61993 143.476 95.375 0.0140088 0.00696382 2024-03-26T03:13:20.156 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:20.183 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:20.285 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:20.324 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:20.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:20.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:20.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:20.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:20.707 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:20.747 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:20.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:20.888 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:20.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:20.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:13:20.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:21.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:21.073 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:21.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:13:21.122 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:13:21.123 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:21.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 28 16 63471 63455 141.615 91.375 0.0245786 0.00705423 2024-03-26T03:13:21.143 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:21.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:21.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:21.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:21.376 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: osdmap e309: 8 total, 7 up, 8 in 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: pgmap v643: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 6 active+clean+remapped, 1 unknown; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 18748/180150 objects degraded (10.407%); 109 KiB/s, 0 objects/s recovering 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 1.1e scrub starts 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 1.1e scrub ok 2024-03-26T03:13:21.377 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 1.19 scrub starts 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 1.19 scrub ok 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 1.16 scrub starts 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[37920]: 1.16 scrub ok 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: osdmap e309: 8 total, 7 up, 8 in 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: pgmap v643: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 6 active+clean+remapped, 1 unknown; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 18748/180150 objects degraded (10.407%); 109 KiB/s, 0 objects/s recovering 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 1.1e scrub starts 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 1.1e scrub ok 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 1.19 scrub starts 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 1.19 scrub ok 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 1.16 scrub starts 2024-03-26T03:13:21.378 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:21 smithi066 ceph-mon[29666]: 1.16 scrub ok 2024-03-26T03:13:21.386 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:21.452 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:13:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:13:21.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: osdmap e309: 8 total, 7 up, 8 in 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: pgmap v643: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 6 active+clean+remapped, 1 unknown; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 18748/180150 objects degraded (10.407%); 109 KiB/s, 0 objects/s recovering 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 1.1e scrub starts 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 1.1e scrub ok 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 1.19 scrub starts 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 1.19 scrub ok 2024-03-26T03:13:21.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 1.16 scrub starts 2024-03-26T03:13:21.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:21 smithi192 ceph-mon[33110]: 1.16 scrub ok 2024-03-26T03:13:21.554 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:21.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:21.697 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:21.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:21.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:21.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:21.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:21.980 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.019 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:22.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:22.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:22.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 29 16 65053 65037 140.141 98.875 0.0124386 0.00713033 2024-03-26T03:13:22.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:22.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:22.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:22.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: 1.18 scrub starts 2024-03-26T03:13:22.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: 1.18 scrub ok 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: 1.15 scrub starts 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: 1.15 scrub ok 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: 1.1a scrub starts 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: 1.1a scrub ok 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: 1.18 scrub starts 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: 1.18 scrub ok 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: 1.15 scrub starts 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: 1.15 scrub ok 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: 1.1a scrub starts 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: 1.1a scrub ok 2024-03-26T03:13:22.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:22 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:13:22.396 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:22.435 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: 1.18 scrub starts 2024-03-26T03:13:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: 1.18 scrub ok 2024-03-26T03:13:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: 1.15 scrub starts 2024-03-26T03:13:22.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: 1.15 scrub ok 2024-03-26T03:13:22.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: 1.1a scrub starts 2024-03-26T03:13:22.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: 1.1a scrub ok 2024-03-26T03:13:22.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:22 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:13:22.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:22.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:22.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:22.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:22.817 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:22.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.858 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:22.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:22.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:22.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:22.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:23.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:23.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 30 16 66618 66602 138.73 97.8125 0.0232084 0.00720241 2024-03-26T03:13:23.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:23.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:23.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:23.376 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:23.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 1.1b scrub starts 2024-03-26T03:13:23.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 1.1b scrub ok 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 1.1c scrub starts 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 1.1c scrub ok 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: osdmap e310: 8 total, 7 up, 8 in 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: pgmap v645: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 7 active+clean+remapped; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s; 19235/184581 objects degraded (10.421%); 1/184581 objects misplaced (0.001%); 103 KiB/s, 0 objects/s recovering 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:13:23.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:23 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:13:23.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 1.1b scrub starts 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 1.1b scrub ok 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 1.1c scrub starts 2024-03-26T03:13:23.534 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 1.1c scrub ok 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: osdmap e310: 8 total, 7 up, 8 in 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: pgmap v645: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 7 active+clean+remapped; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s; 19235/184581 objects degraded (10.421%); 1/184581 objects misplaced (0.001%); 103 KiB/s, 0 objects/s recovering 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 1.1b scrub starts 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 1.1b scrub ok 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:13:23.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:13:23.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 1.1c scrub starts 2024-03-26T03:13:23.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 1.1c scrub ok 2024-03-26T03:13:23.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: osdmap e310: 8 total, 7 up, 8 in 2024-03-26T03:13:23.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: pgmap v645: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 7 active+clean+remapped; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s; 19235/184581 objects degraded (10.421%); 1/184581 objects misplaced (0.001%); 103 KiB/s, 0 objects/s recovering 2024-03-26T03:13:23.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:13:23.536 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:13:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:23.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:23.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:23.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:23.801 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:23.843 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:23.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:23.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:23.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:23.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:24.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:24.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:13:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:13:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:13:24.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 31 16 68012 67996 137.065 87.125 0.0144111 0.00729008 2024-03-26T03:13:24.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:24.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:24.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:24.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:24.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:24.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: 1.17 scrub starts 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: 1.17 scrub ok 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 19235/184581 objects degraded (10.421%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: 1.17 scrub starts 2024-03-26T03:13:24.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: 1.17 scrub ok 2024-03-26T03:13:24.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:13:24.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:13:24.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:13:24.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:13:24.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:13:24.418 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 19235/184581 objects degraded (10.421%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:24.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:24.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: 1.17 scrub starts 2024-03-26T03:13:24.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: 1.17 scrub ok 2024-03-26T03:13:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:13:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:13:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:13:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:13:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:13:24.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 19235/184581 objects degraded (10.421%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:24.544 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:24.591 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:24.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:24.692 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:24.740 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:24.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:24.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:24.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:24.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:24.954 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:13:24.978 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:24.992 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:13:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.023 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:25.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:25.036 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:13:25.076 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:13:25.109 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:13:25.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:25.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 32 16 69403 69387 135.498 86.9375 0.0109573 0.00737512 2024-03-26T03:13:25.148 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:13:25.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:25.188 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:13:25.188 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:13:25.189 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:13:25.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:25.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:25.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: pgmap v646: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 7 active+clean+remapped; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 5.7 KiB/s rd, 90 MiB/s wr, 1.45k op/s; 19235/184581 objects degraded (10.421%); 1/184581 objects misplaced (0.001%); 72 KiB/s, 0 objects/s recovering 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/825238305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:13:25.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: pgmap v646: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 7 active+clean+remapped; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 5.7 KiB/s rd, 90 MiB/s wr, 1.45k op/s; 19235/184581 objects degraded (10.421%); 1/184581 objects misplaced (0.001%); 72 KiB/s, 0 objects/s recovering 2024-03-26T03:13:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:13:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:13:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:25.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/825238305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:25.395 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:25.441 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: pgmap v646: 178 pgs: 1 active+clean+scrubbing, 2 remapped+peering, 97 active+clean, 39 active+undersized, 26 active+undersized+degraded, 6 peering, 7 active+clean+remapped; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 5.7 KiB/s rd, 90 MiB/s wr, 1.45k op/s; 19235/184581 objects degraded (10.421%); 1/184581 objects misplaced (0.001%); 72 KiB/s, 0 objects/s recovering 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/825238305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:25.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:25.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:25.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:25.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:25.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:25.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:25.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:25.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:26.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:26.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:26.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 33 15 70873 70858 134.178 91.9375 0.010585 0.00744815 2024-03-26T03:13:26.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:26.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:26.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:26.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:26 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:13:26.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:26 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:13:26.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:26 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:13:26.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:26 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:13:26.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:26.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:26 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:13:26.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:26 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:13:26.523 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.565 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:26.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:26.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:26.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:26.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:13:26.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:13:26.943 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:26.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:26.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:26.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:27.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:27.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:27.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 34 16 72374 72358 132.989 93.75 0.00791757 0.00751476 2024-03-26T03:13:27.201 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:13:27.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:27.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:27.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:27.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:27.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:13:27.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:13:27.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:13:27.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:13:27.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4060495355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:27.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: pgmap v647: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 138 MiB/s wr, 2.21k op/s; 21605/207171 objects degraded (10.429%); 1/207171 objects misplaced (0.000%) 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4060495355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: pgmap v647: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 138 MiB/s wr, 2.21k op/s; 21605/207171 objects degraded (10.429%); 1/207171 objects misplaced (0.000%) 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:13:27.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:27 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:13:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:13:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:13:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:13:27.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:13:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4060495355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: pgmap v647: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 138 MiB/s wr, 2.21k op/s; 21605/207171 objects degraded (10.429%); 1/207171 objects misplaced (0.000%) 2024-03-26T03:13:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:13:27.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:27 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:13:27.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:27.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:27.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:27.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:27.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:27.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:27.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:27.956 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:27.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:28.057 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:28.094 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:28.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 35 16 73803 73787 131.741 89.3125 0.0105339 0.00758559 2024-03-26T03:13:28.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:28.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:28.335 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:28.370 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:28.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:28.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:28.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:28 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:13:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:28 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T03:13:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:28 smithi192 ceph-mon[33110]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:13:28.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:28 smithi192 ceph-mon[33110]: osdmap e311: 8 total, 7 up, 8 in 2024-03-26T03:13:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:13:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T03:13:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[29666]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:13:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[29666]: osdmap e311: 8 total, 7 up, 8 in 2024-03-26T03:13:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:13:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T03:13:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[37920]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:13:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:28 smithi066 ceph-mon[37920]: osdmap e311: 8 total, 7 up, 8 in 2024-03-26T03:13:28.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:28.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:28.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:28.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:28.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:28.935 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:28.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:28.942 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:13:29.037 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.085 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:29.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:29.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 36 16 75212 75196 130.528 88.0625 0.0221031 0.00765553 2024-03-26T03:13:29.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:29.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:29.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:29.460 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:29 smithi192 ceph-mon[33110]: pgmap v649: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 21605/207171 objects degraded (10.429%); 1/207171 objects misplaced (0.000%) 2024-03-26T03:13:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:29 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:13:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:29 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:13:29.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 21605/207171 objects degraded (10.429%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:29.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:29 smithi192 ceph-mon[33110]: osdmap e312: 8 total, 7 up, 8 in 2024-03-26T03:13:29.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:29.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[29666]: pgmap v649: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 21605/207171 objects degraded (10.429%); 1/207171 objects misplaced (0.000%) 2024-03-26T03:13:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:13:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:13:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 21605/207171 objects degraded (10.429%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[29666]: osdmap e312: 8 total, 7 up, 8 in 2024-03-26T03:13:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[37920]: pgmap v649: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 21605/207171 objects degraded (10.429%); 1/207171 objects misplaced (0.000%) 2024-03-26T03:13:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:13:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:13:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 21605/207171 objects degraded (10.429%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:29 smithi066 ceph-mon[37920]: osdmap e312: 8 total, 7 up, 8 in 2024-03-26T03:13:29.644 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:29.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:29.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:29.884 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:29.927 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:29.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:30.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:30.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:30.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:30.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 37 16 76636 76620 129.405 89 0.0303403 0.00772246 2024-03-26T03:13:30.175 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:30.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:13:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:13:30.218 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:30.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:30.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:30.355 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:30.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:30.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:30.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:13:30.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 5.11 deep-scrub starts 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 5.11 deep-scrub ok 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:13:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:30 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:13:30.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:30.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:30.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:13:30.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 5.11 deep-scrub starts 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 5.11 deep-scrub ok 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:13:30.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 5.11 deep-scrub starts 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 5.11 deep-scrub ok 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:13:30.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:30 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:13:30.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:30.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:30.732 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:30.770 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:30.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:31.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:31.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:31.132 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 38 16 77995 77979 128.235 84.9375 0.0048475 0.00779323 2024-03-26T03:13:31.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:31.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:31.287 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:31.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:31.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:31.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:31 smithi192 ceph-mon[33110]: pgmap v651: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 105 MiB/s wr, 1.67k op/s; 23383/224742 objects degraded (10.404%); 1/224742 objects misplaced (0.000%) 2024-03-26T03:13:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:31 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:13:31.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:31 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:13:31.507 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:31.507 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[29666]: pgmap v651: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 105 MiB/s wr, 1.67k op/s; 23383/224742 objects degraded (10.404%); 1/224742 objects misplaced (0.000%) 2024-03-26T03:13:31.507 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:13:31.507 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:13:31.507 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:31.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[37920]: pgmap v651: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 105 MiB/s wr, 1.67k op/s; 23383/224742 objects degraded (10.404%); 1/224742 objects misplaced (0.000%) 2024-03-26T03:13:31.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:13:31.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:31 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:13:31.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:31.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:31.752 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:31.854 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:31.897 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:31.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:31.989 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:31.999 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.069 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:32.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:32.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 39 16 79487 79471 127.337 93.25 0.00449879 0.00784867 2024-03-26T03:13:32.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:32.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:32.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:32.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:32.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:32.461 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:13:32.461 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:13:32.461 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:13:32.461 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:13:32.462 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:32 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:13:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:32 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:13:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:32 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:13:32.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:32 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:13:32.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:32 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:13:32.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:32 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:13:32.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:32 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:13:32.493 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:32.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:32.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:32.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:32.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:32.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:32.926 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:32.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:32.932 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:13:33.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:33.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:33.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:13:33.133320+0000 min lat: 0.00101075 max lat: 0.162256 avg lat: 0.00791141 2024-03-26T03:13:33.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:13:33.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 40 16 80868 80852 126.311 86.3125 0.0124031 0.00791141 2024-03-26T03:13:33.163 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.216 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:33.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:33.269 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:13:33.269 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:13:33.269 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[29666]: pgmap v652: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s; 23822/228783 objects degraded (10.412%); 1/228783 objects misplaced (0.000%) 2024-03-26T03:13:33.269 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:13:33.269 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:13:33.270 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:13:33.270 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:13:33.270 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[37920]: pgmap v652: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s; 23822/228783 objects degraded (10.412%); 1/228783 objects misplaced (0.000%) 2024-03-26T03:13:33.270 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:13:33.270 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:13:33.279 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:13:33.279 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:13:33.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:33.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:33.452 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:33 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:13:33.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:33 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:13:33.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:33 smithi192 ceph-mon[33110]: pgmap v652: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s; 23822/228783 objects degraded (10.412%); 1/228783 objects misplaced (0.000%) 2024-03-26T03:13:33.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:33 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:13:33.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:33 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:13:33.498 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:33.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29662]: 2024-03-26T03:13:33.266+0000 7f94a71dc700 -1 mon.a@0(leader).osd e312 definitely_dead 0 2024-03-26T03:13:33.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:33.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:33.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:33.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:33.879 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:33.921 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:33.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:34.023 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:34.065 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:34.090 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:13:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:13:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:13:34.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 41 16 82279 82263 125.381 88.1875 0.0105801 0.00797104 2024-03-26T03:13:34.141 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:13:34.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:34.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:34.259 INFO:teuthology.orchestra.run.smithi066.stderr:marked out osd.0. 2024-03-26T03:13:34.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2707450116' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: from='client.25564 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: 6.a deep-scrub starts 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: 6.a deep-scrub ok 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 23822/228783 objects degraded (10.412%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:13:34.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:13:34.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2707450116' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-26T03:13:34.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: from='client.25564 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-26T03:13:34.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-26T03:13:34.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: 6.a deep-scrub starts 2024-03-26T03:13:34.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: 6.a deep-scrub ok 2024-03-26T03:13:34.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 23822/228783 objects degraded (10.412%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:34.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:34.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:34.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:13:34.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:13:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2707450116' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-26T03:13:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: from='client.25564 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-26T03:13:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-26T03:13:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: 6.a deep-scrub starts 2024-03-26T03:13:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: 6.a deep-scrub ok 2024-03-26T03:13:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 23822/228783 objects degraded (10.412%), 26 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:34.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:34.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:34.709 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.737 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:34.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:34.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:34.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:34.964 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:34.991 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:34.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:35.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:35.121 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:35.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:35.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 42 16 82426 82410 122.615 9.1875 0.030495 0.00797908 2024-03-26T03:13:35.223 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:35.255 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:35.296 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:35.359 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:35.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:35.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:35 smithi066 ceph-mon[29666]: from='client.25564 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-26T03:13:35.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:35 smithi066 ceph-mon[29666]: osdmap e313: 8 total, 7 up, 7 in 2024-03-26T03:13:35.451 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:35 smithi066 ceph-mon[29666]: pgmap v654: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 719 B/s rd, 63 MiB/s wr, 1.01k op/s; 23822/228783 objects degraded (10.412%); 1/228783 objects misplaced (0.000%) 2024-03-26T03:13:35.452 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:35 smithi066 ceph-mon[37920]: from='client.25564 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-26T03:13:35.452 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:35 smithi066 ceph-mon[37920]: osdmap e313: 8 total, 7 up, 7 in 2024-03-26T03:13:35.452 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:35 smithi066 ceph-mon[37920]: pgmap v654: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 719 B/s rd, 63 MiB/s wr, 1.01k op/s; 23822/228783 objects degraded (10.412%); 1/228783 objects misplaced (0.000%) 2024-03-26T03:13:35.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:35.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:35.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:35.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:35 smithi192 ceph-mon[33110]: from='client.25564 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-26T03:13:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:35 smithi192 ceph-mon[33110]: osdmap e313: 8 total, 7 up, 7 in 2024-03-26T03:13:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:35 smithi192 ceph-mon[33110]: pgmap v654: 178 pgs: 104 active+clean, 39 active+undersized, 26 active+undersized+degraded, 9 active+clean+remapped; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 719 B/s rd, 63 MiB/s wr, 1.01k op/s; 23822/228783 objects degraded (10.412%); 1/228783 objects misplaced (0.000%) 2024-03-26T03:13:35.741 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.770 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:35.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:35.873 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:35.905 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:35.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:36.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.037 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:36.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:36.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 43 16 82426 82410 119.764 0 - 0.00797908 2024-03-26T03:13:36.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.167 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:36.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:36.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:36.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:36.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:36.441 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:36.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:36.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[29666]: osdmap e314: 8 total, 7 up, 7 in 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:13:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:13:36.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:36 smithi066 ceph-mon[37920]: osdmap e314: 8 total, 7 up, 7 in 2024-03-26T03:13:36.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:36.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:36 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:13:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:36 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:13:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:36 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:13:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:36 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:13:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:36 smithi192 ceph-mon[33110]: osdmap e314: 8 total, 7 up, 7 in 2024-03-26T03:13:36.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:36.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:36.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:36.976 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:37.017 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:37.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:37.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 44 16 83194 83178 118.133 24 0.0112662 0.00845946 2024-03-26T03:13:37.165 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:37.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:37.313 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:37.415 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:37.425 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 312'5187 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: osdmap e315: 8 total, 7 up, 7 in 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5192 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5104 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 312'5182 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5286 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 312'5034 2024-03-26T03:13:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[29666]: pgmap v657: 178 pgs: 2 remapped+peering, 20 unknown, 42 peering, 60 active+clean, 29 active+undersized, 18 active+undersized+degraded, 7 active+clean+remapped; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 2.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 15517/247959 objects degraded (6.258%); 1/247959 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 312'5187 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: osdmap e315: 8 total, 7 up, 7 in 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5192 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5104 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 312'5182 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5286 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 312'5034 2024-03-26T03:13:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:37 smithi066 ceph-mon[37920]: pgmap v657: 178 pgs: 2 remapped+peering, 20 unknown, 42 peering, 60 active+clean, 29 active+undersized, 18 active+undersized+degraded, 7 active+clean+remapped; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 2.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 15517/247959 objects degraded (6.258%); 1/247959 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-26T03:13:37.479 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:37.580 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.622 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:37.723 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:13:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:13:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 312'5187 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: osdmap e315: 8 total, 7 up, 7 in 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5192 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5104 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 312'5182 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 312'5286 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 312'5034 2024-03-26T03:13:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:37 smithi192 ceph-mon[33110]: pgmap v657: 178 pgs: 2 remapped+peering, 20 unknown, 42 peering, 60 active+clean, 29 active+undersized, 18 active+undersized+degraded, 7 active+clean+remapped; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 2.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 15517/247959 objects degraded (6.258%); 1/247959 objects misplaced (0.000%); 13 KiB/s, 0 objects/s recovering 2024-03-26T03:13:37.762 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:37.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:37.908 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:37.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:38.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:38.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:38.134 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 45 16 84288 84272 117.027 68.375 0.0106643 0.008534 2024-03-26T03:13:38.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:38.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:38.291 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:38.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:38.384 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:13:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:38.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:38.419 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:13:38.457 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:13:38.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:38.506 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:13:38.506 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:38 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 14 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-26T03:13:38.506 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:38 smithi066 ceph-mon[29666]: osdmap e316: 8 total, 7 up, 7 in 2024-03-26T03:13:38.506 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:38 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 14 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-26T03:13:38.506 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:38 smithi066 ceph-mon[37920]: osdmap e316: 8 total, 7 up, 7 in 2024-03-26T03:13:38.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:38.548 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:13:38.600 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:13:38.638 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:13:38.639 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:13:38.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:38.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:38 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 14 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-26T03:13:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:38 smithi192 ceph-mon[33110]: osdmap e316: 8 total, 7 up, 7 in 2024-03-26T03:13:38.779 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:38.822 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:38.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:38.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:38.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:39.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:39.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:39.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:39.133 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 46 16 85321 85305 115.886 64.5625 0.0181271 0.00862363 2024-03-26T03:13:39.205 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:39.247 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:39.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:39.349 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:39.389 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:39.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:39.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:39.578 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[29666]: osdmap e317: 8 total, 7 up, 7 in 2024-03-26T03:13:39.578 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[29666]: pgmap v660: 178 pgs: 2 remapped+peering, 20 unknown, 42 peering, 60 active+clean, 29 active+undersized, 18 active+undersized+degraded, 7 active+clean+remapped; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 3.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 15517/247959 objects degraded (6.258%); 1/247959 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-26T03:13:39.578 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/799114751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:39.578 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 15517/247959 objects degraded (6.258%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:39.578 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[29666]: osdmap e318: 8 total, 7 up, 7 in 2024-03-26T03:13:39.579 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[37920]: osdmap e317: 8 total, 7 up, 7 in 2024-03-26T03:13:39.579 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[37920]: pgmap v660: 178 pgs: 2 remapped+peering, 20 unknown, 42 peering, 60 active+clean, 29 active+undersized, 18 active+undersized+degraded, 7 active+clean+remapped; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 3.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 15517/247959 objects degraded (6.258%); 1/247959 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-26T03:13:39.579 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/799114751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:39.579 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 15517/247959 objects degraded (6.258%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:39.579 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:39 smithi066 ceph-mon[37920]: osdmap e318: 8 total, 7 up, 7 in 2024-03-26T03:13:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:13:39.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:13:39.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:39.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:39.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:39.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:39 smithi192 ceph-mon[33110]: osdmap e317: 8 total, 7 up, 7 in 2024-03-26T03:13:39.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:39 smithi192 ceph-mon[33110]: pgmap v660: 178 pgs: 2 remapped+peering, 20 unknown, 42 peering, 60 active+clean, 29 active+undersized, 18 active+undersized+degraded, 7 active+clean+remapped; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 3.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 15517/247959 objects degraded (6.258%); 1/247959 objects misplaced (0.000%); 20 KiB/s, 0 objects/s recovering 2024-03-26T03:13:39.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:39 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/799114751' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:39.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 15517/247959 objects degraded (6.258%), 18 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:39.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:39 smithi192 ceph-mon[33110]: osdmap e318: 8 total, 7 up, 7 in 2024-03-26T03:13:39.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:39.842 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:39.892 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:13:39.892 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:13:39.892 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:13:39.892 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:13:39.892 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:13:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:39.894 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:39.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:39.995 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:40.034 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:40.134 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 47 16 86388 86372 114.839 66.6875 0.0102747 0.00870265 2024-03-26T03:13:40.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:40.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:40.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:13:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:13:40.280 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.323 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:40.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:40.424 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:40.468 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:40.570 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.615 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:40.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:40.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:40.752 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:40.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[29666]: pgmap v662: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 9.1 KiB/s rd, 51 MiB/s wr, 821 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 35 MiB/s, 577 objects/s recovering 2024-03-26T03:13:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:40.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:40 smithi066 ceph-mon[37920]: pgmap v662: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 9.1 KiB/s rd, 51 MiB/s wr, 821 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 35 MiB/s, 577 objects/s recovering 2024-03-26T03:13:40.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:40.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:40.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:40.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:40.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:40 smithi192 ceph-mon[33110]: pgmap v662: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 9.1 KiB/s rd, 51 MiB/s wr, 821 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 35 MiB/s, 577 objects/s recovering 2024-03-26T03:13:41.005 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.060 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:41.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:41.067 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:13:41.134 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 48 16 87485 87469 113.873 68.5625 0.00915545 0.00877443 2024-03-26T03:13:41.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:41.213 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:41.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:41.364 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:41.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:41.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:41.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:13:41.606 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:41.633 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:41.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:41.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:41.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:41 smithi066 ceph-mon[29666]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:13:41.812 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:41 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 7 pgs peering) 2024-03-26T03:13:41.812 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:41 smithi066 ceph-mon[37920]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:13:41.812 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:41 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 7 pgs peering) 2024-03-26T03:13:41.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:41 smithi192 ceph-mon[33110]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:13:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:41 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 7 pgs peering) 2024-03-26T03:13:41.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:41.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:41.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:42.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:42.108 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:42.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 49 16 88469 88453 112.804 61.5 0.0804614 0.00885906 2024-03-26T03:13:42.187 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:13:42.188 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:13:42.188 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T03:13:42.188 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.651006 2024-03-26T03:13:42.188 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 2 0.6510063018087251 2024-03-26T03:13:42.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.278 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:42.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:42.379 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.423 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:42.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:42.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:42.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:42.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:42.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:42.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:42.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:42.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:42.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:42.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:42 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:42 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:42 smithi192 ceph-mon[33110]: osdmap e319: 8 total, 7 up, 7 in 2024-03-26T03:13:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:42 smithi192 ceph-mon[33110]: pgmap v664: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 7.3 KiB/s rd, 41 MiB/s wr, 661 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 28 MiB/s, 465 objects/s recovering 2024-03-26T03:13:43.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:43.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:43.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:43.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[29666]: osdmap e319: 8 total, 7 up, 7 in 2024-03-26T03:13:43.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[29666]: pgmap v664: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 7.3 KiB/s rd, 41 MiB/s wr, 661 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 28 MiB/s, 465 objects/s recovering 2024-03-26T03:13:43.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:43.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:43.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[37920]: osdmap e319: 8 total, 7 up, 7 in 2024-03-26T03:13:43.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:42 smithi066 ceph-mon[37920]: pgmap v664: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 7.3 KiB/s rd, 41 MiB/s wr, 661 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 28 MiB/s, 465 objects/s recovering 2024-03-26T03:13:43.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:43.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 50 16 89428 89412 111.747 59.9375 0.00782051 0.008938 2024-03-26T03:13:43.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:43.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:43.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:43.405 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:43.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:43.549 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:43.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:43.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:43.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:43.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:43.947 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:43.997 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:43.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:44.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:13:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:13:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:13:44.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 51 16 90455 90439 110.814 64.1875 0.013859 0.00901767 2024-03-26T03:13:44.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:44.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:44.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:44.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:44.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:44.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:44.431 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:44.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:44.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 26391/257814 objects degraded (10.236%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:44.442 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:44 smithi066 ceph-mon[29666]: osdmap e320: 8 total, 7 up, 7 in 2024-03-26T03:13:44.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 26391/257814 objects degraded (10.236%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:44.442 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:44 smithi066 ceph-mon[37920]: osdmap e320: 8 total, 7 up, 7 in 2024-03-26T03:13:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 26391/257814 objects degraded (10.236%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:44 smithi192 ceph-mon[33110]: osdmap e320: 8 total, 7 up, 7 in 2024-03-26T03:13:44.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:44.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:44.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:44.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:44.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:44.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:44.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:44.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:44.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:44.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:45.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:45.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:45.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 52 16 91544 91528 109.992 68.0625 0.00471148 0.00908593 2024-03-26T03:13:45.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:45.198 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:13:45.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:45.311 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:45.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:45 smithi066 ceph-mon[29666]: pgmap v666: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 6.2 KiB/s rd, 34 MiB/s wr, 555 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 24 MiB/s, 390 objects/s recovering 2024-03-26T03:13:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2171835780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:13:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:45 smithi066 ceph-mon[37920]: pgmap v666: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 6.2 KiB/s rd, 34 MiB/s wr, 555 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 24 MiB/s, 390 objects/s recovering 2024-03-26T03:13:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2171835780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:13:45.414 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.455 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:45.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:45.490 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:45 smithi192 ceph-mon[33110]: pgmap v666: 178 pgs: 4 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 6.2 KiB/s rd, 34 MiB/s wr, 555 op/s; 26391/257814 objects degraded (10.236%); 51450/257814 objects misplaced (19.956%); 24 MiB/s, 390 objects/s recovering 2024-03-26T03:13:45.491 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2171835780' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:13:45.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:45.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:45.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:45.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:45.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:45.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:45.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:46.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:46.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:46.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 53 16 92603 92587 109.165 66.1875 0.00919365 0.00915482 2024-03-26T03:13:46.140 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:46.186 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:46.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:46.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:46.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:46.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:46 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:13:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2171835780' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:13:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:46 smithi192 ceph-mon[33110]: osdmap e321: 8 total, 7 up, 7 in 2024-03-26T03:13:46.567 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:46.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:46 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:13:46.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2171835780' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:13:46.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:46 smithi066 ceph-mon[37920]: osdmap e321: 8 total, 7 up, 7 in 2024-03-26T03:13:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:46 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:13:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2171835780' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:13:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:46 smithi066 ceph-mon[29666]: osdmap e321: 8 total, 7 up, 7 in 2024-03-26T03:13:46.610 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:46.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.754 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:46.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:46.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:46.890 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:46.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:46.992 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:47.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:47.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 54 16 93725 93709 108.442 70.125 0.00562474 0.00921463 2024-03-26T03:13:47.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:47.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:47.202 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:47.207 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:47.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:47.324 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:47.350 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:47 smithi066 ceph-mon[29666]: pgmap v668: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 858 op/s; 27256/273252 objects degraded (9.975%); 51174/273252 objects misplaced (18.728%); 38 MiB/s, 609 objects/s recovering 2024-03-26T03:13:47.351 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:47 smithi066 ceph-mon[37920]: pgmap v668: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 858 op/s; 27256/273252 objects degraded (9.975%); 51174/273252 objects misplaced (18.728%); 38 MiB/s, 609 objects/s recovering 2024-03-26T03:13:47.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:47.475 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:47 smithi192 ceph-mon[33110]: pgmap v668: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 858 op/s; 27256/273252 objects degraded (9.975%); 51174/273252 objects misplaced (18.728%); 38 MiB/s, 609 objects/s recovering 2024-03-26T03:13:47.578 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:47.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:47.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.775 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:47.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:47.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:47.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:47.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:48.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:48.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:283819 9.6 9:6e668c94:::benchmark_data_smithi066_125102_object137518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:05.009627+0000", 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 103.08007402299999, 2024-03-26T03:13:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14808533300000001, 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 283819 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009627+0000", 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009627+0000", 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009629+0000", 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.996e-06 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009635+0000", 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.195e-06 2024-03-26T03:13:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009648+0000", 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3369e-05 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009657+0000", 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5820000000000002e-06 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009717+0000", 2024-03-26T03:13:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9722999999999997e-05 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009852+0000", 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013481100000000001 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.009981+0000", 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012974500000000001 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013332+0000", 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003351199 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.017952+0000", 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046191540000000003 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.157599+0000", 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13964728700000001 2024-03-26T03:13:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.157644+0000", 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4669999999999998e-05 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.157712+0000", 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8602000000000005e-05 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:283825 9.6 9:6195b2dd:::benchmark_data_smithi066_125102_object137524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:05.013927+0000", 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 103.075773914, 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.14411210499999999, 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:13:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 283825 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013927+0000", 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013927+0000", 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013929+0000", 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8509999999999999e-06 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013935+0000", 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0639999999999999e-06 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013948+0000", 2024-03-26T03:13:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3081000000000001e-05 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.013957+0000", 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9630000000000004e-06 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.014010+0000", 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3498999999999997e-05 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.103 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.014133+0000", 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012212599999999999 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.014227+0000", 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4432000000000007e-05 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.016214+0000", 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0019870880000000001 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.104 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.018000+0000", 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017856790000000001 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.157937+0000", 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13993683800000001 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.157995+0000", 2024-03-26T03:13:48.105 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7944e-05 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:05.158039+0000", 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4539999999999997e-05 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:86119 10.5 10:aa0bb7b1:::benchmark_data_smithi066_148944_object86118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:39.859621+0000", 2024-03-26T03:13:48.106 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 8.2300800469999995, 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.108473533, 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.107 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 86119 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.859621+0000", 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.108 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.859621+0000", 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.859622+0000", 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5099999999999999e-07 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.859642+0000", 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0194999999999999e-05 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.109 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.859652+0000", 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8810000000000008e-06 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.859657+0000", 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7749999999999998e-06 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.946414+0000", 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.086756762000000001 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.110 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.946598+0000", 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000183411 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.946787+0000", 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018988499999999999 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.963646+0000", 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.016858788 2024-03-26T03:13:48.111 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.963691+0000", 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5154999999999997e-05 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.963743+0000", 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.134e-05 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.963797+0000", 2024-03-26T03:13:48.112 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4403000000000001e-05 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.968094+0000", 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042972870000000003 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:86132 10.5 10:a6e2048b:::benchmark_data_smithi066_148944_object86131:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:39.877831+0000", 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 8.2118701709999993, 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10529140200000001, 2024-03-26T03:13:48.113 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 86132 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.877831+0000", 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.114 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.877831+0000", 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.877833+0000", 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9180000000000001e-06 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.115 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.877855+0000", 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2269e-05 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.877867+0000", 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1711e-05 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.877874+0000", 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.841e-06 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.116 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.949459+0000", 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.071584738999999994 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.949602+0000", 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000142829 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.949769+0000", 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016701899999999999 2024-03-26T03:13:48.117 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.963698+0000", 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013929113 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.968150+0000", 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0044520569999999997 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.118 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.978769+0000", 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010618613000000001 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.978912+0000", 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000143036 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:39.983122+0000", 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042102570000000002 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.119 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:87474 10.5 10:a94b078f:::benchmark_data_smithi066_148944_object87473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:41.123293+0000", 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 6.9664075690000002, 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11242368699999999, 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 87474 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.120 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.123293+0000", 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.123293+0000", 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.123295+0000", 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.142e-06 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.121 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.123320+0000", 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4326999999999999e-05 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.123330+0000", 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0396e-05 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.123338+0000", 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1349999999999992e-06 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.122 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.169531+0000", 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.046192426000000002 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.169718+0000", 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018752299999999999 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.123 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.169915+0000", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000196529 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.181869+0000", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011953992 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.193349+0000", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011480209 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.124 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.231451+0000", 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.038101533 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.231562+0000", 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011139199999999999 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.235717+0000", 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041550830000000004 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.125 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:87481 10.5 10:a47fd9f7:::benchmark_data_smithi066_148944_object87480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:41.131035+0000", 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 6.9586654939999999, 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10932452600000001, 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.126 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 87481 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.131035+0000", 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.131035+0000", 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.127 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.131037+0000", 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0310000000000001e-06 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.131063+0000", 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5567000000000001e-05 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.131079+0000", 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5892999999999998e-05 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.128 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.131087+0000", 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2609999999999993e-06 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.172528+0000", 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.041440449999999997 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.172713+0000", 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018514099999999999 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.129 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.172945+0000", 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00023224699999999999 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.181933+0000", 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.008987564 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.193412+0000", 2024-03-26T03:13:48.130 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011479451 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.235783+0000", 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.042370788 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.235876+0000", 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3190999999999996e-05 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:41.240360+0000", 2024-03-26T03:13:48.131 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0044839420000000003 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89174 10.5 10:a0939d52:::benchmark_data_smithi066_148944_object89173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.807289+0000", 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.2824116629999995, 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.107323474, 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.132 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89174 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.807289+0000", 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.807289+0000", 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.133 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.807291+0000", 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8819999999999999e-06 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.807318+0000", 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6979999999999999e-05 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.807328+0000", 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.859e-06 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.134 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.135 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.807333+0000", 2024-03-26T03:13:48.135 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3009999999999997e-06 2024-03-26T03:13:48.135 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.135 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.135 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.859868+0000", 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052534408999999997 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.860167+0000", 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00029964500000000002 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.860498+0000", 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00033050199999999999 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.136 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.876355+0000", 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.015857310999999999 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.876411+0000", 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6329000000000002e-05 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.909943+0000", 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.03353192 2024-03-26T03:13:48.137 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.910009+0000", 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5242999999999999e-05 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.914613+0000", 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046040930000000001 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.138 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89177 10.5 10:ae64499f:::benchmark_data_smithi066_148944_object89176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.808632+0000", 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.2810686750000002, 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11013413, 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89177 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.139 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.808632+0000", 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.808632+0000", 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.808634+0000", 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4449999999999999e-06 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.140 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.808657+0000", 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2853e-05 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.808667+0000", 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0569e-05 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.808674+0000", 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1899999999999998e-06 2024-03-26T03:13:48.141 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.873306+0000", 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.064631258999999996 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.873482+0000", 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000176269 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.873694+0000", 2024-03-26T03:13:48.142 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000212368 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.884788+0000", 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011094062999999999 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.909715+0000", 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.024926488 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.914629+0000", 2024-03-26T03:13:48.143 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049144009999999997 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.914685+0000", 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6048000000000003e-05 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.918766+0000", 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0040811770000000001 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.144 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89179 10.5 10:a9534145:::benchmark_data_smithi066_148944_object89178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.811596+0000", 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.2781049170000003, 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.114279204, 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89179 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.145 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.811596+0000", 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.811596+0000", 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.811597+0000", 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4130000000000001e-06 2024-03-26T03:13:48.146 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.811618+0000", 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0273999999999999e-05 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.811625+0000", 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6750000000000002e-06 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.811630+0000", 2024-03-26T03:13:48.147 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0209999999999999e-06 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.876476+0000", 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.064845416000000003 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.876638+0000", 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000161912 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.148 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.876818+0000", 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000180111 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.909770+0000", 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.032952533999999999 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.909814+0000", 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4147999999999998e-05 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.149 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.921390+0000", 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011575627 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.921436+0000", 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5461999999999999e-05 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.925875+0000", 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0044396110000000004 2024-03-26T03:13:48.150 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89181 10.5 10:aefa80bb:::benchmark_data_smithi066_148944_object89180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.814650+0000", 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.2750511040000001, 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.129001435, 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.151 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89181 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814650+0000", 2024-03-26T03:13:48.153 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814650+0000", 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814651+0000", 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5460000000000001e-06 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814675+0000", 2024-03-26T03:13:48.154 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3853999999999999e-05 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814685+0000", 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7310000000000005e-06 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814693+0000", 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.695e-06 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.881280+0000", 2024-03-26T03:13:48.155 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.066587438999999998 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.881452+0000", 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017241899999999999 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.881662+0000", 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000209972 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.156 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.909860+0000", 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.028197092 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.909924+0000", 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4855000000000002e-05 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.940032+0000", 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.030107144999999998 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.157 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.940071+0000", 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9283999999999999e-05 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.943651+0000", 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0035804029999999998 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89183 10.5 10:a03643c8:::benchmark_data_smithi066_148944_object89182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.814941+0000", 2024-03-26T03:13:48.158 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.2747595729999999, 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13167900099999999, 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89183 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814941+0000", 2024-03-26T03:13:48.159 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814941+0000", 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814942+0000", 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3900000000000003e-07 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.160 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814961+0000", 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8555000000000001e-05 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814968+0000", 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0310000000000001e-06 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.814974+0000", 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6440000000000003e-06 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.161 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.884823+0000", 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.069848533000000004 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.884980+0000", 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015731400000000001 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.162 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.885163+0000", 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018282500000000001 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.918826+0000", 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033662340999999998 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.928717+0000", 2024-03-26T03:13:48.163 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0098912449999999999 2024-03-26T03:13:48.164 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 55 16 94802 94786 107.694 67.3125 0.0105091 0.0092803 2024-03-26T03:13:48.164 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.164 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.164 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.164 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.943662+0000", 2024-03-26T03:13:48.164 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014945082 2024-03-26T03:13:48.164 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.943699+0000", 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6884000000000001e-05 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.946620+0000", 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002921608 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.165 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89302 10.5 10:a4ea7334:::benchmark_data_smithi066_148944_object89301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.957277+0000", 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.1324236479999996, 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.122099867, 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89302 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.166 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.957277+0000", 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.957277+0000", 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.957279+0000", 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.855e-06 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.167 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.957310+0000", 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.1167999999999997e-05 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.957323+0000", 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2767e-05 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.957331+0000", 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7819999999999994e-06 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.168 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.971927+0000", 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014596381 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.972025+0000", 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8088e-05 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.972175+0000", 2024-03-26T03:13:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000149321 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.981932+0000", 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0097574189999999998 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.981960+0000", 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.8313000000000001e-05 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.170 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.015210+0000", 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033249472000000002 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.015271+0000", 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1408999999999994e-05 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.079377+0000", 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.064105891999999998 2024-03-26T03:13:48.171 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89352 10.5 10:a2f8d333:::benchmark_data_smithi066_148944_object89351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.007006+0000", 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.082695041, 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.154437039, 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.172 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89352 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.007006+0000", 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.007006+0000", 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.173 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.007007+0000", 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4309999999999999e-06 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.007035+0000", 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7229999999999998e-05 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.007043+0000", 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1880000000000002e-06 2024-03-26T03:13:48.174 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.007048+0000", 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3199999999999999e-06 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.087924+0000", 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.080875660000000002 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.088066+0000", 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014247199999999999 2024-03-26T03:13:48.175 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.121346+0000", 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033279909000000003 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.129167+0000", 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0078211860000000008 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.138066+0000", 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0088983810000000003 2024-03-26T03:13:48.176 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.157066+0000", 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019000622000000002 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.157103+0000", 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6742000000000002e-05 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.177 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.161443+0000", 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004339898 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89363 10.5 10:ad9e00bd:::benchmark_data_smithi066_148944_object89362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.012217+0000", 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.0774840560000003, 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.153598295, 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.178 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89363 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.012217+0000", 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.012217+0000", 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.179 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.012218+0000", 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2950000000000001e-06 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.012242+0000", 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3879e-05 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.180 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.012252+0000", 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5990000000000008e-06 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.012259+0000", 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0060000000000003e-06 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.126374+0000", 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.114115357 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.181 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.126543+0000", 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016867799999999999 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.126661+0000", 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118541 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.182 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.138047+0000", 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011385664 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.138073+0000", 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6534999999999999e-05 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.161655+0000", 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.023581838000000001 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.183 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.161727+0000", 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1403999999999996e-05 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.165815+0000", 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0040884989999999998 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.184 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89365 10.5 10:a0fdefae:::benchmark_data_smithi066_148944_object89364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.014356+0000", 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.0753448749999999, 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.15484872199999999, 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89365 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.185 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.014356+0000", 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.014356+0000", 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.014357+0000", 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3170000000000001e-06 2024-03-26T03:13:48.186 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.014381+0000", 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4028000000000001e-05 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.014392+0000", 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0179e-05 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.014399+0000", 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1010000000000003e-06 2024-03-26T03:13:48.187 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.129209+0000", 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11481053400000001 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.129388+0000", 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000178522 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.129604+0000", 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000215956 2024-03-26T03:13:48.188 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.138081+0000", 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0084770839999999993 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.138138+0000", 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7224e-05 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.165892+0000", 2024-03-26T03:13:48.189 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.027753565000000001 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.166009+0000", 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117582 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.169205+0000", 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031956300000000001 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.190 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "MOSDPGPushReply(10.5 319/314 [PushReplyOp(10:a454adb1:::benchmark_data_smithi066_148944_object1593:head)])", 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.018112+0000", 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.0715888199999997, 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.13886551499999999, 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.191 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.018112+0000", 2024-03-26T03:13:48.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.018112+0000", 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.193 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.018114+0000", 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.018118+0000", 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.42e-06 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.075802+0000", 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.057683676000000003 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.194 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.075812+0000", 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0555e-05 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.149400+0000", 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.073587446000000001 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.156978+0000", 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0075778180000000001 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.195 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:89387 10.5 10:a4fc3aa4:::benchmark_data_smithi066_148944_object89386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.077003+0000", 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 5.0126982409999998, 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.105160961, 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.196 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 89387 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.077003+0000", 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.077003+0000", 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.197 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.077004+0000", 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2899999999999999e-06 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.077026+0000", 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2118000000000001e-05 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.077036+0000", 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0149999999999999e-05 2024-03-26T03:13:48.198 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.077044+0000", 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6070000000000002e-06 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.135299+0000", 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.05825553 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.135472+0000", 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017261800000000001 2024-03-26T03:13:48.199 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.135661+0000", 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018861999999999999 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.141096+0000", 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0054350969999999998 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.144255+0000", 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031588739999999999 2024-03-26T03:13:48.200 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.178257+0000", 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.034002686999999997 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.178344+0000", 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6394e-05 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.182164+0000", 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038199760000000001 2024-03-26T03:13:48.201 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:90057 10.5 10:ad0e9d88:::benchmark_data_smithi066_148944_object90056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.718921+0000", 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 4.3707794169999996, 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.108520135, 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.202 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 90057 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.718921+0000", 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.718921+0000", 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.203 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.718928+0000", 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.844e-06 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.718969+0000", 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0714999999999999e-05 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.718990+0000", 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0509e-05 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.204 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.718999+0000", 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8209999999999997e-06 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728883+0000", 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.009883223 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.729096+0000", 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021310799999999999 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.205 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.729298+0000", 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020235500000000001 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.738743+0000", 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0094447969999999996 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.742956+0000", 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042130600000000002 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.206 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.817062+0000", 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.074106167000000001 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.817123+0000", 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1099999999999994e-05 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.827442+0000", 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010318436 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.207 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:90063 10.5 10:a60fef6e:::benchmark_data_smithi066_148944_object90062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.722374+0000", 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 4.3673267090000003, 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.108994551, 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.208 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 90063 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.722374+0000", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.722374+0000", 2024-03-26T03:13:48.209 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.722375+0000", 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3e-07 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.722406+0000", 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.1012000000000001e-05 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.210 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.722413+0000", 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8970000000000002e-06 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.722419+0000", 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5989999999999998e-06 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.733172+0000", 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010752404 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.211 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.733326+0000", 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000154186 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.733504+0000", 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000177653 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.738750+0000", 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0052461110000000004 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.212 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.743011+0000", 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042607900000000004 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.827505+0000", 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.084494824999999996 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.213 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.827570+0000", 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4807000000000005e-05 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.214 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.831369+0000", 2024-03-26T03:13:48.215 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0037985369999999998 2024-03-26T03:13:48.215 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.215 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.215 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.215 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.215 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:90074 10.5 10:a16cf7c6:::benchmark_data_smithi066_148944_object90073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:43.728943+0000", 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 4.3607577649999998, 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10642851, 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:13:48.216 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 90074 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.217 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728943+0000", 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728943+0000", 2024-03-26T03:13:48.218 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:13:48.219 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.219 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.219 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:13:48.219 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728945+0000", 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8169999999999999e-06 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728968+0000", 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2799e-05 2024-03-26T03:13:48.220 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728981+0000", 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3213999999999999e-05 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.221 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:13:48.222 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.728989+0000", 2024-03-26T03:13:48.222 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1559999999999995e-06 2024-03-26T03:13:48.222 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.222 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.222 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:13:48.222 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.736526+0000", 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0075364869999999997 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.736642+0000", 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000116846 2024-03-26T03:13:48.223 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.224 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.224 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:13:48.224 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.736761+0000", 2024-03-26T03:13:48.224 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118434 2024-03-26T03:13:48.224 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.224 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.742971+0000", 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0062098040000000002 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.225 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.807794+0000", 2024-03-26T03:13:48.226 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.064822925000000003 2024-03-26T03:13:48.226 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.226 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.226 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:13:48.226 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.831423+0000", 2024-03-26T03:13:48.226 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.02362906 2024-03-26T03:13:48.227 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.227 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.227 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:13:48.227 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.831474+0000", 2024-03-26T03:13:48.227 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1480999999999997e-05 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:43.835372+0000", 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038974869999999998 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.228 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.229 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.229 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:13:48.229 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:13:48.229 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:13:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:48.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:48.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:48 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:48.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:48 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:48.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:48 smithi066 ceph-mon[29666]: osdmap e322: 8 total, 7 up, 7 in 2024-03-26T03:13:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:48 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:48 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:48 smithi066 ceph-mon[37920]: osdmap e322: 8 total, 7 up, 7 in 2024-03-26T03:13:48.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:48.402 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:13:48.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:48.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:48 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:48 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:48.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:48 smithi192 ceph-mon[33110]: osdmap e322: 8 total, 7 up, 7 in 2024-03-26T03:13:48.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:48.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:48.657 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:48.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:48.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:48.905 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:48.952 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:48.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:49.055 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:49.097 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:49.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 56 15 95886 95871 106.982 67.8125 0.0319776 0.00934267 2024-03-26T03:13:49.200 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.246 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:49.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:49.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:49.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:49.472 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:49 smithi066 ceph-mon[29666]: pgmap v670: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 858 op/s; 27256/273252 objects degraded (9.975%); 51174/273252 objects misplaced (18.728%); 38 MiB/s, 609 objects/s recovering 2024-03-26T03:13:49.472 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 27256/273252 objects degraded (9.975%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:49.472 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:49 smithi066 ceph-mon[37920]: pgmap v670: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 858 op/s; 27256/273252 objects degraded (9.975%); 51174/273252 objects misplaced (18.728%); 38 MiB/s, 609 objects/s recovering 2024-03-26T03:13:49.473 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 27256/273252 objects degraded (9.975%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:49.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:49 smithi192 ceph-mon[33110]: pgmap v670: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 858 op/s; 27256/273252 objects degraded (9.975%); 51174/273252 objects misplaced (18.728%); 38 MiB/s, 609 objects/s recovering 2024-03-26T03:13:49.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 27256/273252 objects degraded (9.975%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:49.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:49.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:49.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:49.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:49.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:49.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:49.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:49.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:50.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:50.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:50.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 57 16 96982 96966 106.306 68.4375 0.00631404 0.00940213 2024-03-26T03:13:50.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:50.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:50.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:13:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:13:50.330 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:50.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.387 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:50.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:50.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:13:50.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:50.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:50.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:50.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:50.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:50.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:50.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:50.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:50.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:51.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:51.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 58 16 98027 98011 105.599 65.3125 0.018916 0.00946419 2024-03-26T03:13:51.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:51.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:51.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:51.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:51.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:51.386 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:51.426 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:51.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:51.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:51.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:51.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:51 smithi066 ceph-mon[29666]: pgmap v671: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.6 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 72 MiB/s, 1.15k objects/s recovering 2024-03-26T03:13:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:51 smithi066 ceph-mon[37920]: pgmap v671: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.6 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 72 MiB/s, 1.15k objects/s recovering 2024-03-26T03:13:51.679 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:51.718 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:51.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:51.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:13:51.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:51 smithi192 ceph-mon[33110]: pgmap v671: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.6 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 72 MiB/s, 1.15k objects/s recovering 2024-03-26T03:13:51.767 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:13:51.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:51.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:51.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:51.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:52.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:52.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:52.135 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 59 16 99001 98985 104.841 60.875 0.00693251 0.00953167 2024-03-26T03:13:52.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:52.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:52.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:52.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:52.478 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:52.580 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:52.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:52.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:52.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:52.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:52.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:52.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:52.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:53.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:53.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:13:53.136056+0000 min lat: 0.00101075 max lat: 2.23537 avg lat: 0.00960701 2024-03-26T03:13:53.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:13:53.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 60 16 99885 99869 104.015 55.25 0.00744889 0.00960701 2024-03-26T03:13:53.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:53.177 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:53.279 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.333 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:53.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:53.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:53.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:53 smithi066 ceph-mon[29666]: pgmap v672: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 56 MiB/s, 891 objects/s recovering 2024-03-26T03:13:53.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:53 smithi066 ceph-mon[37920]: pgmap v672: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 56 MiB/s, 891 objects/s recovering 2024-03-26T03:13:53.572 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.621 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:53.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:53.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:53 smithi192 ceph-mon[33110]: pgmap v672: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 56 MiB/s, 891 objects/s recovering 2024-03-26T03:13:53.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:53.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:53.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:53.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:53.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:54.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:54.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:54.087 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:13:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:13:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:13:54.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 61 16 100815 100799 103.262 58.125 0.0181567 0.00967384 2024-03-26T03:13:54.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:54.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:54.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:54.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:54.341 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:54.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:54.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:54.458 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:13:54.497 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:13:54.539 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:13:54.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:54.580 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:13:54.592 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:54.619 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:13:54.647 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:13:54.647 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 28195/289557 objects degraded (9.737%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:54.647 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 28195/289557 objects degraded (9.737%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:54.677 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:13:54.678 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:13:54.679 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:13:54.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:54.698 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 28195/289557 objects degraded (9.737%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:54.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:54.846 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:54.891 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:54.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:54.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:55.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:55.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:55.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 62 16 101767 101751 102.556 59.5 0.0153286 0.00974281 2024-03-26T03:13:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:55.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:55.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:55.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:55.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:55.417 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.2 primary-affinity to 0.651006 (8426642) 2024-03-26T03:13:55.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:55.540 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:55.583 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:55.685 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:55.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:55 smithi192 ceph-mon[33110]: pgmap v673: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 561 B/s rd, 37 MiB/s wr, 596 op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 24 MiB/s, 380 objects/s recovering 2024-03-26T03:13:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/58139924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1582585439' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]: dispatch 2024-03-26T03:13:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:55 smithi192 ceph-mon[33110]: from='client.35603 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]: dispatch 2024-03-26T03:13:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:55.747 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[29666]: pgmap v673: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 561 B/s rd, 37 MiB/s wr, 596 op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 24 MiB/s, 380 objects/s recovering 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/58139924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1582585439' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[29666]: from='client.35603 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[37920]: pgmap v673: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 561 B/s rd, 37 MiB/s wr, 596 op/s; 28195/289557 objects degraded (9.737%); 51037/289557 objects misplaced (17.626%); 24 MiB/s, 380 objects/s recovering 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/58139924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1582585439' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[37920]: from='client.35603 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]: dispatch 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:13:55.748 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:13:55.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:55.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:55.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:55.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:55.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:56.098 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:56.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 63 16 102625 102609 101.78 53.625 0.0117353 0.00982041 2024-03-26T03:13:56.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:56.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:13:56.262 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:13:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:56.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:56.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.428 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:56.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:56.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:56.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:56 smithi066 ceph-mon[29666]: from='client.35603 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]': finished 2024-03-26T03:13:56.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:56 smithi066 ceph-mon[29666]: osdmap e323: 8 total, 7 up, 7 in 2024-03-26T03:13:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:56 smithi066 ceph-mon[37920]: from='client.35603 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]': finished 2024-03-26T03:13:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:56 smithi066 ceph-mon[37920]: osdmap e323: 8 total, 7 up, 7 in 2024-03-26T03:13:56.664 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:56.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:56 smithi192 ceph-mon[33110]: from='client.35603 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6510063018087251}]': finished 2024-03-26T03:13:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:56 smithi192 ceph-mon[33110]: osdmap e323: 8 total, 7 up, 7 in 2024-03-26T03:13:56.809 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:56.849 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:56.951 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:56.981 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:56.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:57.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:57.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 64 16 103587 103571 101.129 60.125 0.0142538 0.00988332 2024-03-26T03:13:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:57.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:57.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:57.298 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:57.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:57.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:57.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:57.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:57.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:13:57.715 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:13:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:57 smithi192 ceph-mon[33110]: pgmap v675: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.13k op/s; 28964/303996 objects degraded (9.528%); 50339/303996 objects misplaced (16.559%); 48 MiB/s, 767 objects/s recovering 2024-03-26T03:13:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:57 smithi192 ceph-mon[33110]: osdmap e324: 8 total, 7 up, 7 in 2024-03-26T03:13:57.803 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:57 smithi066 ceph-mon[29666]: pgmap v675: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.13k op/s; 28964/303996 objects degraded (9.528%); 50339/303996 objects misplaced (16.559%); 48 MiB/s, 767 objects/s recovering 2024-03-26T03:13:57.803 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:57 smithi066 ceph-mon[29666]: osdmap e324: 8 total, 7 up, 7 in 2024-03-26T03:13:57.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:57 smithi066 ceph-mon[37920]: pgmap v675: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.13k op/s; 28964/303996 objects degraded (9.528%); 50339/303996 objects misplaced (16.559%); 48 MiB/s, 767 objects/s recovering 2024-03-26T03:13:57.803 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:57 smithi066 ceph-mon[37920]: osdmap e324: 8 total, 7 up, 7 in 2024-03-26T03:13:57.817 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.863 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:57.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:57.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:57.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:57.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:58.094 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:58.136 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 65 16 104404 104388 100.358 51.0625 0.0113761 0.00995776 2024-03-26T03:13:58.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:58.241 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:58.263 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:13:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:58.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:58.386 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.438 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:58.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:58.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:13:58.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:13:58.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:13:58.700 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:13:58.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:13:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:58 smithi192 ceph-mon[33110]: osdmap e325: 8 total, 7 up, 7 in 2024-03-26T03:13:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:58 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:58 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:58 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:13:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:58 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:13:58.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:58.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[29666]: osdmap e325: 8 total, 7 up, 7 in 2024-03-26T03:13:58.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:58.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[37920]: osdmap e325: 8 total, 7 up, 7 in 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:13:58.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:58 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:13:58.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:58.945 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:58.991 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:58.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:59.092 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:59.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:59.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 66 16 105234 105218 99.6232 51.875 0.00930926 0.0100325 2024-03-26T03:13:59.238 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:59.275 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:59.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:13:59.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:13:59.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:13:59.539 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:13:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:13:59.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:13:59.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:13:59.702 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:13:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:59 smithi192 ceph-mon[33110]: pgmap v678: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 50 MiB/s wr, 802 op/s; 28964/303996 objects degraded (9.528%); 50339/303996 objects misplaced (16.559%); 37 MiB/s, 585 objects/s recovering 2024-03-26T03:13:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:59 smithi192 ceph-mon[33110]: osdmap e326: 8 total, 7 up, 7 in 2024-03-26T03:13:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1443841883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:59 smithi192 ceph-mon[33110]: from='client.35615 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:13:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 28964/303996 objects degraded (9.528%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[29666]: pgmap v678: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 50 MiB/s wr, 802 op/s; 28964/303996 objects degraded (9.528%); 50339/303996 objects misplaced (16.559%); 37 MiB/s, 585 objects/s recovering 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[29666]: osdmap e326: 8 total, 7 up, 7 in 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1443841883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[29666]: from='client.35615 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 28964/303996 objects degraded (9.528%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[37920]: pgmap v678: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 50 MiB/s wr, 802 op/s; 28964/303996 objects degraded (9.528%); 50339/303996 objects misplaced (16.559%); 37 MiB/s, 585 objects/s recovering 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[37920]: osdmap e326: 8 total, 7 up, 7 in 2024-03-26T03:13:59.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1443841883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:59.741 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[37920]: from='client.35615 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:13:59.741 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:13:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 28964/303996 objects degraded (9.528%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-03-26T03:13:59.804 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:59.835 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:13:59.936 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:13:59.976 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:13:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:13:59.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:00.078 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.128 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:00.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:00.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 67 16 106063 106047 98.9096 51.8125 0.0109226 0.0101057 2024-03-26T03:14:00.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:00.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:13:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:14:00.272 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:00.374 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:00.412 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:00.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:00.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:00.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:00.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:00 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:00 smithi192 ceph-mon[33110]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:00 smithi192 ceph-mon[33110]: osdmap e327: 8 total, 7 up, 7 in 2024-03-26T03:14:00.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:00.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:00 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:00 smithi066 ceph-mon[29666]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:00 smithi066 ceph-mon[29666]: osdmap e327: 8 total, 7 up, 7 in 2024-03-26T03:14:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:00 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:00 smithi066 ceph-mon[37920]: from='client.35615 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:00 smithi066 ceph-mon[37920]: osdmap e327: 8 total, 7 up, 7 in 2024-03-26T03:14:00.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:00.974 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:00.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:01.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:01.374 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 68 16 107063 107047 98.3741 62.5 0.00591655 0.0101603 2024-03-26T03:14:01.413 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:01.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:01.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:01.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:01.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:01.657 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:01.698 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:01.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:01 smithi192 ceph-mon[33110]: pgmap v681: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 4.5 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 47 MiB/s, 752 objects/s recovering 2024-03-26T03:14:01.800 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:01.845 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:01.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:01 smithi066 ceph-mon[29666]: pgmap v681: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 4.5 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 47 MiB/s, 752 objects/s recovering 2024-03-26T03:14:01.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:01 smithi066 ceph-mon[37920]: pgmap v681: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 4.5 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 47 MiB/s, 752 objects/s recovering 2024-03-26T03:14:01.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:01.994 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:01.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:02.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:02.100 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-26T03:14:02.100 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:14:02.100 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-26T03:14:02.100 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-03-26T03:14:02.101 DEBUG:teuthology.orchestra.run.smithi066:osd.3> sudo journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T03:14:02.103 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl start ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3 2024-03-26T03:14:02.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:02.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 69 15 108073 108058 97.8641 63.1875 0.0482405 0.0102114 2024-03-26T03:14:02.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:02.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:02.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:02.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:02.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:02.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:02.608 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:02 smithi066 systemd[1]: Starting Ceph osd.3 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:14:02.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:02.677 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:02.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:02.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:02.919 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:02.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:02.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:03.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:03.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:03.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:03.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 70 16 108986 108970 97.2803 57 0.00287493 0.0102727 2024-03-26T03:14:03.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:03.240 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:03.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:03.342 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:03.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:03.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:03.487 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:03.533 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:03.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:03.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:03 smithi192 ceph-mon[33110]: pgmap v682: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 3.1 KiB/s rd, 47 MiB/s wr, 750 op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 32 MiB/s, 511 objects/s recovering 2024-03-26T03:14:03.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:03.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:03.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:03 smithi066 ceph-mon[29666]: pgmap v682: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 3.1 KiB/s rd, 47 MiB/s wr, 750 op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 32 MiB/s, 511 objects/s recovering 2024-03-26T03:14:03.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:03 smithi066 ceph-mon[37920]: pgmap v682: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 3.1 KiB/s rd, 47 MiB/s wr, 750 op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 32 MiB/s, 511 objects/s recovering 2024-03-26T03:14:03.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:03.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:03.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:04.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:04.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:04.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:14:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:14:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:14:04.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 71 16 109821 109805 96.6452 52.1875 0.00890409 0.0103415 2024-03-26T03:14:04.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:04.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:04.326 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:04.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:04.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:04.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:04.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:04.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 29706/317181 objects degraded (9.366%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:04 smithi192 ceph-mon[33110]: osdmap e328: 8 total, 7 up, 7 in 2024-03-26T03:14:04.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:04.767 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:04.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 29706/317181 objects degraded (9.366%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:04.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:04 smithi066 ceph-mon[37920]: osdmap e328: 8 total, 7 up, 7 in 2024-03-26T03:14:04.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 29706/317181 objects degraded (9.366%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:04.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:04 smithi066 ceph-mon[29666]: osdmap e328: 8 total, 7 up, 7 in 2024-03-26T03:14:04.870 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:04.916 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:04.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:05.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:05.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:05.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:05.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:05.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 72 16 110652 110636 96.0242 51.9375 0.0122722 0.0104093 2024-03-26T03:14:05.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:05.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:05.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:05.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:05.349 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:05.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:05.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:05.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:05.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:05.685 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:05.715 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:05.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:05 smithi192 ceph-mon[33110]: pgmap v684: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 3.0 KiB/s rd, 46 MiB/s wr, 735 op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 31 MiB/s, 501 objects/s recovering 2024-03-26T03:14:05.816 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:05.840 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:05 smithi066 ceph-mon[29666]: pgmap v684: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 3.0 KiB/s rd, 46 MiB/s wr, 735 op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 31 MiB/s, 501 objects/s recovering 2024-03-26T03:14:05.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:05 smithi066 ceph-mon[37920]: pgmap v684: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 3.0 KiB/s rd, 46 MiB/s wr, 735 op/s; 29706/317181 objects degraded (9.366%); 49729/317181 objects misplaced (15.678%); 31 MiB/s, 501 objects/s recovering 2024-03-26T03:14:05.859 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:05.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:05.961 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.002 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:06.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:06.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:06.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 73 16 111208 111192 95.1848 34.75 0.015174 0.0104975 2024-03-26T03:14:06.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:06.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:06.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:06.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:06.413 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:06.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:06.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:06.560 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:06.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:06.703 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:06.805 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:06.848 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:06.950 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:06.992 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:06.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:07.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:07.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 74 16 112134 112118 94.6806 57.875 0.0158564 0.0105567 2024-03-26T03:14:07.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:07.244 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:07.278 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:07.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:07.420 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:07.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.576 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:07.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:07.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:07.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:07 smithi066 ceph-mon[29666]: pgmap v685: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 2.9 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 30432/331191 objects degraded (9.189%); 49165/331191 objects misplaced (14.845%); 49 MiB/s, 776 objects/s recovering 2024-03-26T03:14:07.716 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:07 smithi066 ceph-mon[37920]: pgmap v685: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 2.9 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 30432/331191 objects degraded (9.189%); 49165/331191 objects misplaced (14.845%); 49 MiB/s, 776 objects/s recovering 2024-03-26T03:14:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:07.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:07 smithi192 ceph-mon[33110]: pgmap v685: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 2.9 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 30432/331191 objects degraded (9.189%); 49165/331191 objects misplaced (14.845%); 49 MiB/s, 776 objects/s recovering 2024-03-26T03:14:07.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:07.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:07.896 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:14:07.916 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:14:07.939 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:14:07.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:07.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:07.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:07.993 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:07 smithi066 podman[155977]: 2024-03-26 03:14:07.841832153 +0000 UTC m=+0.112480206 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:14:08.012 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:14:08.053 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:14:08.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:08.094 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:14:08.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:08.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 75 16 112967 112951 94.1124 52.0625 0.016234 0.0106206 2024-03-26T03:14:08.143 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:14:08.143 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:14:08.143 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:14:08.143 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:14:08.143 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:14:08.144 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:14:08.144 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:14:08.144 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:14:08.144 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:14:08.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:08.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:08.360 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:07 smithi066 podman[155977]: 2024-03-26 03:14:07.991928678 +0000 UTC m=+0.262576766 container create 2768c0e0164ef72d66260e373eb2049deb4d4a531acf7297a3f3c0a8d4d99a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T03:14:08.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:08.404 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:08.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:08.512 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:14:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.554 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:08.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:08.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:08 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1996363392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:08.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:08.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:08.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:08 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1996363392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:08.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:08 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1996363392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:08.859 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:08 smithi066 podman[155977]: 2024-03-26 03:14:08.504910827 +0000 UTC m=+0.775558880 container init 2768c0e0164ef72d66260e373eb2049deb4d4a531acf7297a3f3c0a8d4d99a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T03:14:08.859 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:08 smithi066 podman[155977]: 2024-03-26 03:14:08.514661936 +0000 UTC m=+0.785309981 container start 2768c0e0164ef72d66260e373eb2049deb4d4a531acf7297a3f3c0a8d4d99a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T03:14:08.859 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:08 smithi066 podman[155977]: 2024-03-26 03:14:08.629972602 +0000 UTC m=+0.900620650 container attach 2768c0e0164ef72d66260e373eb2049deb4d4a531acf7297a3f3c0a8d4d99a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T03:14:08.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:08.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:08.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:09.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:09.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:09.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:09.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 76 16 113895 113879 93.6372 58 0.00827222 0.0106748 2024-03-26T03:14:09.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:09.250 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:09.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:09.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:09.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:09.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:09.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:09.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:09.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:09.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:09.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:09.685 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:09 smithi066 ceph-mon[29666]: pgmap v686: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 2.6 KiB/s rd, 45 MiB/s wr, 716 op/s; 30432/331191 objects degraded (9.189%); 49165/331191 objects misplaced (14.845%); 28 MiB/s, 453 objects/s recovering 2024-03-26T03:14:09.685 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 30432/331191 objects degraded (9.189%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:09.686 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:09 smithi066 ceph-mon[37920]: pgmap v686: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 2.6 KiB/s rd, 45 MiB/s wr, 716 op/s; 30432/331191 objects degraded (9.189%); 49165/331191 objects misplaced (14.845%); 28 MiB/s, 453 objects/s recovering 2024-03-26T03:14:09.686 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 30432/331191 objects degraded (9.189%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:09.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:09 smithi192 ceph-mon[33110]: pgmap v686: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 6.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 2.6 KiB/s rd, 45 MiB/s wr, 716 op/s; 30432/331191 objects degraded (9.189%); 49165/331191 objects misplaced (14.845%); 28 MiB/s, 453 objects/s recovering 2024-03-26T03:14:09.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 30432/331191 objects degraded (9.189%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:14:09.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:14:09.884 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:09.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:09.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:09.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:09.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:14:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:14:10.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:10.081 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:10.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 77 16 114804 114788 93.1589 56.8125 0.0152555 0.0107298 2024-03-26T03:14:10.184 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.231 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:10.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:10.332 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.379 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:10.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:10.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:10.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:10.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:10.609 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T03:14:10.609 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T03:14:10.609 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T03:14:10.609 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T03:14:10.609 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 bash[155977]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T03:14:10.610 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate[156043]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T03:14:10.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:10.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:10.769 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.812 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:10.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:10.912 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:10.956 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:10.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:10.958 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:14:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:14:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:115022 10.1 10:8ebfb77e:::benchmark_data_smithi066_148944_object115021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-26T03:14:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.385919+0000", 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0067695100000000003, 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0068154219999999998, 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 115022 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.385919+0000", 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.385919+0000", 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.385921+0000", 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7889999999999999e-06 2024-03-26T03:14:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.385991+0000", 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0156999999999999e-05 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.386000+0000", 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7289999999999995e-06 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.386010+0000", 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0710000000000001e-06 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.386955+0000", 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00094566500000000005 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.387101+0000", 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000146219 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,7", 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.387573+0000", 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047136999999999998 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392718+0000", 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0051451919999999998 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(10.1 328/314 [PushReplyOp(10:810af931:::benchmark_data_smithi066_148944_object71755:head)])", 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.388179+0000", 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0045095090000000001, 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046615090000000003, 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.388179+0000", 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.388179+0000", 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.388179+0000", 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.82e-07 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.388183+0000", 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4999999999999999e-06 2024-03-26T03:14:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.388189+0000", 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3459999999999999e-06 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.388197+0000", 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4070000000000004e-06 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392761+0000", 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045645110000000003 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:115026 10.1 10:805bad1e:::benchmark_data_smithi066_148944_object115025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.389232+0000", 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0034562, 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036702890000000002, 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 115026 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389232+0000", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389232+0000", 2024-03-26T03:14:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389233+0000", 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2979999999999999e-06 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389285+0000", 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1952e-05 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389297+0000", 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1545999999999999e-05 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389303+0000", 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6550000000000001e-06 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390044+0000", 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00074158199999999996 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:14:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390196+0000", 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000151581 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,7", 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390392+0000", 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019656900000000001 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:115024 10.b e328/314 10:d411da9a:::benchmark_data_smithi066_148944_object115023:head v 328'7296, mlcod=328'7294)", 2024-03-26T03:14:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.389585+0000", 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0031030770000000001, 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033994699999999999, 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389585+0000", 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389585+0000", 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389586+0000", 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2240000000000001e-06 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389651+0000", 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4138000000000002e-05 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389681+0000", 2024-03-26T03:14:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0445000000000001e-05 2024-03-26T03:14:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389690+0000", 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.456e-06 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389774+0000", 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4734999999999994e-05 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389796+0000", 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2072000000000001e-05 2024-03-26T03:14:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392932+0000", 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003136036 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:115027 10.4 10:2e5212ec:::benchmark_data_smithi066_148944_object115026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.389983+0000", 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0027053060000000002, 2024-03-26T03:14:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030782499999999998, 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 115027 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389983+0000", 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389983+0000", 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389984+0000", 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3009999999999999e-06 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390013+0000", 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9051999999999999e-05 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390023+0000", 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4830000000000005e-06 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390030+0000", 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1890000000000001e-06 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390073+0000", 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3414000000000001e-05 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390211+0000", 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013776199999999999 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390392+0000", 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000181063 2024-03-26T03:14:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:115028 10.6 10:66fddb1d:::benchmark_data_smithi066_148944_object115027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.390593+0000", 2024-03-26T03:14:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002095684, 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025531659999999999, 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 115028 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390593+0000", 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390593+0000", 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390594+0000", 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3379999999999999e-06 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390625+0000", 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1325000000000002e-05 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390637+0000", 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1545e-05 2024-03-26T03:14:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390645+0000", 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3240000000000003e-06 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.391174+0000", 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052877600000000001 2024-03-26T03:14:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.391336+0000", 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000161941 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,5", 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.391527+0000", 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000190655 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25522.0:115016 10.1 e328/314)", 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.390901+0000", 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0017868859999999999, 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023126499999999999, 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390901+0000", 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390901+0000", 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390902+0000", 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5000000000000001e-07 2024-03-26T03:14:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390906+0000", 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-06 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390918+0000", 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.19e-05 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.390924+0000", 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1940000000000003e-06 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(10.6 328/314 [PushReplyOp(10:6a638d86:::benchmark_data_smithi066_148944_object107258:head)])", 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.392614+0000", 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.4697000000000007e-05, 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064916599999999996, 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:14:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392614+0000", 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392614+0000", 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392614+0000", 2024-03-26T03:14:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392617+0000", 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4690000000000002e-06 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392623+0000", 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8300000000000001e-06 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392631+0000", 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.43e-06 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25522.0:115022 10.1 e328/314)", 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.392682+0000", 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.3500000000000002e-06, 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00062926699999999998, 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392682+0000", 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392682+0000", 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392683+0000", 2024-03-26T03:14:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4300000000000003e-07 2024-03-26T03:14:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392687+0000", 2024-03-26T03:14:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6179999999999997e-06 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392693+0000", 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.212e-06 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.392807+0000", 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113553 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25522.0:115026 10.1 e328/314)", 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.393157+0000", 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9995313, 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000199655, 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393157+0000", 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393157+0000", 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393158+0000", 2024-03-26T03:14:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1500000000000003e-07 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393162+0000", 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8510000000000001e-06 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393170+0000", 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9249999999999995e-06 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393176+0000", 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4869999999999999e-06 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:115016 10.1 10:8e87e077:::benchmark_data_smithi066_148944_object115015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:10.376988+0000", 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015700298000000001, 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016421301999999999, 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:14:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 115016 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.376988+0000", 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.376988+0000", 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.376990+0000", 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7400000000000001e-06 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.377023+0000", 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3529999999999999e-05 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.377034+0000", 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0564000000000001e-05 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.377041+0000", 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1629999999999999e-06 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.379416+0000", 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023749489999999999 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.379551+0000", 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134702 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,7", 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.379713+0000", 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016244100000000001 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:14:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.385015+0000", 2024-03-26T03:14:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0053020630000000001 2024-03-26T03:14:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.385065+0000", 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0158999999999999e-05 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.389973+0000", 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004908005 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393276+0000", 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033022699999999999 2024-03-26T03:14:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:10.393336+0000", 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0463e-05 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 11 2024-03-26T03:14:11.008 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:14:11.009 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:11.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:11.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:11.137 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 78 16 115670 115654 92.6585 54.125 0.0211616 0.0107875 2024-03-26T03:14:11.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:11.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:11.275 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:10 smithi066 podman[155977]: 2024-03-26 03:14:10.990299214 +0000 UTC m=+3.260947266 container died 2768c0e0164ef72d66260e373eb2049deb4d4a531acf7297a3f3c0a8d4d99a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T03:14:11.337 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:11.379 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:11.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:11.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.524 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:11.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:11 smithi066 ceph-mon[29666]: pgmap v687: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 877 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:14:11.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:11 smithi066 ceph-mon[29666]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:14:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:11 smithi066 ceph-mon[37920]: pgmap v687: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 877 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:14:11.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:11 smithi066 ceph-mon[37920]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:14:11.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:11.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:11.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:11.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:11.813 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:11.813 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:11 smithi192 ceph-mon[33110]: pgmap v687: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 877 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:14:11.813 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:11 smithi192 ceph-mon[33110]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:14:11.884 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:14:11.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:11.985 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:11.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:12.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:12.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:12.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:12.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 79 16 116600 116584 92.2213 58.125 0.0118464 0.0108367 2024-03-26T03:14:12.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:12.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:12.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:12.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:12.422 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:12.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:14:12.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:12.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:12.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:12.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:12.576 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:12 smithi066 podman[156363]: 2024-03-26 03:14:12.16435838 +0000 UTC m=+1.792531248 container remove 2768c0e0164ef72d66260e373eb2049deb4d4a531acf7297a3f3c0a8d4d99a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T03:14:12.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:12.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:12.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:12.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:12.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:12.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:12.896 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:12.984 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:13.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:13.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:14:13.138139+0000 min lat: 0.00101075 max lat: 2.23537 avg lat: 0.0108915 2024-03-26T03:14:13.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:14:13.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 80 16 117504 117488 91.7747 56.5 0.0259637 0.0108915 2024-03-26T03:14:13.145 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.186 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:13.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:13.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:13.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:13.386 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:13.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:13.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:13.507 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:13 smithi066 podman[156613]: 2024-03-26 03:14:13.051050988 +0000 UTC m=+0.045633165 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:14:13.507 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:13 smithi066 podman[156613]: 2024-03-26 03:14:13.273682164 +0000 UTC m=+0.268264329 container create 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:14:13.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:13.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:13.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:13 smithi192 ceph-mon[33110]: pgmap v688: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 877 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:14:13.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:13.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:13 smithi066 ceph-mon[29666]: pgmap v688: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 877 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:14:13.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:13 smithi066 ceph-mon[37920]: pgmap v688: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 877 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:14:13.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:13.900 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:13.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:14.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:14.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:14.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:14.098 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:13 smithi066 podman[156613]: 2024-03-26 03:14:13.855776405 +0000 UTC m=+0.850358579 container init 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:14:14.099 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:13 smithi066 podman[156613]: 2024-03-26 03:14:13.86124662 +0000 UTC m=+0.855828784 container start 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:14:14.099 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:14:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:14:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:14:14.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:14:14.140 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 81 16 118435 118419 91.36 58.1875 0.0567718 0.0109407 2024-03-26T03:14:14.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:14.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:14.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:14.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:14.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:14.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:14.359 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:14 smithi066 bash[156613]: 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 2024-03-26T03:14:14.359 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:14 smithi066 systemd[1]: Started Ceph osd.3 for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:14:14.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:14.471 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:14.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:14.535 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:14:14.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:14.619 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:14.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:14.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:14.850 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:14 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4016052185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:14:14.850 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 30917/343476 objects degraded (9.001%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:14.850 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:14 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4016052185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:14:14.850 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 30917/343476 objects degraded (9.001%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:14.851 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:14 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:14:14.481+0000 7f24f3c337c0 -1 Falling back to public interface 2024-03-26T03:14:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:14.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:14.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:14.967 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:14 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4016052185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:14:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 30917/343476 objects degraded (9.001%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.009 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:15.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:15.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:15.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 82 16 119422 119406 90.9981 61.6875 0.0122397 0.0109837 2024-03-26T03:14:15.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:15.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:15.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:15.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:15.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:15.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:15.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:15.687 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.740 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:15.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:15.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[29666]: pgmap v689: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1004 B/s rd, 54 MiB/s wr, 861 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 681 objects/s recovering 2024-03-26T03:14:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4016052185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:14:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[29666]: osdmap e329: 8 total, 7 up, 7 in 2024-03-26T03:14:15.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[37920]: pgmap v689: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1004 B/s rd, 54 MiB/s wr, 861 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 681 objects/s recovering 2024-03-26T03:14:15.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:15.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4016052185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:14:15.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:15 smithi066 ceph-mon[37920]: osdmap e329: 8 total, 7 up, 7 in 2024-03-26T03:14:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:15.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:15.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:15.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:15.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:15 smithi192 ceph-mon[33110]: pgmap v689: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1004 B/s rd, 54 MiB/s wr, 861 op/s; 30917/343476 objects degraded (9.001%); 47539/343476 objects misplaced (13.841%); 43 MiB/s, 681 objects/s recovering 2024-03-26T03:14:15.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:15 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:15.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:15 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4016052185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:14:15.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:15 smithi192 ceph-mon[33110]: osdmap e329: 8 total, 7 up, 7 in 2024-03-26T03:14:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:16.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:16.029 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:16.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:16.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 83 16 120367 120351 90.6133 59.0625 0.00900732 0.0110309 2024-03-26T03:14:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:16.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:16.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:16.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:16.421 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.464 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:16.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:16.565 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.616 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:16.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:16.717 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.760 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:16.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:16.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:16.910 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:16.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:16.915 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:14:16.962 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:14:16.999 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:14:17.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:17.041 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:14:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:17.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:17.084 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:14:17.128 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:14:17.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 84 16 121396 121380 90.3001 64.3125 0.00775824 0.0110693 2024-03-26T03:14:17.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:17.156 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:14:17.157 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:14:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:17.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:17.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:17.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:17.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:17.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:17.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:17.633 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:17.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:17.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:17.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:17 smithi066 ceph-mon[29666]: pgmap v691: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 872 op/s; 31517/357351 objects degraded (8.820%); 45882/357351 objects misplaced (12.839%); 49 MiB/s, 778 objects/s recovering 2024-03-26T03:14:17.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/697692852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:17 smithi066 ceph-mon[37920]: pgmap v691: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 872 op/s; 31517/357351 objects degraded (8.820%); 45882/357351 objects misplaced (12.839%); 49 MiB/s, 778 objects/s recovering 2024-03-26T03:14:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/697692852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:17.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:17.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:17.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:17 smithi192 ceph-mon[33110]: pgmap v691: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 872 op/s; 31517/357351 objects degraded (8.820%); 45882/357351 objects misplaced (12.839%); 49 MiB/s, 778 objects/s recovering 2024-03-26T03:14:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/697692852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:17.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:14:18.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:18.036 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:18.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:18.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 85 16 122261 122245 89.8738 54.0625 0.00821112 0.0111216 2024-03-26T03:14:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.210 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:18.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:18.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:18.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:18.457 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.506 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:18.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:18.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:18.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:18.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:18.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:18.858 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:18 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:14:18.542+0000 7f24f3c337c0 -1 osd.3 197 log_to_monitors true 2024-03-26T03:14:18.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:18.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:18.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:18.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:19.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:19.149 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 86 16 123094 123078 89.4341 52.0625 0.0344545 0.0111737 2024-03-26T03:14:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:19.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:19.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:19.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:19.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:19.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:19.516 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:19.562 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:19.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:19.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:19.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:19.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:19.819 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: pgmap v692: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 872 op/s; 31517/357351 objects degraded (8.820%); 45882/357351 objects misplaced (12.839%); 49 MiB/s, 778 objects/s recovering 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: from='osd.3 v1:172.21.15.66:6813/52636497' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 31517/357351 objects degraded (8.820%), 20 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: osdmap e330: 8 total, 7 up, 7 in 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: from='osd.3 v1:172.21.15.66:6813/52636497' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T03:14:19.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-mon[33110]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T03:14:19.858 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:14:19.715+0000 7f24e508b700 -1 osd.3 330 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: pgmap v692: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 872 op/s; 31517/357351 objects degraded (8.820%); 45882/357351 objects misplaced (12.839%); 49 MiB/s, 778 objects/s recovering 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: from='osd.3 v1:172.21.15.66:6813/52636497' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 31517/357351 objects degraded (8.820%), 20 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: osdmap e330: 8 total, 7 up, 7 in 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: from='osd.3 v1:172.21.15.66:6813/52636497' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[29666]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T03:14:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: pgmap v692: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 33 active+clean+remapped; 7.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 54 MiB/s wr, 872 op/s; 31517/357351 objects degraded (8.820%); 45882/357351 objects misplaced (12.839%); 49 MiB/s, 778 objects/s recovering 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: from='osd.3 v1:172.21.15.66:6813/52636497' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 31517/357351 objects degraded (8.820%), 20 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: osdmap e330: 8 total, 7 up, 7 in 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: from='osd.3 v1:172.21.15.66:6813/52636497' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T03:14:19.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:19 smithi066 ceph-mon[37920]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi066", "root=default"]}]: dispatch 2024-03-26T03:14:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:19.861 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:19.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:19.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:19.977 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:14:19.977 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:14:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:20.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:20.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:20.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 87 16 124002 123986 89.0583 56.75 0.0130313 0.0112234 2024-03-26T03:14:20.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:20.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:14:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:14:20.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:20.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:20.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:20.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:20.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:20.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:20.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.721 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:20.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:20 smithi192 ceph-mon[33110]: 6.11 scrub starts 2024-03-26T03:14:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:20 smithi192 ceph-mon[33110]: 6.11 scrub ok 2024-03-26T03:14:20.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:14:20.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:20.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:20 smithi066 ceph-mon[29666]: 6.11 scrub starts 2024-03-26T03:14:20.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:20 smithi066 ceph-mon[29666]: 6.11 scrub ok 2024-03-26T03:14:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:20 smithi066 ceph-mon[37920]: 6.11 scrub starts 2024-03-26T03:14:20.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:20 smithi066 ceph-mon[37920]: 6.11 scrub ok 2024-03-26T03:14:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:20.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:20.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:20.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:21.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:21.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:21.138 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 88 16 124429 124413 88.3496 26.6875 0.00871676 0.011238 2024-03-26T03:14:21.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:21.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:21.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:21.379 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:21.409 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:21.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:21.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:21.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:21.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:14:21.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:14:21.784 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:21.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:21.817 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: pgmap v694: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 34 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 32190/371247 objects degraded (8.671%); 44382/371247 objects misplaced (11.955%); 59 MiB/s, 939 objects/s recovering 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: osd.3 v1:172.21.15.66:6813/52636497 boot 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: osdmap e331: 8 total, 8 up, 7 in 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[29666]: 10.9 scrub starts 2024-03-26T03:14:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:21.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: pgmap v694: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 34 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 32190/371247 objects degraded (8.671%); 44382/371247 objects misplaced (11.955%); 59 MiB/s, 939 objects/s recovering 2024-03-26T03:14:21.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T03:14:21.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: osd.3 v1:172.21.15.66:6813/52636497 boot 2024-03-26T03:14:21.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: osdmap e331: 8 total, 8 up, 7 in 2024-03-26T03:14:21.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:14:21.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:21 smithi066 ceph-mon[37920]: 10.9 scrub starts 2024-03-26T03:14:21.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:21.949 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:21.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: pgmap v694: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 34 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 32190/371247 objects degraded (8.671%); 44382/371247 objects misplaced (11.955%); 59 MiB/s, 939 objects/s recovering 2024-03-26T03:14:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T03:14:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: osd.3 v1:172.21.15.66:6813/52636497 boot 2024-03-26T03:14:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: osdmap e331: 8 total, 8 up, 7 in 2024-03-26T03:14:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:14:21.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:21 smithi192 ceph-mon[33110]: 10.9 scrub starts 2024-03-26T03:14:22.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:22.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:22.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 89 16 124441 124425 87.3654 0.75 0.00559339 0.0112738 2024-03-26T03:14:22.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:22.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:22.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:22.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:22.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:22.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:22.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:22.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:22.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:22.844 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:22.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7792 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 330'7642 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: osdmap e332: 8 total, 8 up, 7 in 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 330'7766 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 10.4 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7659 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 1.9 scrub starts 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 1.9 scrub ok 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: 10.9 scrub ok 2024-03-26T03:14:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[29666]: pgmap v697: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 34 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 48 MiB/s wr, 772 op/s; 32190/371247 objects degraded (8.671%); 44382/371247 objects misplaced (11.955%); 37 MiB/s, 597 objects/s recovering 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7792 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 330'7642 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: osdmap e332: 8 total, 8 up, 7 in 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 330'7766 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 10.4 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7659 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 1.9 scrub starts 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 1.9 scrub ok 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: 10.9 scrub ok 2024-03-26T03:14:22.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:22 smithi066 ceph-mon[37920]: pgmap v697: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 34 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 48 MiB/s wr, 772 op/s; 32190/371247 objects degraded (8.671%); 44382/371247 objects misplaced (11.955%); 37 MiB/s, 597 objects/s recovering 2024-03-26T03:14:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:22.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:22.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:22.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:14:22.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:14:22.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7792 2024-03-26T03:14:22.907 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 330'7642 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: osdmap e332: 8 total, 8 up, 7 in 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 330'7766 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 10.4 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7659 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 1.9 scrub starts 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 1.9 scrub ok 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: 10.9 scrub ok 2024-03-26T03:14:22.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:22 smithi192 ceph-mon[33110]: pgmap v697: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 91 active+clean, 29 active+undersized, 1 active+recovering+undersized+degraded+remapped, 18 active+undersized+degraded, 34 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 48 MiB/s wr, 772 op/s; 32190/371247 objects degraded (8.671%); 44382/371247 objects misplaced (11.955%); 37 MiB/s, 597 objects/s recovering 2024-03-26T03:14:22.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:22.992 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:23.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:23.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:23.141 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 90 13 124644 124631 86.5377 12.875 0.00750806 0.0115351 2024-03-26T03:14:23.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:23.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:23.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:23.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:23.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:23.506 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:23.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:23.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:23.674 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:23.702 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:14:23.776 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:23.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[29666]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7929 2024-03-26T03:14:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[29666]: 10.5 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7811 2024-03-26T03:14:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[29666]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 330'7833 2024-03-26T03:14:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[29666]: osdmap e333: 8 total, 8 up, 7 in 2024-03-26T03:14:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[29666]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7629 2024-03-26T03:14:23.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[37920]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7929 2024-03-26T03:14:23.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[37920]: 10.5 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7811 2024-03-26T03:14:23.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[37920]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 330'7833 2024-03-26T03:14:23.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[37920]: osdmap e333: 8 total, 8 up, 7 in 2024-03-26T03:14:23.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-mon[37920]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7629 2024-03-26T03:14:23.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:23.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:23.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:23.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:23 smithi192 ceph-mon[33110]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7929 2024-03-26T03:14:23.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:23 smithi192 ceph-mon[33110]: 10.5 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7811 2024-03-26T03:14:23.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:23 smithi192 ceph-mon[33110]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 330'7833 2024-03-26T03:14:23.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:23 smithi192 ceph-mon[33110]: osdmap e333: 8 total, 8 up, 7 in 2024-03-26T03:14:23.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:23 smithi192 ceph-mon[33110]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 330'7629 2024-03-26T03:14:24.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:24.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:14:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:14:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total time run: 90.4728 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total writes made: 124644 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Write size: 65536 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object size: 65536 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Bandwidth (MB/sec): 86.106 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Bandwidth: 46.9504 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max bandwidth (MB/sec): 234.875 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min bandwidth (MB/sec): 0 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average IOPS: 1377 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev IOPS: 751.207 2024-03-26T03:14:24.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max IOPS: 3758 2024-03-26T03:14:24.140 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min IOPS: 0 2024-03-26T03:14:24.140 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average Latency(s): 0.0115975 2024-03-26T03:14:24.140 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Latency(s): 0.0358631 2024-03-26T03:14:24.140 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max latency(s): 2.28532 2024-03-26T03:14:24.140 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min latency(s): 0.00101075 2024-03-26T03:14:24.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:24.195 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:14:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:24.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:24.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.358 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:24.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:24.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:24.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:24.590 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.639 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:24.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:24.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:24.754 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:24 smithi066 ceph-mon[29666]: osdmap e334: 8 total, 8 up, 7 in 2024-03-26T03:14:24.754 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 32190/371247 objects degraded (8.671%), 20 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:24.754 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:24 smithi066 ceph-mon[29666]: pgmap v700: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 122 active+clean, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 22 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 32162/371250 objects degraded (8.663%); 44354/371250 objects misplaced (11.947%); 0 B/s, 3 objects/s recovering 2024-03-26T03:14:24.755 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:24 smithi066 ceph-mon[37920]: osdmap e334: 8 total, 8 up, 7 in 2024-03-26T03:14:24.755 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 32190/371247 objects degraded (8.671%), 20 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:24.755 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:24 smithi066 ceph-mon[37920]: pgmap v700: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 122 active+clean, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 22 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 32162/371250 objects degraded (8.663%); 44354/371250 objects misplaced (11.947%); 0 B/s, 3 objects/s recovering 2024-03-26T03:14:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:24.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:24.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:24.880 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:14:24.880 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:14:24.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.880 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:125071 10.f e335/332 10:f0b3f9dd:::benchmark_data_smithi066_148944_object424:head v 335'7867, mlcod=335'7865)", 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:24.872344+0000", 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0062617180000000003, 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0062964060000000001, 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872344+0000", 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:14:24.881 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872344+0000", 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872346+0000", 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5829999999999999e-06 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.882 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872353+0000", 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4560000000000003e-06 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872365+0000", 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1375999999999999e-05 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872375+0000", 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9210000000000004e-06 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872657+0000", 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000282322 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.872676+0000", 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9179999999999999e-05 2024-03-26T03:14:24.884 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:125073 10.f e335/332 10:f3c497d2:::benchmark_data_smithi066_148944_object426:head v 335'7868, mlcod=335'7865)", 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:24.876932+0000", 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0016740990000000001, 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018007330000000001, 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:14:24.885 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.876932+0000", 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.876932+0000", 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.876933+0000", 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.505e-06 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.876941+0000", 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9950000000000005e-06 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.876952+0000", 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1024e-05 2024-03-26T03:14:24.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.876960+0000", 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1409999999999999e-06 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.888 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877028+0000", 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8010999999999996e-05 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877047+0000", 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9802000000000001e-05 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.889 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:125079 10.5 e335/332)", 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:24.877245+0000", 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0013606250000000001, 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0015513370000000001, 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877245+0000", 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:14:24.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877245+0000", 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877246+0000", 2024-03-26T03:14:24.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0389999999999999e-06 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877253+0000", 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.99e-06 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877261+0000", 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7009999999999996e-06 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:14:24.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:24.877268+0000", 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5899999999999996e-06 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 3 2024-03-26T03:14:24.893 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:14:24.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:24.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:24 smithi192 ceph-mon[33110]: osdmap e334: 8 total, 8 up, 7 in 2024-03-26T03:14:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 32190/371247 objects degraded (8.671%), 20 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-03-26T03:14:24.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:24 smithi192 ceph-mon[33110]: pgmap v700: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfilling, 122 active+clean, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 10 active+undersized+degraded, 22 active+clean+remapped; 7.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 32162/371250 objects degraded (8.663%); 44354/371250 objects misplaced (11.947%); 0 B/s, 3 objects/s recovering 2024-03-26T03:14:25.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:25.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:25.160 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:25.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:25.312 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.363 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:25.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:25.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:25.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:25.601 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.646 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:25.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:25.747 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.790 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:25.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:25.839 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: osdmap e335: 8 total, 8 up, 7 in 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:14:25.840 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:25 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:14:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: osdmap e335: 8 total, 8 up, 7 in 2024-03-26T03:14:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: osdmap e335: 8 total, 8 up, 7 in 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:14:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:14:25.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:14:25.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:14:25.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:25 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:14:25.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:25.933 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:25.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:26.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:26.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:26.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:26.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:26.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:26.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:26.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:26.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:26.550 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:14:26.550 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:14:26.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:26.693 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:26.720 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:26 smithi066 ceph-mon[29666]: osdmap e336: 8 total, 8 up, 7 in 2024-03-26T03:14:26.720 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:26 smithi066 ceph-mon[29666]: pgmap v703: 178 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 167 active+clean, 1 active+remapped+backfill_wait; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 18 MiB/s wr, 336 op/s; 26996/373908 objects degraded (7.220%); 39811/373908 objects misplaced (10.647%); 99 MiB/s, 1.60k objects/s recovering 2024-03-26T03:14:26.720 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:26 smithi066 ceph-mon[37920]: osdmap e336: 8 total, 8 up, 7 in 2024-03-26T03:14:26.720 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:26 smithi066 ceph-mon[37920]: pgmap v703: 178 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 167 active+clean, 1 active+remapped+backfill_wait; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 18 MiB/s wr, 336 op/s; 26996/373908 objects degraded (7.220%); 39811/373908 objects misplaced (10.647%); 99 MiB/s, 1.60k objects/s recovering 2024-03-26T03:14:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:26 smithi192 ceph-mon[33110]: osdmap e336: 8 total, 8 up, 7 in 2024-03-26T03:14:26.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:26 smithi192 ceph-mon[33110]: pgmap v703: 178 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 167 active+clean, 1 active+remapped+backfill_wait; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 18 MiB/s wr, 336 op/s; 26996/373908 objects degraded (7.220%); 39811/373908 objects misplaced (10.647%); 99 MiB/s, 1.60k objects/s recovering 2024-03-26T03:14:26.750 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:26.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:26.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:26.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:26.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:26.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:27.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:27.133 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:27.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:27.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:27.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:14:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:27.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:27.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:27.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:27.424 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:27.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:27.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:27.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:27.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:27.673 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:27.701 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:14:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:27.728 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:27.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:27.831 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:27.875 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:27.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:27.978 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:27 smithi192 ceph-mon[33110]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:27 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:14:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:27 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:14:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/270202659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:27 smithi192 ceph-mon[33110]: from='client.35642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:28.021 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:28.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[29666]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:28.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:14:28.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:14:28.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/270202659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:28.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[29666]: from='client.35642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:28.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[37920]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:28.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:14:28.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:14:28.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/270202659' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:28.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:27 smithi066 ceph-mon[37920]: from='client.35642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:28.123 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.160 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:28.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:28.160 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:28.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:28.304 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:28.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:28.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:28.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:28.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:28.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:28.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:28.847 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:28.905 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:14:28.905 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[29666]: from='client.35642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:28.905 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[29666]: osdmap e337: 8 total, 8 up, 7 in 2024-03-26T03:14:28.905 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[29666]: pgmap v705: 178 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 167 active+clean, 1 active+remapped+backfill_wait; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 15 MiB/s wr, 282 op/s; 26996/373908 objects degraded (7.220%); 39811/373908 objects misplaced (10.647%); 83 MiB/s, 1.34k objects/s recovering 2024-03-26T03:14:28.906 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:14:28.906 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[37920]: from='client.35642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:28.906 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[37920]: osdmap e337: 8 total, 8 up, 7 in 2024-03-26T03:14:28.906 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:28 smithi066 ceph-mon[37920]: pgmap v705: 178 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 167 active+clean, 1 active+remapped+backfill_wait; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 15 MiB/s wr, 282 op/s; 26996/373908 objects degraded (7.220%); 39811/373908 objects misplaced (10.647%); 83 MiB/s, 1.34k objects/s recovering 2024-03-26T03:14:28.929 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:28.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:28.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:28 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:14:28.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:28 smithi192 ceph-mon[33110]: from='client.35642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:28.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:28 smithi192 ceph-mon[33110]: osdmap e337: 8 total, 8 up, 7 in 2024-03-26T03:14:28.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:28 smithi192 ceph-mon[33110]: pgmap v705: 178 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 167 active+clean, 1 active+remapped+backfill_wait; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 15 MiB/s wr, 282 op/s; 26996/373908 objects degraded (7.220%); 39811/373908 objects misplaced (10.647%); 83 MiB/s, 1.34k objects/s recovering 2024-03-26T03:14:29.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:29.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:29.187 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:29.229 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:29.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:29.373 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:29.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:29.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:29.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:29.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:29.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 26996/373908 objects degraded (7.220%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:29.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:29 smithi192 ceph-mon[33110]: osdmap e338: 8 total, 8 up, 7 in 2024-03-26T03:14:29.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:29.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 26996/373908 objects degraded (7.220%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:29 smithi066 ceph-mon[29666]: osdmap e338: 8 total, 8 up, 7 in 2024-03-26T03:14:29.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 26996/373908 objects degraded (7.220%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:29.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:29 smithi066 ceph-mon[37920]: osdmap e338: 8 total, 8 up, 7 in 2024-03-26T03:14:29.922 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:29.964 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:29.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:29.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:14:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:14:30.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:30.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:30.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:30.209 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:30.254 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:30.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:30.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:30.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:30.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:30.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:30.503 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:14:30.503 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:14:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:30.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:30.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:30.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:30.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:30.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:30.955 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:30.956 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:30 smithi192 ceph-mon[33110]: pgmap v707: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s rd, 8.8 MiB/s wr, 1.07k op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T03:14:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:30.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:31.057 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:31.057 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:30 smithi066 ceph-mon[29666]: pgmap v707: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s rd, 8.8 MiB/s wr, 1.07k op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T03:14:31.057 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:31.058 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:30 smithi066 ceph-mon[37920]: pgmap v707: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s rd, 8.8 MiB/s wr, 1.07k op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T03:14:31.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:31.119 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:31.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:31.211 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 7] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:14:31.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:14:31.212 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:14:31.212 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd out 7 2024-03-26T03:14:31.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:31.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:31.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:31.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:31.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:14:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:31.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:31.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:31.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:14:31.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:31.677 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:31.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:31.697 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:31 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:14:31.698 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:31 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:14:31.698 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:31 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:14:31.698 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:31 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:14:31.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:31.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:31.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:31.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:31.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:31.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:31 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:14:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:31 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:14:31.997 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:32.041 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:14:32.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:32.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:32.093 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:14:32.128 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:14:32.135 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:32.160 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:14:32.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:32.203 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:14:32.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:32.247 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:14:32.294 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:14:32.340 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:14:32.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:14:32.353 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:14:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:32.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:32.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:32.536 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:32.638 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:32.685 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:32.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[37920]: osdmap e339: 8 total, 8 up, 7 in 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2440850586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[37920]: pgmap v709: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 19 MiB/s, 307 objects/s recovering 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:14:32.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:14:32.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[29666]: osdmap e339: 8 total, 8 up, 7 in 2024-03-26T03:14:32.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2440850586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:32.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:32 smithi066 ceph-mon[29666]: pgmap v709: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 19 MiB/s, 307 objects/s recovering 2024-03-26T03:14:32.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:32.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:32.975 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:32.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:32 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:14:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:32 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:14:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:32 smithi192 ceph-mon[33110]: osdmap e339: 8 total, 8 up, 7 in 2024-03-26T03:14:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2440850586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:32.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:32 smithi192 ceph-mon[33110]: pgmap v709: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 19 MiB/s, 307 objects/s recovering 2024-03-26T03:14:33.034 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:33.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:33.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:33.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:33.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:33.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:33.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:33.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:33.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:33.656 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:33.714 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:33.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:33.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:33.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:33.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:33.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:34.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:14:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:14:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:14:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:34.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:34.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:34.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:34.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:34.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:34.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:34.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:34.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 24894/358332 objects degraded (6.947%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:34.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:34 smithi192 ceph-mon[33110]: osdmap e340: 8 total, 8 up, 7 in 2024-03-26T03:14:34.521 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 24894/358332 objects degraded (6.947%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:34.521 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:34 smithi066 ceph-mon[29666]: osdmap e340: 8 total, 8 up, 7 in 2024-03-26T03:14:34.522 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 24894/358332 objects degraded (6.947%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:34.522 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:34 smithi066 ceph-mon[37920]: osdmap e340: 8 total, 8 up, 7 in 2024-03-26T03:14:34.538 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:34.597 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:34.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:34.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:34.714 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:14:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:14:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:14:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:14:34.714 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:14:34.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:34.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:34.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:34.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:34.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:34.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:35.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:35.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:35.188 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:35.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:35.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:35.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:14:35.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:14:35.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-mon[37920]: pgmap v711: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 19 MiB/s, 307 objects/s recovering 2024-03-26T03:14:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:14:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:14:35.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-mon[29666]: pgmap v711: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 19 MiB/s, 307 objects/s recovering 2024-03-26T03:14:35.423 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:35.479 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:35 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:14:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:35 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:14:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:35 smithi192 ceph-mon[33110]: pgmap v711: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 7.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 24894/358332 objects degraded (6.947%); 37579/358332 objects misplaced (10.487%); 19 MiB/s, 307 objects/s recovering 2024-03-26T03:14:35.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:35.607 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:14:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:35.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:35.727 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:35.778 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:35.880 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:35.934 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:35.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:36.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:36.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:36.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:35 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29662]: 2024-03-26T03:14:35.885+0000 7f94a71dc700 -1 mon.a@0(leader).osd e340 definitely_dead 0 2024-03-26T03:14:36.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:36.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:36.228 INFO:teuthology.orchestra.run.smithi066.stderr:marked out osd.7. 2024-03-26T03:14:36.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:36.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:36.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.470 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:36.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:36 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2555800481' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T03:14:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:36 smithi192 ceph-mon[33110]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-26T03:14:36.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:36.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:36 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2555800481' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T03:14:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:36 smithi066 ceph-mon[29666]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-26T03:14:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:36 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2555800481' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T03:14:36.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:36 smithi066 ceph-mon[37920]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-26T03:14:36.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:36.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:36.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:36.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:36.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:36.985 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:36.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:37.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:37.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:37.217 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:37.267 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:37.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2555800481' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[37920]: osdmap e341: 8 total, 8 up, 6 in 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[37920]: 5.1c scrub starts 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[37920]: 5.1c scrub ok 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[37920]: pgmap v713: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 907 op/s; 22876/342009 objects degraded (6.689%); 35306/342009 objects misplaced (10.323%); 17 MiB/s, 279 objects/s recovering 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2555800481' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[29666]: osdmap e341: 8 total, 8 up, 6 in 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[29666]: 5.1c scrub starts 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[29666]: 5.1c scrub ok 2024-03-26T03:14:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:37 smithi066 ceph-mon[29666]: pgmap v713: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 907 op/s; 22876/342009 objects degraded (6.689%); 35306/342009 objects misplaced (10.323%); 17 MiB/s, 279 objects/s recovering 2024-03-26T03:14:37.370 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.401 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:37.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2555800481' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-26T03:14:37.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:37 smithi192 ceph-mon[33110]: osdmap e341: 8 total, 8 up, 6 in 2024-03-26T03:14:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:37 smithi192 ceph-mon[33110]: 5.1c scrub starts 2024-03-26T03:14:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:37 smithi192 ceph-mon[33110]: 5.1c scrub ok 2024-03-26T03:14:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:37 smithi192 ceph-mon[33110]: pgmap v713: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 907 op/s; 22876/342009 objects degraded (6.689%); 35306/342009 objects misplaced (10.323%); 17 MiB/s, 279 objects/s recovering 2024-03-26T03:14:37.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:37.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:37.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:37.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:37.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:37.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:37.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:37.922 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:37.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:38.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:38.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:38.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:38.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:38.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.315 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:38.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:38.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:14:38.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:38.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:14:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 10.1 continuing backfill to osd.1 from (312'4972,340'8707] 10:89fb4802:::benchmark_data_smithi066_148944_object117230:head to 340'8707 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 340'8707 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 10.5 continuing backfill to osd.3 from (310'4161,340'8587] MIN to 340'8587 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 340'8587 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: osdmap e342: 8 total, 8 up, 6 in 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 4.9 scrub starts 2024-03-26T03:14:38.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:38 smithi192 ceph-mon[33110]: 4.9 scrub ok 2024-03-26T03:14:38.522 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 10.1 continuing backfill to osd.1 from (312'4972,340'8707] 10:89fb4802:::benchmark_data_smithi066_148944_object117230:head to 340'8707 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 340'8707 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 10.5 continuing backfill to osd.3 from (310'4161,340'8587] MIN to 340'8587 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 340'8587 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: osdmap e342: 8 total, 8 up, 6 in 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 4.9 scrub starts 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[29666]: 4.9 scrub ok 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:14:38.523 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 10.1 continuing backfill to osd.1 from (312'4972,340'8707] 10:89fb4802:::benchmark_data_smithi066_148944_object117230:head to 340'8707 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 10.1 starting backfill to osd.2 from (0'0,0'0] MAX to 340'8707 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 10.5 continuing backfill to osd.3 from (310'4161,340'8587] MIN to 340'8587 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 10.5 starting backfill to osd.5 from (0'0,0'0] MAX to 340'8587 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: osdmap e342: 8 total, 8 up, 6 in 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 4.9 scrub starts 2024-03-26T03:14:38.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:38 smithi066 ceph-mon[37920]: 4.9 scrub ok 2024-03-26T03:14:38.554 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:38.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:38.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:38.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:38.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:38.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:38.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:38.981 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:38.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:39.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:39.128 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:39.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:39.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:39.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.425 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:39.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:39.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: osdmap e343: 8 total, 8 up, 6 in 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: 10.8 starting backfill to osd.4 from (0'0,0'0] MAX to 340'8564 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: 10.c starting backfill to osd.3 from (312'4667,340'8427] MIN to 340'8428 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 340'8541 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 340'8428 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: pgmap v716: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 22876/342009 objects degraded (6.689%); 35306/342009 objects misplaced (10.323%); 25 MiB/s, 400 objects/s recovering 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 22876/342009 objects degraded (6.689%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-mon[33110]: osdmap e344: 8 total, 8 up, 6 in 2024-03-26T03:14:39.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: osdmap e343: 8 total, 8 up, 6 in 2024-03-26T03:14:39.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: 10.8 starting backfill to osd.4 from (0'0,0'0] MAX to 340'8564 2024-03-26T03:14:39.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: 10.c starting backfill to osd.3 from (312'4667,340'8427] MIN to 340'8428 2024-03-26T03:14:39.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 340'8541 2024-03-26T03:14:39.508 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 340'8428 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: pgmap v716: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 22876/342009 objects degraded (6.689%); 35306/342009 objects misplaced (10.323%); 25 MiB/s, 400 objects/s recovering 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 22876/342009 objects degraded (6.689%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[37920]: osdmap e344: 8 total, 8 up, 6 in 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: osdmap e343: 8 total, 8 up, 6 in 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: 10.8 starting backfill to osd.4 from (0'0,0'0] MAX to 340'8564 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: 10.c starting backfill to osd.3 from (312'4667,340'8427] MIN to 340'8428 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 340'8541 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 340'8428 2024-03-26T03:14:39.509 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: pgmap v716: 178 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 6.9 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 22876/342009 objects degraded (6.689%); 35306/342009 objects misplaced (10.323%); 25 MiB/s, 400 objects/s recovering 2024-03-26T03:14:39.510 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 22876/342009 objects degraded (6.689%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:39.510 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:39 smithi066 ceph-mon[29666]: osdmap e344: 8 total, 8 up, 6 in 2024-03-26T03:14:39.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:39.590 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:39.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:39.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:39.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:39.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:39.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:39.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:39.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:40.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:40.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:40.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:40.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:14:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:14:40.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:40.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:40.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:40.480 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:40.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:40.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:40.624 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:40.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:40.768 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:40.803 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: 2.9 scrub starts 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: 2.9 scrub ok 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: osdmap e345: 8 total, 8 up, 6 in 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: pgmap v719: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 16 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 150 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 18 KiB/s rd, 132 KiB/s wr, 588 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 52 MiB/s, 855 objects/s recovering 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1080241383' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: 2.9 scrub starts 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: 2.9 scrub ok 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: osdmap e345: 8 total, 8 up, 6 in 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: pgmap v719: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 16 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 150 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 18 KiB/s rd, 132 KiB/s wr, 588 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 52 MiB/s, 855 objects/s recovering 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1080241383' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:40.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:40.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: 2.9 scrub starts 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: 2.9 scrub ok 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: osdmap e345: 8 total, 8 up, 6 in 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: pgmap v719: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 16 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 150 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 18 KiB/s rd, 132 KiB/s wr, 588 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 52 MiB/s, 855 objects/s recovering 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1080241383' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:14:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='client.25630 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:14:40.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:40.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:41.044 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:41.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:41.170 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:14:41.198 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:41.234 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:41.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.389 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:41.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:41.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:41.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:41.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:41.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:41.771 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:41.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[29666]: 1.15 scrub starts 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[29666]: 1.15 scrub ok 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[29666]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[29666]: osdmap e346: 8 total, 8 up, 6 in 2024-03-26T03:14:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[37920]: 1.15 scrub starts 2024-03-26T03:14:41.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[37920]: 1.15 scrub ok 2024-03-26T03:14:41.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:41.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:41.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[37920]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:14:41.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:41 smithi066 ceph-mon[37920]: osdmap e346: 8 total, 8 up, 6 in 2024-03-26T03:14:41.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:41.957 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:41.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:41.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:41 smithi192 ceph-mon[33110]: 1.15 scrub starts 2024-03-26T03:14:41.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:41 smithi192 ceph-mon[33110]: 1.15 scrub ok 2024-03-26T03:14:41.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:41 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:41.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:41 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:41.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:41 smithi192 ceph-mon[33110]: from='client.25630 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:14:41.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:41 smithi192 ceph-mon[33110]: osdmap e346: 8 total, 8 up, 6 in 2024-03-26T03:14:42.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:42.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:42.199 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:42.245 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:42.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:42.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:42.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:42.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:42.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:42.681 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:42.783 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:42.792 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:42 smithi066 ceph-mon[29666]: pgmap v721: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 16 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 150 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 17 KiB/s rd, 129 KiB/s wr, 572 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 51 MiB/s, 832 objects/s recovering 2024-03-26T03:14:42.793 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:42 smithi066 ceph-mon[37920]: pgmap v721: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 16 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 150 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 17 KiB/s rd, 129 KiB/s wr, 572 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 51 MiB/s, 832 objects/s recovering 2024-03-26T03:14:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.829 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:42.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:42.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:42.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:42.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:42 smithi192 ceph-mon[33110]: pgmap v721: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 16 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 150 active+clean; 6.8 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 17 KiB/s rd, 129 KiB/s wr, 572 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 51 MiB/s, 832 objects/s recovering 2024-03-26T03:14:43.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:43.117 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:43.162 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:14:43.186 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:14:43.208 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:14:43.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:43.247 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:14:43.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:43.290 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:14:43.314 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:14:43.315 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:14:43.315 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-26T03:14:43.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight-by-utilization 1000 1.0 2024-03-26T03:14:43.329 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:14:43.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:43.406 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:14:43.415 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:43.447 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:14:43.455 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:14:43.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:14:43.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:43.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:43.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:43.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:43.795 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:43.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:43.942 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:43.981 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:43.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:43.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:43 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1207982101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:44.052 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:44.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:44.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:43 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1207982101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:14:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:14:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:14:44.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:43 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1207982101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:44.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:44.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:44.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:44.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:44.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:44.415 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:44.459 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:44.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:44.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:44.563 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:14:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:44.605 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:44.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:44.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:44.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:44.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:44.954 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:45.009 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:45.112 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.148 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:45.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:45.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:45.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 20454/335211 objects degraded (6.102%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[29666]: pgmap v722: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 12 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 154 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 12 KiB/s rd, 88 KiB/s wr, 392 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 35 MiB/s, 570 objects/s recovering 2024-03-26T03:14:45.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:45.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:45.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 20454/335211 objects degraded (6.102%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:45.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:45 smithi066 ceph-mon[37920]: pgmap v722: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 12 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 154 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 12 KiB/s rd, 88 KiB/s wr, 392 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 35 MiB/s, 570 objects/s recovering 2024-03-26T03:14:45.399 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:45.442 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:45 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:45.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:45 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:45 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 20454/335211 objects degraded (6.102%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:45 smithi192 ceph-mon[33110]: pgmap v722: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 12 peering, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 154 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 12 KiB/s rd, 88 KiB/s wr, 392 op/s; 20454/335211 objects degraded (6.102%); 60810/335211 objects misplaced (18.141%); 35 MiB/s, 570 objects/s recovering 2024-03-26T03:14:45.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:45.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:45.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:45.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:45.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:45.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:45.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:45.972 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.020 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:46.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:46.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.169 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:46.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:46.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:46.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:46.418 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.451 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:46.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:46 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:14:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:46 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:14:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:46 smithi192 ceph-mon[33110]: osdmap e347: 8 total, 8 up, 6 in 2024-03-26T03:14:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:46 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:14:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:46 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:14:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:46 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:46.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:46.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:14:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:14:46.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[29666]: osdmap e347: 8 total, 8 up, 6 in 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[37920]: osdmap e347: 8 total, 8 up, 6 in 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:14:46.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:46 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:46.690 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:46.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:46.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:46.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:46.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:46.972 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:47.014 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:47.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:47.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:47.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:47.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:47.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:47.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:47.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:47 smithi192 ceph-mon[33110]: pgmap v724: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 18975/320649 objects degraded (5.918%); 56513/320649 objects misplaced (17.625%); 20 MiB/s, 315 objects/s recovering 2024-03-26T03:14:47.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:47.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:47 smithi066 ceph-mon[29666]: pgmap v724: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 18975/320649 objects degraded (5.918%); 56513/320649 objects misplaced (17.625%); 20 MiB/s, 315 objects/s recovering 2024-03-26T03:14:47.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:47 smithi066 ceph-mon[37920]: pgmap v724: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 18975/320649 objects degraded (5.918%); 56513/320649 objects misplaced (17.625%); 20 MiB/s, 315 objects/s recovering 2024-03-26T03:14:47.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:47.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:47.782 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.819 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:47.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:47.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:47.969 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:47.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:48.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:48.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:48.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:48.367 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:48.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:48.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:48.625 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:48.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:48.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:48.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:48.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:49.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:49.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:49.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.209 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:49.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:49.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.358 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:49.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:49.459 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.490 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:49.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:49.591 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:49.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:49.646 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:49 smithi066 ceph-mon[29666]: pgmap v725: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 18975/320649 objects degraded (5.918%); 56513/320649 objects misplaced (17.625%); 15 MiB/s, 244 objects/s recovering 2024-03-26T03:14:49.646 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 18975/320649 objects degraded (5.918%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:49.646 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:49 smithi066 ceph-mon[29666]: osdmap e348: 8 total, 8 up, 6 in 2024-03-26T03:14:49.646 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:49 smithi066 ceph-mon[37920]: pgmap v725: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 18975/320649 objects degraded (5.918%); 56513/320649 objects misplaced (17.625%); 15 MiB/s, 244 objects/s recovering 2024-03-26T03:14:49.646 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 18975/320649 objects degraded (5.918%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:49.647 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:49 smithi066 ceph-mon[37920]: osdmap e348: 8 total, 8 up, 6 in 2024-03-26T03:14:49.652 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:49 smithi192 ceph-mon[33110]: pgmap v725: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.5 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 606 op/s; 18975/320649 objects degraded (5.918%); 56513/320649 objects misplaced (17.625%); 15 MiB/s, 244 objects/s recovering 2024-03-26T03:14:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 18975/320649 objects degraded (5.918%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:49 smithi192 ceph-mon[33110]: osdmap e348: 8 total, 8 up, 6 in 2024-03-26T03:14:49.738 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:49.781 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:49.883 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:49.922 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:49.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:50.024 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:50.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:50.160 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:50.205 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:50.224 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:14:50.225 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:14:50.226 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:14:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:14:50.269 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:14:50.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:50.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:50.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:50.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:50.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:50.632 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:50.735 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.778 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:50.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:50.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:50.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:50.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:51.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:51.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:51.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:51.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:51.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:51.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:51.448 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:51.493 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:51.594 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:51.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:51 smithi066 ceph-mon[29666]: pgmap v727: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.22k op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 28 MiB/s, 449 objects/s recovering 2024-03-26T03:14:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:51 smithi066 ceph-mon[37920]: pgmap v727: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.22k op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 28 MiB/s, 449 objects/s recovering 2024-03-26T03:14:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.635 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:51.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:51.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:14:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:51 smithi192 ceph-mon[33110]: pgmap v727: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.22k op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 28 MiB/s, 449 objects/s recovering 2024-03-26T03:14:51.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:51.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:51.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:51.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:52.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:52.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:52.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.203 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:52.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:52.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:52.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:52.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:52.491 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:52.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:52.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:52.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:52.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:52.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:52.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:52.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:52.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:14:53.012 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:53.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:53.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:53.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:53.195 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:53.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:53.296 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:53.333 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:53.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:53.429 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:53.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:53.464 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:53.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:53.565 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:53.609 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:53.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:53.711 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:53.966 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:53 smithi066 ceph-mon[29666]: pgmap v728: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.22k op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 28 MiB/s, 449 objects/s recovering 2024-03-26T03:14:53.966 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:53 smithi066 ceph-mon[37920]: pgmap v728: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.22k op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 28 MiB/s, 449 objects/s recovering 2024-03-26T03:14:53.966 INFO:teuthology.orchestra.run.smithi066.stdout:moved 0 / 534 (0%) 2024-03-26T03:14:53.967 INFO:teuthology.orchestra.run.smithi066.stdout:avg 89 2024-03-26T03:14:53.967 INFO:teuthology.orchestra.run.smithi066.stdout:stddev 5.06623 -> 5.06623 (expected baseline 8.61201) 2024-03-26T03:14:53.967 INFO:teuthology.orchestra.run.smithi066.stdout:min osd.2 with 81 -> 81 pgs (0.910112 -> 0.910112 * mean) 2024-03-26T03:14:53.967 INFO:teuthology.orchestra.run.smithi066.stdout:max osd.5 with 95 -> 95 pgs (1.06742 -> 1.06742 * mean) 2024-03-26T03:14:53.967 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:14:53.968 INFO:teuthology.orchestra.run.smithi066.stdout:oload 1000 2024-03-26T03:14:53.968 INFO:teuthology.orchestra.run.smithi066.stdout:max_change 1 2024-03-26T03:14:53.968 INFO:teuthology.orchestra.run.smithi066.stdout:max_change_osds 4 2024-03-26T03:14:53.968 INFO:teuthology.orchestra.run.smithi066.stdout:average_utilization 0.0363 2024-03-26T03:14:53.968 INFO:teuthology.orchestra.run.smithi066.stdout:overload_utilization 0.3630 2024-03-26T03:14:53.968 INFO:teuthology.orchestra.run.smithi066.stderr:no change 2024-03-26T03:14:53.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:53 smithi192 ceph-mon[33110]: pgmap v728: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.22k op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 28 MiB/s, 449 objects/s recovering 2024-03-26T03:14:54.004 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:54.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:14:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:14:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:14:54.136 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:54.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:54.246 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:54.300 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:54.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:54.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:54.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:54.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:54.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[29666]: from='client.25639 v1:172.21.15.66:0/2022932103' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 17465/306030 objects degraded (5.707%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[37920]: from='client.25639 v1:172.21.15.66:0/2022932103' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 17465/306030 objects degraded (5.707%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:14:54.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:54 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:14:54.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:54.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:54 smithi192 ceph-mon[33110]: from='client.25639 v1:172.21.15.66:0/2022932103' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:14:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:54 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 17465/306030 objects degraded (5.707%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:54 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:14:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:54 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:14:54.749 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-26T03:14:54.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-26T03:14:54.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:54.869 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:54.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:54.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.037 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:55.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:55.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.195 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:55.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:55.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:55.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:55.442 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:55.486 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:55.577 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:55 smithi192 ceph-mon[33110]: pgmap v729: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 558 B/s rd, 0 B/s wr, 532 op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 11 MiB/s, 178 objects/s recovering 2024-03-26T03:14:55.577 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:55 smithi192 ceph-mon[33110]: osdmap e349: 8 total, 8 up, 6 in 2024-03-26T03:14:55.577 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:55.577 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:55.589 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:55.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[29666]: pgmap v729: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 558 B/s rd, 0 B/s wr, 532 op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 11 MiB/s, 178 objects/s recovering 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[29666]: osdmap e349: 8 total, 8 up, 6 in 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[37920]: pgmap v729: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 558 B/s rd, 0 B/s wr, 532 op/s; 17465/306030 objects degraded (5.707%); 52561/306030 objects misplaced (17.175%); 11 MiB/s, 178 objects/s recovering 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[37920]: osdmap e349: 8 total, 8 up, 6 in 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:14:55.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:14:55.627 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:55.694 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:14:55.694 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:14:55.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:55.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:55.873 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:55.914 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:55.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:56.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:56.057 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:56.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:56.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:56.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:56.336 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:56.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:14:56.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:56.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:56.582 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:56 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:56 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:56.618 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:56 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:14:56.622 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:56.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:56.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:56.877 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:14:56.914 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:14:56.927 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:56.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:14:57.029 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:57.058 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:14:57.092 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:57.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:57.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:57.352 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:57.406 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:57.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:57.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:57.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:57 smithi066 ceph-mon[29666]: pgmap v731: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 15956/290067 objects degraded (5.501%); 48189/290067 objects misplaced (16.613%); 29 MiB/s, 465 objects/s recovering 2024-03-26T03:14:57.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:57 smithi066 ceph-mon[37920]: pgmap v731: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 15956/290067 objects degraded (5.501%); 48189/290067 objects misplaced (16.613%); 29 MiB/s, 465 objects/s recovering 2024-03-26T03:14:57.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:57.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:57 smithi192 ceph-mon[33110]: pgmap v731: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 15956/290067 objects degraded (5.501%); 48189/290067 objects misplaced (16.613%); 29 MiB/s, 465 objects/s recovering 2024-03-26T03:14:57.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:14:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:57.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:14:57.910 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:14:57.930 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:57.953 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:14:57.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:57.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:57.997 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:14:58.043 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:14:58.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:58.089 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:14:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:58.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:58.131 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:14:58.175 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:14:58.219 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:14:58.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:58.223 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:14:58.223 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:14:58.223 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:14:58.223 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:14:58.224 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:14:58.224 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:14:58.224 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:14:58.224 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:14:58.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:58.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:58.413 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:58.499 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:14:58.515 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:58.545 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:58.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.692 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:58.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2026329726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:58 smithi192 ceph-mon[33110]: from='client.35675 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:58 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1148976668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:58.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:58.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2026329726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:58 smithi066 ceph-mon[29666]: from='client.35675 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:58 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1148976668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2026329726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:58 smithi066 ceph-mon[37920]: from='client.35675 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:14:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:58 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1148976668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:14:58.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:14:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:58.981 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:58.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:14:59.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:14:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:59.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:14:59.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:59.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:59.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:59.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:14:59.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:14:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:59.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:14:59.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[29666]: pgmap v732: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 15956/290067 objects degraded (5.501%); 48189/290067 objects misplaced (16.613%); 25 MiB/s, 405 objects/s recovering 2024-03-26T03:14:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[29666]: from='client.35675 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[29666]: osdmap e350: 8 total, 8 up, 6 in 2024-03-26T03:14:59.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 15956/290067 objects degraded (5.501%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[37920]: pgmap v732: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 15956/290067 objects degraded (5.501%); 48189/290067 objects misplaced (16.613%); 25 MiB/s, 405 objects/s recovering 2024-03-26T03:14:59.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:59.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[37920]: from='client.35675 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:59.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[37920]: osdmap e350: 8 total, 8 up, 6 in 2024-03-26T03:14:59.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:14:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 15956/290067 objects degraded (5.501%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:59.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:14:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.666 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:14:59.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:14:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:59 smithi192 ceph-mon[33110]: pgmap v732: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s; 15956/290067 objects degraded (5.501%); 48189/290067 objects misplaced (16.613%); 25 MiB/s, 405 objects/s recovering 2024-03-26T03:14:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:59 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:14:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:59 smithi192 ceph-mon[33110]: from='client.35675 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:14:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:59 smithi192 ceph-mon[33110]: osdmap e350: 8 total, 8 up, 6 in 2024-03-26T03:14:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:14:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 15956/290067 objects degraded (5.501%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:14:59.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:14:59.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:14:59.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:14:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:14:59.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:14:59.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:00.048 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:00.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:00.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.233 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:00.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:00.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:14:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:15:00.335 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.392 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:00.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:00.493 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:00.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:00.642 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.690 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:00.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:00.791 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.827 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:00.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:00.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:00.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:00.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:01.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:01.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:01.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:01.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:01.319 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:15:01.332 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.397 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:01.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:01.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:moved 20 / 534 (3.74532%) 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:avg 89 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:stddev 5.06623 -> 9.71253 (expected baseline 8.61201) 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:min osd.2 with 81 -> 84 pgs (0.910112 -> 0.94382 * mean) 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:max osd.5 with 95 -> 99 pgs (1.06742 -> 1.11236 * mean) 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:oload 110 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:max_change 3 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:max_change_osds 4 2024-03-26T03:15:01.503 INFO:teuthology.orchestra.run.smithi066.stdout:average_utilization 0.0340 2024-03-26T03:15:01.504 INFO:teuthology.orchestra.run.smithi066.stdout:overload_utilization 0.0374 2024-03-26T03:15:01.504 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4 weight 1.0000 -> 0.7930 2024-03-26T03:15:01.549 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[29666]: pgmap v734: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 33 MiB/s, 530 objects/s recovering 2024-03-26T03:15:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[29666]: osdmap e351: 8 total, 8 up, 6 in 2024-03-26T03:15:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'51971'}"}]: dispatch 2024-03-26T03:15:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[37920]: pgmap v734: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 33 MiB/s, 530 objects/s recovering 2024-03-26T03:15:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[37920]: osdmap e351: 8 total, 8 up, 6 in 2024-03-26T03:15:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'51971'}"}]: dispatch 2024-03-26T03:15:01.651 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:01.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:01 smithi192 ceph-mon[33110]: pgmap v734: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 33 MiB/s, 530 objects/s recovering 2024-03-26T03:15:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:01 smithi192 ceph-mon[33110]: osdmap e351: 8 total, 8 up, 6 in 2024-03-26T03:15:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'51971'}"}]: dispatch 2024-03-26T03:15:01.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:01.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:01.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:01.963 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:01.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:02.065 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:02.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:02.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:02.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:02.333 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:02.364 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:02.401 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-26T03:15:02.402 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-03-26T03:15:02.402 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:15:02.466 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:02.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:02.536 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:02.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:02.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:02 smithi066 ceph-mon[29666]: from='client.25657 v1:172.21.15.66:0/2140293610' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:02.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:02 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'51971'}"}]': finished 2024-03-26T03:15:02.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:02 smithi066 ceph-mon[29666]: osdmap e352: 8 total, 8 up, 6 in 2024-03-26T03:15:02.675 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:02 smithi066 ceph-mon[37920]: from='client.25657 v1:172.21.15.66:0/2140293610' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:02.676 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:02 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'51971'}"}]': finished 2024-03-26T03:15:02.676 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:02 smithi066 ceph-mon[37920]: osdmap e352: 8 total, 8 up, 6 in 2024-03-26T03:15:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.677 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:02.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:02.778 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:02.811 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:02.913 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:02.944 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:02.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:02 smithi192 ceph-mon[33110]: from='client.25657 v1:172.21.15.66:0/2140293610' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:02 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'51971'}"}]': finished 2024-03-26T03:15:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:02 smithi192 ceph-mon[33110]: osdmap e352: 8 total, 8 up, 6 in 2024-03-26T03:15:03.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:03.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:03.173 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.203 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:03.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:03.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:03.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:03.341 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:03.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:03.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:03.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:03.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:03.690 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:03.729 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:03.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:03 smithi066 ceph-mon[29666]: pgmap v737: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 853 op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 23 MiB/s, 360 objects/s recovering 2024-03-26T03:15:03.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:03 smithi066 ceph-mon[29666]: osdmap e353: 8 total, 8 up, 6 in 2024-03-26T03:15:03.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:03 smithi066 ceph-mon[37920]: pgmap v737: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 853 op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 23 MiB/s, 360 objects/s recovering 2024-03-26T03:15:03.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:03 smithi066 ceph-mon[37920]: osdmap e353: 8 total, 8 up, 6 in 2024-03-26T03:15:03.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.863 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:03.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:03.873 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:15:03.874 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:15:03.874 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:15:03.874 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:15:03.874 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:15:03.874 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:03 smithi192 ceph-mon[33110]: pgmap v737: 178 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 166 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 853 op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 23 MiB/s, 360 objects/s recovering 2024-03-26T03:15:03.874 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:03 smithi192 ceph-mon[33110]: osdmap e353: 8 total, 8 up, 6 in 2024-03-26T03:15:03.965 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:03.995 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:03.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:04.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:15:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:15:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:15:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:04.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:04.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:04.265 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.277 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:04.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:04.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:04.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:04.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:04.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:04.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.696 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:04.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:04.797 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:04.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:04.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[29666]: osdmap e354: 8 total, 8 up, 6 in 2024-03-26T03:15:04.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[29666]: 10.0 starting backfill to osd.5 from (0'0,0'0] MAX to 351'9976 2024-03-26T03:15:04.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 14520/274710 objects degraded (5.286%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:04.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[29666]: osdmap e355: 8 total, 8 up, 6 in 2024-03-26T03:15:04.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[37920]: osdmap e354: 8 total, 8 up, 6 in 2024-03-26T03:15:04.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[37920]: 10.0 starting backfill to osd.5 from (0'0,0'0] MAX to 351'9976 2024-03-26T03:15:04.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 14520/274710 objects degraded (5.286%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:04.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:04 smithi066 ceph-mon[37920]: osdmap e355: 8 total, 8 up, 6 in 2024-03-26T03:15:04.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:04.982 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:04.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:04 smithi192 ceph-mon[33110]: osdmap e354: 8 total, 8 up, 6 in 2024-03-26T03:15:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:04 smithi192 ceph-mon[33110]: 10.0 starting backfill to osd.5 from (0'0,0'0] MAX to 351'9976 2024-03-26T03:15:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 14520/274710 objects degraded (5.286%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:04 smithi192 ceph-mon[33110]: osdmap e355: 8 total, 8 up, 6 in 2024-03-26T03:15:05.084 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:05.124 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:05.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:05.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:15:05.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:05.267 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:05.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:05.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:05.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:05.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:05.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:05.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:05.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:05.624 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:15:05.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:05.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:05.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:05.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[29666]: pgmap v741: 178 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 165 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.8 KiB/s rd, 14 KiB/s wr, 2 op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 10 KiB/s, 1 objects/s recovering 2024-03-26T03:15:05.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:15:05.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:15:05.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[29666]: osdmap e356: 8 total, 8 up, 6 in 2024-03-26T03:15:05.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[37920]: pgmap v741: 178 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 165 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.8 KiB/s rd, 14 KiB/s wr, 2 op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 10 KiB/s, 1 objects/s recovering 2024-03-26T03:15:05.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:15:05.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:15:05.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:05 smithi066 ceph-mon[37920]: osdmap e356: 8 total, 8 up, 6 in 2024-03-26T03:15:05.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:05.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:05 smithi192 ceph-mon[33110]: pgmap v741: 178 pgs: 1 peering, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 165 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.8 KiB/s rd, 14 KiB/s wr, 2 op/s; 14520/274710 objects degraded (5.286%); 43984/274710 objects misplaced (16.011%); 10 KiB/s, 1 objects/s recovering 2024-03-26T03:15:05.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:05 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:15:05.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:05 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:15:05.824 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:05 smithi192 ceph-mon[33110]: osdmap e356: 8 total, 8 up, 6 in 2024-03-26T03:15:05.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:05.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:05.965 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.013 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:06.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:06.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:06.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:06.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.314 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:06.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:06.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:06.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:06.556 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:06.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:06.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:06.738 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:06.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:06 smithi192 ceph-mon[33110]: pgmap v743: 178 pgs: 4 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 163 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 545 op/s; 12801/268146 objects degraded (4.774%); 44518/268146 objects misplaced (16.602%); 67 MiB/s, 1.07k objects/s recovering 2024-03-26T03:15:06.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:06 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:06.830 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:06 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1126057586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:15:06.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:06.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:06 smithi066 ceph-mon[29666]: pgmap v743: 178 pgs: 4 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 163 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 545 op/s; 12801/268146 objects degraded (4.774%); 44518/268146 objects misplaced (16.602%); 67 MiB/s, 1.07k objects/s recovering 2024-03-26T03:15:06.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:06 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:06.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:06 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1126057586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:15:06.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:06 smithi066 ceph-mon[37920]: pgmap v743: 178 pgs: 4 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 163 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 545 op/s; 12801/268146 objects degraded (4.774%); 44518/268146 objects misplaced (16.602%); 67 MiB/s, 1.07k objects/s recovering 2024-03-26T03:15:06.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:06 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:06.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:06 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1126057586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:15:06.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:06.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:06.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:07.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:07.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:07.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:07.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:07.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:07.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:07.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:07.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:07.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:07.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:07.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:07.533 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:15:07.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:07.612 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:07.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:07.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:07.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:07.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159163 10.0 10:07efa237:::benchmark_data_smithi066_148944_object34516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.568736+0000", 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.2227063500000002, 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.024685979, 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.796 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159163 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568736+0000", 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568736+0000", 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568737+0000", 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.26e-06 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568742+0000", 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568756+0000", 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4408e-05 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568765+0000", 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4689999999999997e-06 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568828+0000", 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2966000000000007e-05 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568843+0000", 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5216e-05 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571792+0000", 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.002948945 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571804+0000", 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.221e-05 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.586864+0000", 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.015060139 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.587027+0000", 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000162974 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.587388+0000", 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000361245 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593072+0000", 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056834980000000004 2024-03-26T03:15:07.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593094+0000", 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1801999999999999e-05 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593154+0000", 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0226000000000002e-05 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593211+0000", 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6708e-05 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593287+0000", 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6550000000000004e-05 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593422+0000", 2024-03-26T03:15:07.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013466800000000001 2024-03-26T03:15:07.806 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159168 10.0 10:057c738e:::benchmark_data_smithi066_148944_object34521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.573141+0000", 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.2183014419999996, 2024-03-26T03:15:07.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0208405119999999, 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159168 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573141+0000", 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573141+0000", 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573143+0000", 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1490000000000001e-06 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573149+0000", 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6350000000000003e-06 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573164+0000", 2024-03-26T03:15:07.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5319e-05 2024-03-26T03:15:07.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573172+0000", 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5360000000000004e-06 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573569+0000", 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039733100000000001 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573579+0000", 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0019e-05 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571833+0000", 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99825384900000003 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571842+0000", 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6509999999999996e-06 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.590252+0000", 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0184094100000001 2024-03-26T03:15:07.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.590353+0000", 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010076799999999999 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.590460+0000", 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107865 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593438+0000", 2024-03-26T03:15:07.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029778199999999999 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593501+0000", 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2299999999999996e-05 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593741+0000", 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000239986 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593815+0000", 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4321999999999998e-05 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593868+0000", 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2905999999999998e-05 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593981+0000", 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113646 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.818 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159183 10.0 10:0070b402:::benchmark_data_smithi066_148944_object34536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.582557+0000", 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.2088851319999998, 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0118551139999998, 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159183 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582557+0000", 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582557+0000", 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582560+0000", 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4700000000000001e-06 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582567+0000", 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8519999999999998e-06 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582581+0000", 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4518999999999999e-05 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582589+0000", 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8339999999999999e-06 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582653+0000", 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3851999999999998e-05 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582664+0000", 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1088e-05 2024-03-26T03:15:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571867+0000", 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98920343799999999 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571875+0000", 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8099999999999998e-06 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.591745+0000", 2024-03-26T03:15:07.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0198699179999999 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.591829+0000", 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4005000000000006e-05 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.591907+0000", 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7583000000000006e-05 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593674+0000", 2024-03-26T03:15:07.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001767598 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594043+0000", 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036837300000000002 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594109+0000", 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6549000000000003e-05 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594314+0000", 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000204565 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594349+0000", 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5070000000000001e-05 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594412+0000", 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3590000000000006e-05 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.826 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159193 10.0 10:0574c5af:::benchmark_data_smithi066_148944_object34546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.594587+0000", 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.1968550100000002, 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0062639359999999, 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159193 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.827 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594587+0000", 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594587+0000", 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594589+0000", 2024-03-26T03:15:07.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3829999999999999e-06 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594594+0000", 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1680000000000003e-06 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594605+0000", 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1472e-05 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594614+0000", 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6119999999999999e-06 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594680+0000", 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5573000000000004e-05 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594702+0000", 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2402e-05 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571899+0000", 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97719667600000004 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571907+0000", 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2649999999999996e-06 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594452+0000", 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0225446570000001 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594577+0000", 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125141 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594711+0000", 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134618 2024-03-26T03:15:07.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600474+0000", 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057629489999999998 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600602+0000", 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012815099999999999 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600616+0000", 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3191e-05 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600666+0000", 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0908000000000001e-05 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600727+0000", 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0872999999999998e-05 2024-03-26T03:15:07.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600851+0000", 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012389700000000001 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159211 10.0 10:0157bc40:::benchmark_data_smithi066_148944_object34564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.614464+0000", 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.1769782080000004, 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9955604789999999, 2024-03-26T03:15:07.835 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159211 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614464+0000", 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614464+0000", 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614467+0000", 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4279999999999999e-06 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614478+0000", 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1469e-05 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614489+0000", 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1049999999999999e-05 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614497+0000", 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9990000000000008e-06 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614590+0000", 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2986999999999996e-05 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614602+0000", 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1805e-05 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571940+0000", 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95733820400000003 2024-03-26T03:15:07.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571949+0000", 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1959999999999997e-06 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.596335+0000", 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0243859980000001 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.596454+0000", 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119243 2024-03-26T03:15:07.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.596573+0000", 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118182 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600533+0000", 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039598990000000002 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600620+0000", 2024-03-26T03:15:07.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7175000000000005e-05 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600911+0000", 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029089099999999999 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.609788+0000", 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008877579 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.609859+0000", 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0611000000000005e-05 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610025+0000", 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016576300000000001 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159263 10.0 10:04aad00e:::benchmark_data_smithi066_148944_object34616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.657070+0000", 2024-03-26T03:15:07.843 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.1343718770000004, 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9531957680000001, 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159263 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657070+0000", 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657070+0000", 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657071+0000", 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2099999999999996e-07 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657074+0000", 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.898e-06 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657080+0000", 2024-03-26T03:15:07.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6620000000000002e-06 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657086+0000", 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8900000000000004e-06 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657143+0000", 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7263999999999999e-05 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657152+0000", 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5000000000000005e-06 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571986+0000", 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91483341600000001 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571994+0000", 2024-03-26T03:15:07.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0569999999999993e-06 2024-03-26T03:15:07.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.598412+0000", 2024-03-26T03:15:07.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.026417981 2024-03-26T03:15:07.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.598534+0000", 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012180699999999999 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.598676+0000", 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000142664 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600928+0000", 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022519150000000002 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600984+0000", 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5504999999999998e-05 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.601048+0000", 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4571000000000002e-05 2024-03-26T03:15:07.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610083+0000", 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0090350069999999994 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610141+0000", 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7781000000000001e-05 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610266+0000", 2024-03-26T03:15:07.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125229 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159266 10.0 10:03307f1d:::benchmark_data_smithi066_148944_object34619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.662107+0000", 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.1293357100000003, 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9524305749999999, 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.856 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159266 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662107+0000", 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662107+0000", 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662108+0000", 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0899999999999999e-06 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662112+0000", 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5210000000000004e-06 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662121+0000", 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3180000000000007e-06 2024-03-26T03:15:07.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662127+0000", 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7109999999999998e-06 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662187+0000", 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0081000000000003e-05 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662197+0000", 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0027000000000001e-05 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572018+0000", 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90982019599999997 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572026+0000", 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1750000000000005e-06 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.607627+0000", 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0356016690000001 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.607753+0000", 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012607300000000001 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.607891+0000", 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013716800000000001 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610329+0000", 2024-03-26T03:15:07.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:07.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:07 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:15:07.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1126057586' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:07 smithi066 ceph-mon[29666]: osdmap e357: 8 total, 8 up, 6 in 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:07 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1126057586' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:07 smithi192 ceph-mon[33110]: osdmap e357: 8 total, 8 up, 6 in 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:07 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1126057586' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:15:07.864 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:07 smithi066 ceph-mon[37920]: osdmap e357: 8 total, 8 up, 6 in 2024-03-26T03:15:07.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024379060000000001 2024-03-26T03:15:07.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610339+0000", 2024-03-26T03:15:07.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0549e-05 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614276+0000", 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039369419999999997 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614339+0000", 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3368999999999996e-05 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614401+0000", 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1693999999999996e-05 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614537+0000", 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136086 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.867 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159278 10.0 10:08f1fbe6:::benchmark_data_smithi066_148944_object34631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.671412+0000", 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.1200298709999998, 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.947903535, 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159278 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671412+0000", 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671412+0000", 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671414+0000", 2024-03-26T03:15:07.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4330000000000001e-06 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671419+0000", 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2660000000000001e-06 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671430+0000", 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0395e-05 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671438+0000", 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7739999999999993e-06 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671514+0000", 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5190000000000003e-05 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671525+0000", 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1328999999999999e-05 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572065+0000", 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90054052500000004 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572074+0000", 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-06 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610454+0000", 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0383793560000001 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610535+0000", 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1655000000000003e-05 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610662+0000", 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012715200000000001 2024-03-26T03:15:07.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614550+0000", 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038874399999999998 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614594+0000", 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4543999999999999e-05 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.618986+0000", 2024-03-26T03:15:07.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043918339999999998 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619134+0000", 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014828700000000001 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619195+0000", 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0328000000000002e-05 2024-03-26T03:15:07.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619316+0000", 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121227 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.877 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159281 10.0 10:0da9889e:::benchmark_data_smithi066_148944_object34634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.675662+0000", 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.1157807880000004, 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.944017356, 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.880 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159281 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675662+0000", 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675662+0000", 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675663+0000", 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2419999999999999e-06 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675668+0000", 2024-03-26T03:15:07.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0350000000000001e-06 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675678+0000", 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8609999999999993e-06 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675685+0000", 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5229999999999998e-06 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675774+0000", 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8348000000000002e-05 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675783+0000", 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4499999999999993e-06 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572099+0000", 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89631563800000003 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572107+0000", 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5080000000000005e-06 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.612698+0000", 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0405907619999999 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.612815+0000", 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116841 2024-03-26T03:15:07.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.612973+0000", 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015818000000000001 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.616650+0000", 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036775190000000002 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619061+0000", 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024105720000000001 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619366+0000", 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030508299999999999 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619521+0000", 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015539199999999999 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619575+0000", 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3526000000000002e-05 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619679+0000", 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010387600000000001 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159300 10.0 10:0d9da093:::benchmark_data_smithi066_148944_object34653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.888 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.696572+0000", 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.0948707049999999, 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.923426979, 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159300 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696572+0000", 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696572+0000", 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696573+0000", 2024-03-26T03:15:07.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7829999999999999e-06 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696579+0000", 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6389999999999997e-06 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696589+0000", 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6739999999999992e-06 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696597+0000", 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3259999999999996e-06 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696650+0000", 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3443000000000003e-05 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696658+0000", 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3139999999999998e-06 2024-03-26T03:15:07.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572176+0000", 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87551864400000001 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572186+0000", 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8279999999999998e-06 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614632+0000", 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.042446032 2024-03-26T03:15:07.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614749+0000", 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117083 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614902+0000", 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015228799999999999 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619379+0000", 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044775839999999997 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619465+0000", 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5970000000000005e-05 2024-03-26T03:15:07.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619717+0000", 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025213900000000002 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619831+0000", 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113278 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619886+0000", 2024-03-26T03:15:07.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5619999999999999e-05 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619999+0000", 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112334 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159309 10.0 10:08341859:::benchmark_data_smithi066_148944_object34662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.709099+0000", 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.082343034, 2024-03-26T03:15:07.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.917722997, 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159309 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709099+0000", 2024-03-26T03:15:07.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709099+0000", 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709100+0000", 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.212e-06 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709105+0000", 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.967e-06 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709114+0000", 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1030000000000002e-06 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709121+0000", 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2749999999999998e-06 2024-03-26T03:15:07.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709181+0000", 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0652000000000001e-05 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709191+0000", 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0890000000000006e-06 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572211+0000", 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86302047800000004 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572219+0000", 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9279999999999993e-06 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.616805+0000", 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0445862749999999 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.617013+0000", 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000207744 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.617128+0000", 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115396 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619385+0000", 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022570530000000002 2024-03-26T03:15:07.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619784+0000", 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039864300000000001 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.620061+0000", 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027677399999999998 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626596+0000", 2024-03-26T03:15:07.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006534832 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626656+0000", 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.071e-05 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626822+0000", 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000165866 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159331 10.0 10:0d8ff564:::benchmark_data_smithi066_148944_object34684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.730469+0000", 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.060973132, 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.900683873, 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159331 2024-03-26T03:15:07.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730469+0000", 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730469+0000", 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730470+0000", 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0449999999999999e-06 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730474+0000", 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9910000000000004e-06 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730483+0000", 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7530000000000007e-06 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730488+0000", 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4099999999999999e-06 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730552+0000", 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3639000000000005e-05 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.730559+0000", 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0600000000000002e-06 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572243+0000", 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.84168437299999999 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572252+0000", 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5089999999999993e-06 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.620092+0000", 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.047839964 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.620226+0000", 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013415200000000001 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.620347+0000", 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120418 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626449+0000", 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0061021169999999998 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626504+0000", 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5621000000000002e-05 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626560+0000", 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5358000000000001e-05 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.630978+0000", 2024-03-26T03:15:07.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044182320000000002 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631036+0000", 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8505999999999999e-05 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631153+0000", 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116725 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159344 10.0 10:071e9c67:::benchmark_data_smithi066_148944_object34697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.745889+0000", 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.0455533319999999, 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.885550678, 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159344 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.914 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745889+0000", 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745889+0000", 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745891+0000", 2024-03-26T03:15:07.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.525e-06 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745896+0000", 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745908+0000", 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2636999999999999e-05 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745917+0000", 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3979999999999999e-06 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745976+0000", 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9231999999999998e-05 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.745986+0000", 2024-03-26T03:15:07.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9720000000000004e-06 2024-03-26T03:15:07.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572308+0000", 2024-03-26T03:15:07.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.82632224499999996 2024-03-26T03:15:07.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572317+0000", 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2269999999999999e-06 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.622720+0000", 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0504025020000001 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.622851+0000", 2024-03-26T03:15:07.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013080599999999999 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.623062+0000", 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000211358 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626548+0000", 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034864789999999998 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.630933+0000", 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043849320000000002 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631200+0000", 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026660200000000001 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631250+0000", 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0303000000000003e-05 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631310+0000", 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0177999999999999e-05 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631440+0000", 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012919200000000001 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159352 10.0 10:0d249f60:::benchmark_data_smithi066_148944_object34705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.922 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.757500+0000", 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.0339425120000003, 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.878893546, 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159352 2024-03-26T03:15:07.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757500+0000", 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757500+0000", 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757501+0000", 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6160000000000001e-06 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757506+0000", 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6229999999999996e-06 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757517+0000", 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0549e-05 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757525+0000", 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0339999999999997e-06 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757588+0000", 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2933000000000002e-05 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.757599+0000", 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1491000000000001e-05 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572344+0000", 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.81474498200000001 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572353+0000", 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5609999999999999e-06 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626858+0000", 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.054505603 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626977+0000", 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119233 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.627089+0000", 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011128699999999999 2024-03-26T03:15:07.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631214+0000", 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041251869999999998 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631488+0000", 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027386400000000002 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.631533+0000", 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5432000000000001e-05 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.636170+0000", 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046364320000000002 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.636278+0000", 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108865 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:07.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.636393+0000", 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114854 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159359 10.0 10:0d1471c2:::benchmark_data_smithi066_148944_object34712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.768444+0000", 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.0229980190000001, 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.869021222, 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:07.932 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159359 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768444+0000", 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:07.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768444+0000", 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768446+0000", 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.55e-06 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768451+0000", 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1569999999999999e-06 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768460+0000", 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7800000000000006e-06 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768467+0000", 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6580000000000002e-06 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768536+0000", 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8953000000000002e-05 2024-03-26T03:15:07.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.768551+0000", 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4980999999999999e-05 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:07.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.573458+0000", 2024-03-26T03:15:08.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80490622300000003 2024-03-26T03:15:08.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.573476+0000", 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8410000000000002e-05 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.628944+0000", 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.055468064 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.629056+0000", 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112408 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.629176+0000", 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011955099999999999 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.636409+0000", 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072328239999999997 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637100+0000", 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00069105499999999999 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637211+0000", 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011136700000000001 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637287+0000", 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6165e-05 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:08.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637347+0000", 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9431000000000001e-05 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637466+0000", 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118645 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159375 10.0 10:093e9dea:::benchmark_data_smithi066_148944_object34728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.817524+0000", 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.9739179240000002, 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8292163559999999, 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:08.022 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159375 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817524+0000", 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817524+0000", 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817526+0000", 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.984e-06 2024-03-26T03:15:08.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817533+0000", 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7769999999999997e-06 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817545+0000", 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2299e-05 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817554+0000", 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5699999999999993e-06 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817649+0000", 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4629999999999994e-05 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.817660+0000", 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1538e-05 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.574052+0000", 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.75639226500000001 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.574063+0000", 2024-03-26T03:15:08.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0838e-05 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.633607+0000", 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.059543914 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.633724+0000", 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001166 2024-03-26T03:15:08.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.633850+0000", 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012617999999999999 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637243+0000", 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033928550000000002 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.637524+0000", 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028122500000000002 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.641919+0000", 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043945820000000002 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.646562+0000", 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046432330000000001 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.646622+0000", 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9868999999999999e-05 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:08.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.646741+0000", 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118997 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159416 10.f 10:f62808e3:::benchmark_data_smithi066_148944_object34769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:03.642121+0000", 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.1493208340000001, 2024-03-26T03:15:08.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55304394700000004, 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159416 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.033 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642121+0000", 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642121+0000", 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642123+0000", 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.897e-06 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642130+0000", 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3820000000000001e-06 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642142+0000", 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2517999999999999e-05 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642151+0000", 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6500000000000002e-06 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642208+0000", 2024-03-26T03:15:08.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6709999999999997e-05 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.642216+0000", 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3259999999999996e-06 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.180746+0000", 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53853013000000005 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.180844+0000", 2024-03-26T03:15:08.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8014e-05 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.181130+0000", 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028583500000000002 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.189215+0000", 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080850599999999998 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.194637+0000", 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054217400000000004 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195019+0000", 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038253199999999999 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195127+0000", 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108072 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195165+0000", 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8080999999999997e-05 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159422 10.f 10:f864a42b:::benchmark_data_smithi066_148944_object34775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-03-26T03:15:08.040 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:03.651867+0000", 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.1395748980000002, 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54349737099999995, 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159422 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651867+0000", 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651867+0000", 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:08.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651869+0000", 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.576e-06 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651874+0000", 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.293e-06 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651882+0000", 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9959999999999993e-06 2024-03-26T03:15:08.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651890+0000", 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2520000000000002e-06 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.651992+0000", 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102934 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.652007+0000", 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4124999999999999e-05 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.183705+0000", 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53169792400000004 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.183780+0000", 2024-03-26T03:15:08.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5197999999999994e-05 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.183845+0000", 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5339000000000006e-05 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.189241+0000", 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0053957650000000003 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.194684+0000", 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054432830000000001 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195275+0000", 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00059078400000000001 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195324+0000", 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8699000000000002e-05 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195365+0000", 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1202999999999998e-05 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159439 10.f 10:ffa313c3:::benchmark_data_smithi066_148944_object34792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:03.662403+0000", 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.1290389000000003, 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53553353699999995, 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:08.049 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159439 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662403+0000", 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:08.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662403+0000", 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662406+0000", 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:08.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662411+0000", 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5090000000000001e-06 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662427+0000", 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5573000000000002e-05 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662435+0000", 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0020000000000006e-06 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662489+0000", 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4063000000000001e-05 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.662497+0000", 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0330000000000008e-06 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.184699+0000", 2024-03-26T03:15:08.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.52220169500000002 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.184807+0000", 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108774 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.184928+0000", 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120643 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.189245+0000", 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043169819999999996 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.194735+0000", 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054901020000000002 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.195526+0000", 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000790901 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.197890+0000", 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023637929999999999 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.197937+0000", 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7237000000000002e-05 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.056 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159466 10.f 10:feeed75f:::benchmark_data_smithi066_148944_object34819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:03.679016+0000", 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.1124261449999997, 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51922974700000002, 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:15:08.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159466 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679016+0000", 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679016+0000", 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679018+0000", 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8020000000000001e-06 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:15:08.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679022+0000", 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5059999999999997e-06 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679034+0000", 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1065e-05 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679039+0000", 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9340000000000003e-06 2024-03-26T03:15:08.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679087+0000", 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7435999999999998e-05 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.679096+0000", 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5089999999999999e-06 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.186100+0000", 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50700371399999999 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.186170+0000", 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0312e-05 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,3", 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.186241+0000", 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0486000000000002e-05 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.189249+0000", 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003008419 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.194778+0000", 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055282170000000002 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.198148+0000", 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033699619999999998 2024-03-26T03:15:08.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.198204+0000", 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6478999999999999e-05 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:15:08.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:04.198246+0000", 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1906000000000002e-05 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:15:08.066 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:15:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:08.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:08.170 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:08.214 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:08.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:08.378 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:08.481 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:08.533 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:08.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:08.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:08.708 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:15:08.708 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:08 smithi066 ceph-mon[29666]: pgmap v745: 178 pgs: 4 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 163 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 12801/268146 objects degraded (4.774%); 44518/268146 objects misplaced (16.602%); 55 MiB/s, 888 objects/s recovering 2024-03-26T03:15:08.708 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:08 smithi066 ceph-mon[37920]: pgmap v745: 178 pgs: 4 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 163 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 12801/268146 objects degraded (4.774%); 44518/268146 objects misplaced (16.602%); 55 MiB/s, 888 objects/s recovering 2024-03-26T03:15:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:08 smithi192 ceph-mon[33110]: pgmap v745: 178 pgs: 4 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 163 active+clean; 5.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 0 B/s wr, 451 op/s; 12801/268146 objects degraded (4.774%); 44518/268146 objects misplaced (16.602%); 55 MiB/s, 888 objects/s recovering 2024-03-26T03:15:08.765 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:15:08.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:08.786 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:15:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:08.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:08.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:15:08.840 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:15:08.877 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:15:08.917 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:15:08.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:08.961 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:08.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:08.963 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:15:09.008 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:15:09.010 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:15:09.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:09.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:09.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:09.249 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:09.296 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:15:09.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:09.399 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:09.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:09.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:09.627 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:09 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/131067762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:09.628 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:09 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/131067762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:09.647 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:09.682 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:09 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/131067762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 12801/268146 objects degraded (4.774%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:09 smithi192 ceph-mon[33110]: osdmap e358: 8 total, 8 up, 6 in 2024-03-26T03:15:09.784 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:09.819 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:09.921 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:09.969 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:09.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:10.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:10.102 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 12801/268146 objects degraded (4.774%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:10.102 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:09 smithi066 ceph-mon[29666]: osdmap e358: 8 total, 8 up, 6 in 2024-03-26T03:15:10.102 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 12801/268146 objects degraded (4.774%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:10.102 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:09 smithi066 ceph-mon[37920]: osdmap e358: 8 total, 8 up, 6 in 2024-03-26T03:15:10.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:10.218 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:10.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:15:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:15:10.257 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:10.297 INFO:teuthology.orchestra.run.smithi066.stdout:moved 0 / 534 (0%) 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:avg 89 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:stddev 9.71253 -> 9.71253 (expected baseline 8.61201) 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:min osd.4 with 71 -> 71 pgs (0.797753 -> 0.797753 * mean) 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:max osd.3 with 99 -> 99 pgs (1.11236 -> 1.11236 * mean) 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:oload 1000 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:max_change 3 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:max_change_osds 4 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:average_utilization 0.0338 2024-03-26T03:15:10.298 INFO:teuthology.orchestra.run.smithi066.stdout:overload_utilization 0.3384 2024-03-26T03:15:10.299 INFO:teuthology.orchestra.run.smithi066.stderr:no change 2024-03-26T03:15:10.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:10.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:10.499 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:10.547 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:10.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:10.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:10.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:10.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:10.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[29666]: 2.a scrub starts 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[29666]: 2.a scrub ok 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[29666]: from='client.35705 v1:172.21.15.66:0/4126389901' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[29666]: pgmap v747: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.28k op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 62 MiB/s, 990 objects/s recovering 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[37920]: 2.a scrub starts 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[37920]: 2.a scrub ok 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[37920]: from='client.35705 v1:172.21.15.66:0/4126389901' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:10.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:10 smithi066 ceph-mon[37920]: pgmap v747: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.28k op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 62 MiB/s, 990 objects/s recovering 2024-03-26T03:15:10.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:10.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:10.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:10 smithi192 ceph-mon[33110]: 2.a scrub starts 2024-03-26T03:15:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:10 smithi192 ceph-mon[33110]: 2.a scrub ok 2024-03-26T03:15:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:10 smithi192 ceph-mon[33110]: from='client.35705 v1:172.21.15.66:0/4126389901' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:10.969 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:10 smithi192 ceph-mon[33110]: pgmap v747: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 0 B/s wr, 1.28k op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 62 MiB/s, 990 objects/s recovering 2024-03-26T03:15:11.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:11.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:11.215 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:11.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:11.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:11.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.407 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:11.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:11.508 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:11.544 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:11.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:11.689 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:11.791 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:11.823 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:11.925 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:11.966 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:11.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:12.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.110 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:12.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:12.211 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:12.249 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:12.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:12.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:12.495 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:12.537 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:12.602 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:15:12.602 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:15:12.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:12.677 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:12.779 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.824 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:12.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:12.926 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:12.963 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:12.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:13.065 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:13.102 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:13.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:13.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:13.244 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:13.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:13.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:13.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:13.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:13 smithi192 ceph-mon[33110]: pgmap v748: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 711 B/s rd, 0 B/s wr, 765 op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 14 MiB/s, 230 objects/s recovering 2024-03-26T03:15:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:13 smithi192 ceph-mon[33110]: 4.4 scrub starts 2024-03-26T03:15:13.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:13 smithi192 ceph-mon[33110]: 4.4 scrub ok 2024-03-26T03:15:13.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:13.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:13.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:13.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:13.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:15:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:13.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:13.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:13.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:13.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-mon[29666]: pgmap v748: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 711 B/s rd, 0 B/s wr, 765 op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 14 MiB/s, 230 objects/s recovering 2024-03-26T03:15:13.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-mon[29666]: 4.4 scrub starts 2024-03-26T03:15:13.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-mon[29666]: 4.4 scrub ok 2024-03-26T03:15:13.842 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-mon[37920]: pgmap v748: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 711 B/s rd, 0 B/s wr, 765 op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 14 MiB/s, 230 objects/s recovering 2024-03-26T03:15:13.842 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-mon[37920]: 4.4 scrub starts 2024-03-26T03:15:13.842 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-mon[37920]: 4.4 scrub ok 2024-03-26T03:15:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:13.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:13.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:13.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:14.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:15:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:15:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:15:14.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:14.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:14.206 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:15:14.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.341 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:14.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:14.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:14.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:14.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:14.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:14.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:14 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:14.655 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 11985/251631 objects degraded (4.763%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:14.731 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:14.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:14 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 11985/251631 objects degraded (4.763%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:14 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 11985/251631 objects degraded (4.763%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:14.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:14.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:14.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:15.013 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:15.055 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:15.157 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:15.197 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:15.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:15.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:15.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:15.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:15.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:15.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:15.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:15 smithi192 ceph-mon[33110]: pgmap v749: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 13 MiB/s, 206 objects/s recovering 2024-03-26T03:15:15.683 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:15 smithi192 ceph-mon[33110]: 5.12 scrub starts 2024-03-26T03:15:15.684 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:15 smithi192 ceph-mon[33110]: 5.12 scrub ok 2024-03-26T03:15:15.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:15.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:15 smithi066 ceph-mon[29666]: pgmap v749: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 13 MiB/s, 206 objects/s recovering 2024-03-26T03:15:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:15 smithi066 ceph-mon[29666]: 5.12 scrub starts 2024-03-26T03:15:15.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:15 smithi066 ceph-mon[29666]: 5.12 scrub ok 2024-03-26T03:15:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:15 smithi066 ceph-mon[37920]: pgmap v749: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 5.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s; 11985/251631 objects degraded (4.763%); 40356/251631 objects misplaced (16.038%); 13 MiB/s, 206 objects/s recovering 2024-03-26T03:15:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:15 smithi066 ceph-mon[37920]: 5.12 scrub starts 2024-03-26T03:15:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:15 smithi066 ceph-mon[37920]: 5.12 scrub ok 2024-03-26T03:15:15.880 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:15.920 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:15.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:16.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:16.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:16.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:16.205 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:16.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:16.343 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:16.445 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:16 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:15:16.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:16 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:15:16.488 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:15:16.516 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:15:16.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:16.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:16.729 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:16.760 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:16 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:15:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:16 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:15:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:16 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:15:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:16 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:15:16.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:16.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:16.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:17.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:17.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:17.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:17.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:17.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:17.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:15:17.285 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:17.326 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:17.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:17.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:17 smithi192 ceph-mon[33110]: pgmap v750: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s; 11183/236172 objects degraded (4.735%); 36063/236172 objects misplaced (15.270%); 26 MiB/s, 417 objects/s recovering 2024-03-26T03:15:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:17 smithi192 ceph-mon[33110]: 2.d scrub starts 2024-03-26T03:15:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:17 smithi192 ceph-mon[33110]: 2.d scrub ok 2024-03-26T03:15:17.578 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:17.620 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:17.686 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:15:17.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:17.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:17.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:17 smithi066 ceph-mon[29666]: pgmap v750: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s; 11183/236172 objects degraded (4.735%); 36063/236172 objects misplaced (15.270%); 26 MiB/s, 417 objects/s recovering 2024-03-26T03:15:17.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:17 smithi066 ceph-mon[29666]: 2.d scrub starts 2024-03-26T03:15:17.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:17 smithi066 ceph-mon[29666]: 2.d scrub ok 2024-03-26T03:15:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:17 smithi066 ceph-mon[37920]: pgmap v750: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.21k op/s; 11183/236172 objects degraded (4.735%); 36063/236172 objects misplaced (15.270%); 26 MiB/s, 417 objects/s recovering 2024-03-26T03:15:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:17 smithi066 ceph-mon[37920]: 2.d scrub starts 2024-03-26T03:15:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:17 smithi066 ceph-mon[37920]: 2.d scrub ok 2024-03-26T03:15:17.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:17.910 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:17.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:18.011 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:18.051 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:18.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:18.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:18.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:18.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:18.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:18.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:18.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:18.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.623 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:18.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:18.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:18 smithi192 ceph-mon[33110]: 6.4 scrub starts 2024-03-26T03:15:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:18 smithi192 ceph-mon[33110]: 6.4 scrub ok 2024-03-26T03:15:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:18.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:18.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:18 smithi066 ceph-mon[29666]: 6.4 scrub starts 2024-03-26T03:15:18.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:18 smithi066 ceph-mon[29666]: 6.4 scrub ok 2024-03-26T03:15:18.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:18 smithi066 ceph-mon[37920]: 6.4 scrub starts 2024-03-26T03:15:18.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:18 smithi066 ceph-mon[37920]: 6.4 scrub ok 2024-03-26T03:15:18.865 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:18.908 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:18.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:19.011 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:19.050 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:19.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:19.152 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:19.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:19.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:19.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:19.434 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:19.477 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:19.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:19.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:19.720 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:19 smithi192 ceph-mon[33110]: pgmap v751: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 11183/236172 objects degraded (4.735%); 36063/236172 objects misplaced (15.270%); 23 MiB/s, 367 objects/s recovering 2024-03-26T03:15:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 11183/236172 objects degraded (4.735%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.765 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:19.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:19 smithi066 ceph-mon[29666]: pgmap v751: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 11183/236172 objects degraded (4.735%); 36063/236172 objects misplaced (15.270%); 23 MiB/s, 367 objects/s recovering 2024-03-26T03:15:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 11183/236172 objects degraded (4.735%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:19 smithi066 ceph-mon[37920]: pgmap v751: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s; 11183/236172 objects degraded (4.735%); 36063/236172 objects misplaced (15.270%); 23 MiB/s, 367 objects/s recovering 2024-03-26T03:15:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 11183/236172 objects degraded (4.735%), 4 pgs degraded (PG_DEGRADED) 2024-03-26T03:15:19.866 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:19.898 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:19.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:19.999 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:20.042 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:20.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:20.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:20.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:15:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:15:20.288 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:20.330 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:20.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:20.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:20.574 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.606 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:20.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:20.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:20.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:20.872 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:20.915 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:20.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:21.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:21.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:21.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:21.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:21.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:21.338 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:21.344 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:15:21.344 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:15:21.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:21.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:21.584 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:21.634 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:21.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:21 smithi192 ceph-mon[33110]: pgmap v752: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.42k op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 33 MiB/s, 534 objects/s recovering 2024-03-26T03:15:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:21 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:15:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:21 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:15:21.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:21.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[29666]: pgmap v752: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.42k op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 33 MiB/s, 534 objects/s recovering 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[37920]: pgmap v752: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.42k op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 33 MiB/s, 534 objects/s recovering 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:15:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:21 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:15:21.869 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:21.899 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:21.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:22.001 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:22.033 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:22.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.169 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:22.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:22.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.302 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:22.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:22.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:22.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:22.531 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:22.539 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-26T03:15:22.539 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-03-26T03:15:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.575 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:22.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:22.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:22.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:22 smithi192 ceph-mon[33110]: osdmap e359: 8 total, 8 up, 6 in 2024-03-26T03:15:22.810 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:22.853 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:22.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:22 smithi066 ceph-mon[29666]: osdmap e359: 8 total, 8 up, 6 in 2024-03-26T03:15:22.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:22 smithi066 ceph-mon[37920]: osdmap e359: 8 total, 8 up, 6 in 2024-03-26T03:15:22.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:22.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:22.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:23.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:23.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:23.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:23.216 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:23.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:23.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:23.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:23.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:23.456 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:23.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:23.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:23.604 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:23.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:23 smithi066 ceph-mon[29666]: pgmap v754: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 27 MiB/s, 432 objects/s recovering 2024-03-26T03:15:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:23 smithi066 ceph-mon[29666]: osdmap e360: 8 total, 8 up, 6 in 2024-03-26T03:15:23.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:23 smithi066 ceph-mon[37920]: pgmap v754: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 27 MiB/s, 432 objects/s recovering 2024-03-26T03:15:23.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:23 smithi066 ceph-mon[37920]: osdmap e360: 8 total, 8 up, 6 in 2024-03-26T03:15:23.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:15:23.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:15:23.705 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:23 smithi192 ceph-mon[33110]: pgmap v754: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 27 MiB/s, 432 objects/s recovering 2024-03-26T03:15:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:23 smithi192 ceph-mon[33110]: osdmap e360: 8 total, 8 up, 6 in 2024-03-26T03:15:23.785 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:23.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:23.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:23.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:23.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:23.927 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:15:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:15:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:15:24.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:24.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:24.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:24.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:24.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:24.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:24.456 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:24.502 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:24.604 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:24.647 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:24 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 10240/220416 objects degraded (4.646%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:24.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:24.796 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:24 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 10240/220416 objects degraded (4.646%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:24 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 10240/220416 objects degraded (4.646%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:24.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:24.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:24.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:25.037 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:25.041 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:25.123 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:25.225 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:25.274 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:25.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:25.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:25.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:25.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:25.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:25.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:25.720 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:25 smithi192 ceph-mon[33110]: pgmap v756: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 18 MiB/s, 288 objects/s recovering 2024-03-26T03:15:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:25.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:25.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:25.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:25 smithi066 ceph-mon[29666]: pgmap v756: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 18 MiB/s, 288 objects/s recovering 2024-03-26T03:15:25.833 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:25.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:25 smithi066 ceph-mon[37920]: pgmap v756: 178 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 167 active+clean; 4.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 656 op/s; 10240/220416 objects degraded (4.646%); 31551/220416 objects misplaced (14.314%); 18 MiB/s, 288 objects/s recovering 2024-03-26T03:15:25.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:25.932 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:25.979 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:25.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:26.075 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:15:26.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:26.181 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:26.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:26.215 INFO:teuthology.orchestra.run.smithi066.stdout:moved 0 / 534 (0%) 2024-03-26T03:15:26.215 INFO:teuthology.orchestra.run.smithi066.stdout:avg 89 2024-03-26T03:15:26.215 INFO:teuthology.orchestra.run.smithi066.stdout:stddev 9.71253 -> 9.71253 (expected baseline 8.61201) 2024-03-26T03:15:26.215 INFO:teuthology.orchestra.run.smithi066.stdout:min osd.4 with 71 -> 71 pgs (0.797753 -> 0.797753 * mean) 2024-03-26T03:15:26.215 INFO:teuthology.orchestra.run.smithi066.stdout:max osd.3 with 99 -> 99 pgs (1.11236 -> 1.11236 * mean) 2024-03-26T03:15:26.215 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:15:26.216 INFO:teuthology.orchestra.run.smithi066.stdout:oload 1000 2024-03-26T03:15:26.216 INFO:teuthology.orchestra.run.smithi066.stdout:max_change 1 2024-03-26T03:15:26.216 INFO:teuthology.orchestra.run.smithi066.stdout:max_change_osds 4 2024-03-26T03:15:26.216 INFO:teuthology.orchestra.run.smithi066.stdout:average_utilization 0.0300 2024-03-26T03:15:26.216 INFO:teuthology.orchestra.run.smithi066.stdout:overload_utilization 0.3000 2024-03-26T03:15:26.224 INFO:teuthology.orchestra.run.smithi066.stderr:no change 2024-03-26T03:15:26.283 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:26.367 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:26.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:26.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:26.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:26.609 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:26.657 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:26.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:26.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:26.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:26.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:26.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:26 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:26.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:26 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:26.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:26.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:26.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:26.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:26 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:27.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:27.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:27.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:27.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:27.307 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:27.348 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:27.450 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:27.490 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:27.592 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:27.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:27.739 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:27.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:27.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:27 smithi066 ceph-mon[29666]: from='client.15861 v1:172.21.15.66:0/1900945006' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:27.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:27 smithi066 ceph-mon[29666]: pgmap v757: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 8927/207936 objects degraded (4.293%); 27241/207936 objects misplaced (13.101%); 42 MiB/s, 673 objects/s recovering 2024-03-26T03:15:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:27 smithi066 ceph-mon[37920]: from='client.15861 v1:172.21.15.66:0/1900945006' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:27 smithi066 ceph-mon[37920]: pgmap v757: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 8927/207936 objects degraded (4.293%); 27241/207936 objects misplaced (13.101%); 42 MiB/s, 673 objects/s recovering 2024-03-26T03:15:27.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:27.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:27.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:27 smithi192 ceph-mon[33110]: from='client.15861 v1:172.21.15.66:0/1900945006' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:27.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:27 smithi192 ceph-mon[33110]: pgmap v757: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 8927/207936 objects degraded (4.293%); 27241/207936 objects misplaced (13.101%); 42 MiB/s, 673 objects/s recovering 2024-03-26T03:15:28.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:28.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:28.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.224 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:28.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:28.325 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:28.368 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:28.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:28.606 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:28.649 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:28.751 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.795 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:28.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:28.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:28 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:15:28.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:28 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:15:28.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:28 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:15:28.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:28 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:15:28.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:28.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:28.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:28 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:15:28.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:28 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:15:29.041 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:29.088 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:29.190 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.235 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:29.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:29.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:29.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:29.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:29.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[29666]: pgmap v758: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 520 op/s; 8927/207936 objects degraded (4.293%); 27241/207936 objects misplaced (13.101%); 24 MiB/s, 385 objects/s recovering 2024-03-26T03:15:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[29666]: osdmap e361: 8 total, 8 up, 6 in 2024-03-26T03:15:29.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 8927/207936 objects degraded (4.293%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:29.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[29666]: osdmap e362: 8 total, 8 up, 6 in 2024-03-26T03:15:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[37920]: pgmap v758: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 520 op/s; 8927/207936 objects degraded (4.293%); 27241/207936 objects misplaced (13.101%); 24 MiB/s, 385 objects/s recovering 2024-03-26T03:15:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[37920]: osdmap e361: 8 total, 8 up, 6 in 2024-03-26T03:15:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 8927/207936 objects degraded (4.293%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:29.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:29 smithi066 ceph-mon[37920]: osdmap e362: 8 total, 8 up, 6 in 2024-03-26T03:15:29.640 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:29.684 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:29.786 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:29.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:29.898 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-26T03:15:29.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-26T03:15:29.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:29 smithi192 ceph-mon[33110]: pgmap v758: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 520 op/s; 8927/207936 objects degraded (4.293%); 27241/207936 objects misplaced (13.101%); 24 MiB/s, 385 objects/s recovering 2024-03-26T03:15:29.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:29 smithi192 ceph-mon[33110]: osdmap e361: 8 total, 8 up, 6 in 2024-03-26T03:15:29.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:29 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 8927/207936 objects degraded (4.293%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:29.910 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:29 smithi192 ceph-mon[33110]: osdmap e362: 8 total, 8 up, 6 in 2024-03-26T03:15:29.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:29.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:29.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:30.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:30.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:30.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:30.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:30.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:30.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:15:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:15:30.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:30.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:30.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:30.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:30.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:30.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:30.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:30.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:30.641 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:30.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:30.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:30.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:30.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:30.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:30.956 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:31.027 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:31.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:31.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:31.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:31.267 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:31.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:31.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:31.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:31.533 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:31.545 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:31.566 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:31.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:31.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:31 smithi066 ceph-mon[29666]: pgmap v761: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.24k op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:15:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:31.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:31 smithi066 ceph-mon[37920]: pgmap v761: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.24k op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:15:31.641 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:15:31.641 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:31.693 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:15:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:31 smithi192 ceph-mon[33110]: pgmap v761: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.24k op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 43 MiB/s, 694 objects/s recovering 2024-03-26T03:15:31.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:31.749 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:15:31.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:31.795 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:15:31.844 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:15:31.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:31.890 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:15:31.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:31.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:31.937 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:15:31.983 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:15:31.992 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:15:32.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:32.066 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:32.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.216 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:32.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:32.361 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:32.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:32.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:32.586 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:15:32.604 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:32.653 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:32.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:32.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3767719742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:32 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/293200484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:15:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:32 smithi066 ceph-mon[29666]: from='client.35717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:15:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3767719742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:32 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/293200484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:15:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:32 smithi066 ceph-mon[37920]: from='client.35717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:15:32.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:32.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:32.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3767719742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:32.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:32 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/293200484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:15:32.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:32 smithi192 ceph-mon[33110]: from='client.35717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:15:33.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:33.050 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:33.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.186 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:33.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:33.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:33.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:33.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:33.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:33.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:33.557 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:15:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=sqlstore.transactions t=2024-03-26T03:15:33.267715267Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T03:15:33.557 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:15:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=sqlstore.transactions t=2024-03-26T03:15:33.278036803Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T03:15:33.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:33.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:33.676 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[29666]: pgmap v762: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 43 MiB/s, 685 objects/s recovering 2024-03-26T03:15:33.676 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:15:33.676 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[29666]: from='client.35717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:15:33.677 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[29666]: osdmap e363: 8 total, 8 up, 6 in 2024-03-26T03:15:33.677 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[37920]: pgmap v762: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 43 MiB/s, 685 objects/s recovering 2024-03-26T03:15:33.677 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:15:33.677 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[37920]: from='client.35717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:15:33.677 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-mon[37920]: osdmap e363: 8 total, 8 up, 6 in 2024-03-26T03:15:33.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:33.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:33.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:33.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:33.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:33.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:33.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:33 smithi192 ceph-mon[33110]: pgmap v762: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 43 MiB/s, 685 objects/s recovering 2024-03-26T03:15:33.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:33 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:15:33.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:33 smithi192 ceph-mon[33110]: from='client.35717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:15:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:33 smithi192 ceph-mon[33110]: osdmap e363: 8 total, 8 up, 6 in 2024-03-26T03:15:34.062 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:15:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:15:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:15:34.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:34.107 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:34.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:34.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:34.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:34.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:34.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:34.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:34.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:34.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:34.767 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.817 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:34.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:34 smithi066 ceph-mon[29666]: osdmap e364: 8 total, 8 up, 6 in 2024-03-26T03:15:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:34 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 7333/191109 objects degraded (3.837%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:34.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:34 smithi066 ceph-mon[29666]: pgmap v765: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 878 B/s rd, 0 B/s wr, 963 op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 26 MiB/s, 411 objects/s recovering 2024-03-26T03:15:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:34 smithi066 ceph-mon[37920]: osdmap e364: 8 total, 8 up, 6 in 2024-03-26T03:15:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:34 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 7333/191109 objects degraded (3.837%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:34.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:34 smithi066 ceph-mon[37920]: pgmap v765: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 878 B/s rd, 0 B/s wr, 963 op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 26 MiB/s, 411 objects/s recovering 2024-03-26T03:15:34.918 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:34.967 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:34.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:34.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:34 smithi192 ceph-mon[33110]: osdmap e364: 8 total, 8 up, 6 in 2024-03-26T03:15:34.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:34 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 7333/191109 objects degraded (3.837%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:34 smithi192 ceph-mon[33110]: pgmap v765: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 168 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 878 B/s rd, 0 B/s wr, 963 op/s; 7333/191109 objects degraded (3.837%); 23485/191109 objects misplaced (12.289%); 26 MiB/s, 411 objects/s recovering 2024-03-26T03:15:35.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:35.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:35.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:35.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:35.352 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:35.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:35.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:35.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:35.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:35.697 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:35.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:15:35.777 INFO:teuthology.orchestra.run.smithi066.stdout:moved 13 / 534 (2.43446%) 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:avg 89 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:stddev 9.71253 -> 14.6401 (expected baseline 8.61201) 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:min osd.4 with 71 -> 58 pgs (0.797753 -> 0.651685 * mean) 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:max osd.3 with 99 -> 99 pgs (1.11236 -> 1.11236 * mean) 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:oload 110 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:max_change 3 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:max_change_osds 4 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:average_utilization 0.0277 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:overload_utilization 0.0305 2024-03-26T03:15:35.778 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4 weight 0.7930 -> 0.6559 2024-03-26T03:15:35.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:35.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:35.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:35.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:35.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:36.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:36.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:36.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:35 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'42987'}"}]: dispatch 2024-03-26T03:15:36.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:35 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'42987'}"}]: dispatch 2024-03-26T03:15:36.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:36.229 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:36.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:35 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'4':'42987'}"}]: dispatch 2024-03-26T03:15:36.330 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:36.364 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:36.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:36.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:36.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:36.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:36.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:15:36.720 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.755 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:36.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:36.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:36.891 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:36.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:36.993 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.068 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:37.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:37.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[29666]: from='client.35729 v1:172.21.15.66:0/313060774' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'42987'}"}]': finished 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[29666]: osdmap e365: 8 total, 8 up, 6 in 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[29666]: pgmap v767: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 169 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 758 op/s; 5269/177459 objects degraded (2.969%); 19983/177459 objects misplaced (11.261%); 32 MiB/s, 509 objects/s recovering 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[29666]: osdmap e366: 8 total, 8 up, 6 in 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[37920]: from='client.35729 v1:172.21.15.66:0/313060774' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'42987'}"}]': finished 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[37920]: osdmap e365: 8 total, 8 up, 6 in 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[37920]: pgmap v767: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 169 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 758 op/s; 5269/177459 objects degraded (2.969%); 19983/177459 objects misplaced (11.261%); 32 MiB/s, 509 objects/s recovering 2024-03-26T03:15:37.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:36 smithi066 ceph-mon[37920]: osdmap e366: 8 total, 8 up, 6 in 2024-03-26T03:15:37.169 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:37.203 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:37.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:36 smithi192 ceph-mon[33110]: from='client.35729 v1:172.21.15.66:0/313060774' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:15:37.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:36 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'42987'}"}]': finished 2024-03-26T03:15:37.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:36 smithi192 ceph-mon[33110]: osdmap e365: 8 total, 8 up, 6 in 2024-03-26T03:15:37.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:36 smithi192 ceph-mon[33110]: pgmap v767: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 169 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 758 op/s; 5269/177459 objects degraded (2.969%); 19983/177459 objects misplaced (11.261%); 32 MiB/s, 509 objects/s recovering 2024-03-26T03:15:37.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:36 smithi192 ceph-mon[33110]: osdmap e366: 8 total, 8 up, 6 in 2024-03-26T03:15:37.304 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:37.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:37.439 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:37.471 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:37.574 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:37.607 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:37.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:37.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:37.839 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:37.874 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:37.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:37.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:38.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:38.116 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:38.188 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:38.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:38.290 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.345 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:38.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:38.361 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:38.362 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:38.445 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:38.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:38.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:38.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:38.720 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:38.749 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:38.851 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:38.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:38.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:38.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.034 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:39.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:39.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:38 smithi066 ceph-mon[29666]: osdmap e367: 8 total, 8 up, 6 in 2024-03-26T03:15:39.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:38 smithi066 ceph-mon[29666]: 10.b starting backfill to osd.2 from (0'0,0'0] MAX to 364'12160 2024-03-26T03:15:39.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:38 smithi066 ceph-mon[29666]: pgmap v770: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 169 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s; 5269/177459 objects degraded (2.969%); 19983/177459 objects misplaced (11.261%); 40 MiB/s, 637 objects/s recovering 2024-03-26T03:15:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:38 smithi066 ceph-mon[37920]: osdmap e367: 8 total, 8 up, 6 in 2024-03-26T03:15:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:38 smithi066 ceph-mon[37920]: 10.b starting backfill to osd.2 from (0'0,0'0] MAX to 364'12160 2024-03-26T03:15:39.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:38 smithi066 ceph-mon[37920]: pgmap v770: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 169 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s; 5269/177459 objects degraded (2.969%); 19983/177459 objects misplaced (11.261%); 40 MiB/s, 637 objects/s recovering 2024-03-26T03:15:39.135 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:39.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:39.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:38 smithi192 ceph-mon[33110]: osdmap e367: 8 total, 8 up, 6 in 2024-03-26T03:15:39.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:38 smithi192 ceph-mon[33110]: 10.b starting backfill to osd.2 from (0'0,0'0] MAX to 364'12160 2024-03-26T03:15:39.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:38 smithi192 ceph-mon[33110]: pgmap v770: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 169 active+clean; 3.6 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s; 5269/177459 objects degraded (2.969%); 19983/177459 objects misplaced (11.261%); 40 MiB/s, 637 objects/s recovering 2024-03-26T03:15:39.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.365 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:39.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:39.466 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:39.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:39.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:39.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:39.753 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:39.798 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:39.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:39.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:39.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:40.038 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.073 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:40.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:40.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[29666]: osdmap e368: 8 total, 8 up, 6 in 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 5269/177459 objects degraded (2.969%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[37920]: osdmap e368: 8 total, 8 up, 6 in 2024-03-26T03:15:40.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 5269/177459 objects degraded (2.969%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:40.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:40.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:40.174 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:40.222 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:40.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:15:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:15:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:15:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-mon[33110]: osdmap e368: 8 total, 8 up, 6 in 2024-03-26T03:15:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 5269/177459 objects degraded (2.969%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:40.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:40.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:40.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:40.512 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:40.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:40.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:40.761 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:40.806 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:40.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:40 smithi066 ceph-mon[29666]: osdmap e369: 8 total, 8 up, 6 in 2024-03-26T03:15:40.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:40 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:40.863 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:40 smithi066 ceph-mon[29666]: pgmap v773: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 peering, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 166 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 924 op/s; 3085/166371 objects degraded (1.854%); 26436/166371 objects misplaced (15.890%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T03:15:40.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:40 smithi066 ceph-mon[37920]: osdmap e369: 8 total, 8 up, 6 in 2024-03-26T03:15:40.864 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:40 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:40.864 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:40 smithi066 ceph-mon[37920]: pgmap v773: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 peering, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 166 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 924 op/s; 3085/166371 objects degraded (1.854%); 26436/166371 objects misplaced (15.890%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T03:15:40.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:40.952 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:40.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:41.054 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:41.097 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:41.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:40 smithi192 ceph-mon[33110]: osdmap e369: 8 total, 8 up, 6 in 2024-03-26T03:15:41.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:40 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:41.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:40 smithi192 ceph-mon[33110]: pgmap v773: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 peering, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 166 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 924 op/s; 3085/166371 objects degraded (1.854%); 26436/166371 objects misplaced (15.890%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T03:15:41.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:41.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:41.342 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:41.392 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:41.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:41.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:41.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:41.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:41.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:41.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:41.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:41.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:41.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:41.995 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:15:41.995 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:15:41.996 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:15:41.996 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:15:42.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:42.105 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:41 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:42.105 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:41 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:42.116 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:42.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:42.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:41 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:42.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:42.352 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.401 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:42.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:42.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:42.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:42.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.694 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:42.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:42.796 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:42.840 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:42.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:42.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:42.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:43.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:43.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:42 smithi066 ceph-mon[29666]: pgmap v774: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 peering, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 166 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 917 B/s rd, 0 B/s wr, 662 op/s; 3085/166371 objects degraded (1.854%); 26436/166371 objects misplaced (15.890%); 46 MiB/s, 736 objects/s recovering 2024-03-26T03:15:43.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:42 smithi066 ceph-mon[37920]: pgmap v774: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 peering, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 166 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 917 B/s rd, 0 B/s wr, 662 op/s; 3085/166371 objects degraded (1.854%); 26436/166371 objects misplaced (15.890%); 46 MiB/s, 736 objects/s recovering 2024-03-26T03:15:43.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:43.218 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:43.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:42 smithi192 ceph-mon[33110]: pgmap v774: 178 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 peering, 2 active+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 166 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 917 B/s rd, 0 B/s wr, 662 op/s; 3085/166371 objects degraded (1.854%); 26436/166371 objects misplaced (15.890%); 46 MiB/s, 736 objects/s recovering 2024-03-26T03:15:43.263 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:43.365 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:43.411 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:43.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:43.531 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:15:43.531 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:196615 10.d e369/331 10:b2ab1b18:::benchmark_data_smithi066_148944_object71968:head v 369'12291, mlcod=369'12289)", 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.981335+0000", 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0040357570000000001, 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0040672709999999999, 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981335+0000", 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.532 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981335+0000", 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981336+0000", 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4309999999999999e-06 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:15:43.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981344+0000", 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2989999999999999e-06 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981356+0000", 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2101e-05 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981363+0000", 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8929999999999999e-06 2024-03-26T03:15:43.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981425+0000", 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2107999999999996e-05 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.981444+0000", 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9582e-05 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:196617 10.5 e369/341 10:a1c27b58:::benchmark_data_smithi066_148944_object71970:head v 369'12318, mlcod=369'12316)", 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.982565+0000", 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0028057379999999999, 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0029091709999999999, 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.536 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982565+0000", 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982565+0000", 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982566+0000", 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4e-07 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982573+0000", 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7059999999999998e-06 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982582+0000", 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8789999999999992e-06 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982588+0000", 2024-03-26T03:15:43.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6649999999999997e-06 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982660+0000", 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1567000000000001e-05 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.539 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.982681+0000", 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0825000000000001e-05 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:196607 10.3 e369/332 10:c4e28996:::benchmark_data_smithi066_148944_object71960:head v 369'12480, mlcod=369'12476)", 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.984562+0000", 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.00080838399999999999, 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00097561399999999997, 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:15:43.540 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984562+0000", 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984562+0000", 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984564+0000", 2024-03-26T03:15:43.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.269e-06 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984571+0000", 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6720000000000004e-06 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984586+0000", 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4479e-05 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:15:43.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984594+0000", 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8429999999999993e-06 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984670+0000", 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6235000000000004e-05 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984692+0000", 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2172999999999999e-05 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:15:43.543 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "MOSDPGPush(10.d 369/331 [PushOp(10:bb997a1b:::benchmark_data_smithi066_148944_object121711:head, version: 329'7598, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:bb997a1b:::benchmark_data_smithi066_148944_object121711:head@329'7598, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.984946+0000", 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.00042432499999999997, 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00071010799999999998, 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984946+0000", 2024-03-26T03:15:43.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984946+0000", 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.984948+0000", 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4020000000000001e-06 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:15:43.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985005+0000", 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6994999999999999e-05 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985021+0000", 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6422e-05 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985036+0000", 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5091e-05 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:15:43.546 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "pg_backfill(progress 10.d e 369/369 lb 10:bb97b30e:::benchmark_data_smithi066_148944_object108570:head)", 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.985643+0000", 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 4294967295.9997277, 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5647000000000003e-05, 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "queued for pg", 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985643+0000", 2024-03-26T03:15:43.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985643+0000", 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985644+0000", 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.09e-07 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:15:43.548 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985652+0000", 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6969999999999993e-06 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985660+0000", 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.456e-06 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:15:43.549 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.985667+0000", 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.489e-06 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 5 2024-03-26T03:15:43.550 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:15:43.556 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:43.658 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:43.705 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:43.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:43.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:43.944 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:43.991 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:43.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:44.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:15:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:15:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:15:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:44.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:44.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:44.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:44.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:44.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:44.477 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:15:44.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:44.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:44.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:44.662 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:44.764 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:44.801 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:44.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:44.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:44.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:45.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:45.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:45.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:45.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:44 smithi192 ceph-mon[33110]: osdmap e370: 8 total, 8 up, 6 in 2024-03-26T03:15:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:44 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 3085/166371 objects degraded (1.854%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:45.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:44 smithi192 ceph-mon[33110]: pgmap v776: 178 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 168 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 665 op/s; 2741/165483 objects degraded (1.656%); 26436/165483 objects misplaced (15.975%); 46 MiB/s, 741 objects/s recovering 2024-03-26T03:15:45.306 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:45.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:45.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:44 smithi066 ceph-mon[29666]: osdmap e370: 8 total, 8 up, 6 in 2024-03-26T03:15:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:44 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 3085/166371 objects degraded (1.854%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:45.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:44 smithi066 ceph-mon[29666]: pgmap v776: 178 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 168 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 665 op/s; 2741/165483 objects degraded (1.656%); 26436/165483 objects misplaced (15.975%); 46 MiB/s, 741 objects/s recovering 2024-03-26T03:15:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:44 smithi066 ceph-mon[37920]: osdmap e370: 8 total, 8 up, 6 in 2024-03-26T03:15:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:44 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 3085/166371 objects degraded (1.854%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:45.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:44 smithi066 ceph-mon[37920]: pgmap v776: 178 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 2 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 168 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 665 op/s; 2741/165483 objects degraded (1.656%); 26436/165483 objects misplaced (15.975%); 46 MiB/s, 741 objects/s recovering 2024-03-26T03:15:45.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:45.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:45.572 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.611 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:45.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:45.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:45.749 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:45.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:45.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:45.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:45.986 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:46.014 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:46.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:45 smithi066 ceph-mon[29666]: osdmap e371: 8 total, 8 up, 6 in 2024-03-26T03:15:46.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:45 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:15:46.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:45 smithi066 ceph-mon[29666]: osdmap e372: 8 total, 8 up, 6 in 2024-03-26T03:15:46.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:45 smithi066 ceph-mon[37920]: osdmap e371: 8 total, 8 up, 6 in 2024-03-26T03:15:46.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:45 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:15:46.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:45 smithi066 ceph-mon[37920]: osdmap e372: 8 total, 8 up, 6 in 2024-03-26T03:15:46.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:46.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:46.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:45 smithi192 ceph-mon[33110]: osdmap e371: 8 total, 8 up, 6 in 2024-03-26T03:15:46.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:45 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:15:46.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:45 smithi192 ceph-mon[33110]: osdmap e372: 8 total, 8 up, 6 in 2024-03-26T03:15:46.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:46.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:46.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:46.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:46.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:46.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:46.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:46.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:46.837 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:46.872 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:15:46.895 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:46.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:46.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:47.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:47.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:15:47.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:15:47.108 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: pgmap v779: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 6 active+remapped+backfilling, 168 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 2097/151455 objects degraded (1.385%); 19897/151455 objects misplaced (13.137%); 46 MiB/s, 739 objects/s recovering 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2904003360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: from='client.35735 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: from='client.35735 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[37920]: osdmap e373: 8 total, 8 up, 6 in 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:15:47.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: pgmap v779: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 6 active+remapped+backfilling, 168 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 2097/151455 objects degraded (1.385%); 19897/151455 objects misplaced (13.137%); 46 MiB/s, 739 objects/s recovering 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2904003360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: from='client.35735 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: from='client.35735 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:15:47.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:46 smithi066 ceph-mon[29666]: osdmap e373: 8 total, 8 up, 6 in 2024-03-26T03:15:47.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:47.179 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:47.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:15:47.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: pgmap v779: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 6 active+remapped+backfilling, 168 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 829 op/s; 2097/151455 objects degraded (1.385%); 19897/151455 objects misplaced (13.137%); 46 MiB/s, 739 objects/s recovering 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2904003360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: from='client.35735 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: from='client.35735 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:15:47.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:46 smithi192 ceph-mon[33110]: osdmap e373: 8 total, 8 up, 6 in 2024-03-26T03:15:47.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:47.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:47.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:47.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:47.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.603 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:47.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:47.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:47.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:47.835 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:47.880 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:47.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:47.982 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:48.026 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:48.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.162 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:48.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:48.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:47 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:48.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:48.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:47 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:48.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:47 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:15:48.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:47 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:15:48.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:48.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:15:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:47 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:15:48.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:48.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:48.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:48.598 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:48.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:48.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:48.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:48.842 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:48.883 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:48.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:48.984 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.031 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:49.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:49.132 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:49.173 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:49.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:49.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:49.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:48 smithi066 ceph-mon[29666]: osdmap e374: 8 total, 8 up, 6 in 2024-03-26T03:15:49.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:48 smithi066 ceph-mon[29666]: pgmap v782: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 6 active+remapped+backfilling, 168 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 2097/151455 objects degraded (1.385%); 19897/151455 objects misplaced (13.137%); 64 MiB/s, 1.02k objects/s recovering 2024-03-26T03:15:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:48 smithi066 ceph-mon[37920]: osdmap e374: 8 total, 8 up, 6 in 2024-03-26T03:15:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:48 smithi066 ceph-mon[37920]: pgmap v782: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 6 active+remapped+backfilling, 168 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 2097/151455 objects degraded (1.385%); 19897/151455 objects misplaced (13.137%); 64 MiB/s, 1.02k objects/s recovering 2024-03-26T03:15:49.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.455 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:49.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:49.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:48 smithi192 ceph-mon[33110]: osdmap e374: 8 total, 8 up, 6 in 2024-03-26T03:15:49.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:48 smithi192 ceph-mon[33110]: pgmap v782: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+remapped, 6 active+remapped+backfilling, 168 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 2097/151455 objects degraded (1.385%); 19897/151455 objects misplaced (13.137%); 64 MiB/s, 1.02k objects/s recovering 2024-03-26T03:15:49.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:49.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:49.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:49.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:49.831 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:49.866 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:49.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:49.969 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:49.999 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:50.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:50.143 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:50.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:15:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:15:50.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:49 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:15:50.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:49 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:15:50.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:49 smithi192 ceph-mon[33110]: osdmap e375: 8 total, 8 up, 6 in 2024-03-26T03:15:50.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:49 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 2097/151455 objects degraded (1.385%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:50.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:49 smithi192 ceph-mon[33110]: osdmap e376: 8 total, 8 up, 6 in 2024-03-26T03:15:50.245 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.287 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:50.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:50.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:15:50.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:15:50.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[37920]: osdmap e375: 8 total, 8 up, 6 in 2024-03-26T03:15:50.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 2097/151455 objects degraded (1.385%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:50.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[37920]: osdmap e376: 8 total, 8 up, 6 in 2024-03-26T03:15:50.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:15:50.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:15:50.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[29666]: osdmap e375: 8 total, 8 up, 6 in 2024-03-26T03:15:50.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 2097/151455 objects degraded (1.385%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:15:50.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:49 smithi066 ceph-mon[29666]: osdmap e376: 8 total, 8 up, 6 in 2024-03-26T03:15:50.388 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:50.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:50.536 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:50.571 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:50.673 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:50.713 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:50.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:50.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:50.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:50.994 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:50.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:51.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:51.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:51.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:51.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:51.261 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:51.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:51.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:51.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:51 smithi066 ceph-mon[37920]: pgmap v785: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 880 op/s; 1944/140901 objects degraded (1.380%); 12703/140901 objects misplaced (9.016%); 84 MiB/s, 1.35k objects/s recovering 2024-03-26T03:15:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:51 smithi066 ceph-mon[29666]: pgmap v785: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 880 op/s; 1944/140901 objects degraded (1.380%); 12703/140901 objects misplaced (9.016%); 84 MiB/s, 1.35k objects/s recovering 2024-03-26T03:15:51.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:51.427 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:51.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:15:51.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:51 smithi192 ceph-mon[33110]: pgmap v785: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 880 op/s; 1944/140901 objects degraded (1.380%); 12703/140901 objects misplaced (9.016%); 84 MiB/s, 1.35k objects/s recovering 2024-03-26T03:15:51.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:51.581 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:51.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:51.682 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:51.727 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:51.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:51.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:51.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:51.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:51.968 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.014 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:52.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:52.116 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:52.160 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:52.262 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:52.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:52.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:15:52.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:15:52.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:15:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:15:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:52.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:52.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:52.467 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:15:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:15:52.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:52.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:52.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:52.687 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:52.731 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:52.833 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:52.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:52.967 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:53.012 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:53.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:53.157 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:53.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:53.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:53.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:53 smithi066 ceph-mon[29666]: pgmap v786: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 638 op/s; 1944/140901 objects degraded (1.380%); 12703/140901 objects misplaced (9.016%); 61 MiB/s, 976 objects/s recovering 2024-03-26T03:15:53.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:53 smithi066 ceph-mon[37920]: pgmap v786: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 638 op/s; 1944/140901 objects degraded (1.380%); 12703/140901 objects misplaced (9.016%); 61 MiB/s, 976 objects/s recovering 2024-03-26T03:15:53.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:53.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:53.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:53 smithi192 ceph-mon[33110]: pgmap v786: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 638 op/s; 1944/140901 objects degraded (1.380%); 12703/140901 objects misplaced (9.016%); 61 MiB/s, 976 objects/s recovering 2024-03-26T03:15:53.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:53.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:53.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:53.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:53.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:53.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:53.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:53.989 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:53.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:54.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:15:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:15:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:15:54.129 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:54.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:54.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:54.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:54.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:54.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:54.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:54.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:54.687 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:54.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:54.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:54.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:54.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:54.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:55.057 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:55.093 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:15:55.132 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:55.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:55.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:55.288 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:55.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:55.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[37920]: osdmap e377: 8 total, 8 up, 6 in 2024-03-26T03:15:55.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 1944/140901 objects degraded (1.380%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:55.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[37920]: osdmap e378: 8 total, 8 up, 6 in 2024-03-26T03:15:55.331 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[37920]: pgmap v789: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 952 B/s rd, 0 B/s wr, 718 op/s; 1944/139890 objects degraded (1.390%); 12703/139890 objects misplaced (9.081%); 63 MiB/s, 1.00k objects/s recovering 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[29666]: osdmap e377: 8 total, 8 up, 6 in 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 1944/140901 objects degraded (1.380%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[29666]: osdmap e378: 8 total, 8 up, 6 in 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[29666]: pgmap v789: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 952 B/s rd, 0 B/s wr, 718 op/s; 1944/139890 objects degraded (1.390%); 12703/139890 objects misplaced (9.081%); 63 MiB/s, 1.00k objects/s recovering 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:55.332 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:55.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:55.443 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:55.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:55.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:55 smithi192 ceph-mon[33110]: osdmap e377: 8 total, 8 up, 6 in 2024-03-26T03:15:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:55 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 1944/140901 objects degraded (1.380%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:55 smithi192 ceph-mon[33110]: osdmap e378: 8 total, 8 up, 6 in 2024-03-26T03:15:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:55 smithi192 ceph-mon[33110]: pgmap v789: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+remapped+backfilling, 171 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 952 B/s rd, 0 B/s wr, 718 op/s; 1944/139890 objects degraded (1.390%); 12703/139890 objects misplaced (9.081%); 63 MiB/s, 1.00k objects/s recovering 2024-03-26T03:15:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:15:55.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:15:55.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:55.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:55.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:55.679 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:55.727 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:55.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:55.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:55.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:55.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:55.963 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:56.009 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:56.111 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:56.158 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:56.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:56.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:56.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:56.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:56.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.569 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:56.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:56.608 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:15:56.651 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:15:56.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:56.699 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:15:56.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:56.743 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:15:56.786 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:15:56.813 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:56.828 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:15:56.858 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:56.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:56.870 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:15:56.915 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:15:56.922 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:15:56.923 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:15:56.960 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:57.005 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:57.085 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:15:57.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:57.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:57.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:57.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:57.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:57.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:57.521 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:57.580 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:57.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:57.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:57 smithi192 ceph-mon[33110]: pgmap v790: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 172 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.23k op/s; 1730/124995 objects degraded (1.384%); 8073/124995 objects misplaced (6.459%); 70 MiB/s, 1.12k objects/s recovering 2024-03-26T03:15:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2709064009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:57.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:57.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:57 smithi066 ceph-mon[29666]: pgmap v790: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 172 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.23k op/s; 1730/124995 objects degraded (1.384%); 8073/124995 objects misplaced (6.459%); 70 MiB/s, 1.12k objects/s recovering 2024-03-26T03:15:57.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2709064009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:57 smithi066 ceph-mon[37920]: pgmap v790: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 172 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.23k op/s; 1730/124995 objects degraded (1.384%); 8073/124995 objects misplaced (6.459%); 70 MiB/s, 1.12k objects/s recovering 2024-03-26T03:15:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2709064009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:15:57.870 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:57.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:57.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:58.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:58.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:58.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:58.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:58.294 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:58.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:58.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:58.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:15:58.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:15:58.671 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:15:58.710 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:15:58.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:15:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:58.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:15:58.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:58.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:58.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:59.071 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:15:59.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:15:59.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:59.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:15:59.340 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:59.386 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:59.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:15:59.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:15:59.624 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:15:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.675 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:59.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:15:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:59 smithi192 ceph-mon[33110]: pgmap v791: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 172 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 662 op/s; 1730/124995 objects degraded (1.384%); 8073/124995 objects misplaced (6.459%); 21 MiB/s, 331 objects/s recovering 2024-03-26T03:15:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:59 smithi192 ceph-mon[33110]: osdmap e379: 8 total, 8 up, 6 in 2024-03-26T03:15:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:59 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 1730/124995 objects degraded (1.384%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:15:59 smithi192 ceph-mon[33110]: osdmap e380: 8 total, 8 up, 6 in 2024-03-26T03:15:59.744 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[29666]: pgmap v791: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 172 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 662 op/s; 1730/124995 objects degraded (1.384%); 8073/124995 objects misplaced (6.459%); 21 MiB/s, 331 objects/s recovering 2024-03-26T03:15:59.744 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[29666]: osdmap e379: 8 total, 8 up, 6 in 2024-03-26T03:15:59.744 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 1730/124995 objects degraded (1.384%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:59.744 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[29666]: osdmap e380: 8 total, 8 up, 6 in 2024-03-26T03:15:59.745 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[37920]: pgmap v791: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 4 active+remapped+backfilling, 172 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 662 op/s; 1730/124995 objects degraded (1.384%); 8073/124995 objects misplaced (6.459%); 21 MiB/s, 331 objects/s recovering 2024-03-26T03:15:59.746 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[37920]: osdmap e379: 8 total, 8 up, 6 in 2024-03-26T03:15:59.746 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 1730/124995 objects degraded (1.384%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:15:59.746 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:15:59 smithi066 ceph-mon[37920]: osdmap e380: 8 total, 8 up, 6 in 2024-03-26T03:15:59.776 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:15:59.821 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:15:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:15:59.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:15:59.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:15:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:15:59.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:00.061 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:00.105 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:00.207 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:00.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:15:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:16:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.251 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:00.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:00.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:00.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:00.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:00.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:00.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:00.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:00.754 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.793 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:00.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:00.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:00.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:00.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:01.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:01.083 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:01.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:01.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:01.310 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:01.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:01.440 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:01.474 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:01.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:01.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:01.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:01 smithi192 ceph-mon[33110]: pgmap v794: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.72k op/s; 1506/108906 objects degraded (1.383%); 3654/108906 objects misplaced (3.355%); 50 MiB/s, 806 objects/s recovering 2024-03-26T03:16:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:01 smithi192 ceph-mon[33110]: osdmap e381: 8 total, 8 up, 6 in 2024-03-26T03:16:01.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:01.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:01 smithi066 ceph-mon[29666]: pgmap v794: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.72k op/s; 1506/108906 objects degraded (1.383%); 3654/108906 objects misplaced (3.355%); 50 MiB/s, 806 objects/s recovering 2024-03-26T03:16:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:01 smithi066 ceph-mon[29666]: osdmap e381: 8 total, 8 up, 6 in 2024-03-26T03:16:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:01 smithi066 ceph-mon[37920]: pgmap v794: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.72k op/s; 1506/108906 objects degraded (1.383%); 3654/108906 objects misplaced (3.355%); 50 MiB/s, 806 objects/s recovering 2024-03-26T03:16:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:01 smithi066 ceph-mon[37920]: osdmap e381: 8 total, 8 up, 6 in 2024-03-26T03:16:01.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:01.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:01.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:01.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:02.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.119 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:02.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:02.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:02.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:02.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:02.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:02.472 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.509 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:02.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:02.560 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:02 smithi066 ceph-mon[29666]: osdmap e382: 8 total, 8 up, 6 in 2024-03-26T03:16:02.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:02 smithi066 ceph-mon[37920]: osdmap e382: 8 total, 8 up, 6 in 2024-03-26T03:16:02.610 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:02.648 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:02 smithi192 ceph-mon[33110]: osdmap e382: 8 total, 8 up, 6 in 2024-03-26T03:16:02.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:02.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:02.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:02.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:02.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:03.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:03.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:03.074 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:16:03.074 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:16:03.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:03.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:03.251 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:03.278 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 383, 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:16:02.468809+0000", 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T03:14:20.518826+0000", 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T03:14:36.200040+0000", 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5801984, 2024-03-26T03:16:03.366 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "noscrub", 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 19, 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 10, 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:16:03.367 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.368 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 31, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 31, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 31, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 31, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 31, 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.369 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "305", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "303", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.370 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.371 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.372 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:16:03.373 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.7999999523162842, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.7999999523162842, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.3500000238418579, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.3500000238418579, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.374 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.375 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.376 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:16:03.377 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.379 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.25, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.25, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.690000057220459, 2024-03-26T03:16:03.380 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.690000057220459, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.381 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:16:03.382 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:16:03.383 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.384 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.385 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:16:03.386 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 5.2899999618530273, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 5.2899999618530273, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 3.3299999237060547, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 3.3299999237060547, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.388 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.389 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.390 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.391 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.392 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 3, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 3, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.25, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.25, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:16:03.394 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:16:03.395 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.396 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.397 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.398 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.5099999904632568, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.5099999904632568, 2024-03-26T03:16:03.399 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.1299999952316284, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.1299999952316284, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:16:03.400 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:16:03.401 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.402 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.403 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.404 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.405 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 3, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 3, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.25, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.25, 2024-03-26T03:16:03.406 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 10, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_3", 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:12:34.415377+0000", 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:16:03.407 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 16, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 16, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 16, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 16, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 16, 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:16:03.408 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "296", 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:16:03.409 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:16:03.410 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:16:03.411 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 3, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 3, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.75, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.25, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.25, 2024-03-26T03:16:03.412 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.90595245361328125, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.9100000262260437 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 0, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 0, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.413 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 313, 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.416 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 379, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.418 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.419 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.420 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:16:03.421 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.6510009765625, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.422 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 382, 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:16:03.423 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.424 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:16:03.425 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:16:03.426 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 26, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 197, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 331, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 370, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 198, 2024-03-26T03:16:03.427 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.428 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.429 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/52636497", 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/52636497", 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/52636497", 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/52636497", 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.430 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 0.6559295654296875, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:16:03.431 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 382, 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.432 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.433 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:16:03.435 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.436 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 377, 2024-03-26T03:16:03.437 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.438 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.440 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:16:03.441 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 381, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.442 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.444 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.445 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:16:03.446 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 0, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 0, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 341, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:16:03.447 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.448 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:16:03.449 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:16:03.450 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.451 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 65536, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:15:27.782296+0000", 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:15:48.254003+0000", 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:15:47.413461+0000", 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.453 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:15:46.313960+0000", 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 197 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:16:03.454 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:15:38.724804+0000", 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:14:44.557445+0000", 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:15:04.768039+0000", 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.456 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 65536, 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:14:54.343043+0000", 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.457 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [ 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.f", 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:16:03.458 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.459 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.460 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [ 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "10.7", 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: 6, 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: 5 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "10.c", 2024-03-26T03:16:03.461 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: 7, 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: 0 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:16:03.462 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:16:03.463 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:16:03.464 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:16:03.465 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:16:03.465 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:03.465 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:16:03.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:03.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:03.593 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:03.639 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:03 smithi192 ceph-mon[33110]: pgmap v797: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 1506/108906 objects degraded (1.383%); 3654/108906 objects misplaced (3.355%); 36 MiB/s, 583 objects/s recovering 2024-03-26T03:16:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:03 smithi192 ceph-mon[33110]: osdmap e383: 8 total, 8 up, 6 in 2024-03-26T03:16:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/473428107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:16:03.741 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:03.749 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-mon[29666]: pgmap v797: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 1506/108906 objects degraded (1.383%); 3654/108906 objects misplaced (3.355%); 36 MiB/s, 583 objects/s recovering 2024-03-26T03:16:03.749 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-mon[29666]: osdmap e383: 8 total, 8 up, 6 in 2024-03-26T03:16:03.749 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/473428107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:16:03.749 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-mon[37920]: pgmap v797: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 1506/108906 objects degraded (1.383%); 3654/108906 objects misplaced (3.355%); 36 MiB/s, 583 objects/s recovering 2024-03-26T03:16:03.749 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-mon[37920]: osdmap e383: 8 total, 8 up, 6 in 2024-03-26T03:16:03.749 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/473428107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:16:03.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:03.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:03.935 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:03.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:04.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:04.052 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:16:04.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:04.094 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:16:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:16:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:16:04.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:04.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:04.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:04.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:04.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:04.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:04.557 INFO:tasks.thrashosds.thrasher:j is {'epoch': 383, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:16:02.468809+0000', 'last_up_change': '2024-03-26T03:14:20.518826+0000', 'last_in_change': '2024-03-26T03:14:36.200040+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '305', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '303', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7999999523162842, 'score_stable': 1.7999999523162842, 'optimal_score': 0.75, 'raw_score_acting': 1.350000023841858, 'raw_score_stable': 1.350000023841858, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.289999961853027, 'score_stable': 5.289999961853027, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 3.3299999237060547, 'raw_score_stable': 3.3299999237060547, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.75, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-26T03:12:34.415377+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '296', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9059524536132812, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.9100000262260437}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 313, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 379, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6510009765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 197, 'up_from': 331, 'up_thru': 370, 'down_at': 198, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 52636497}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 52636497}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 52636497}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 52636497}]}, 'public_addr': '172.21.15.66:6813/52636497', 'cluster_addr': '172.21.15.66:6814/52636497', 'heartbeat_back_addr': '172.21.15.66:6816/52636497', 'heartbeat_front_addr': '172.21.15.66:6815/52636497', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 0.6559295654296875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 377, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 381, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T03:15:27.782296+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:15:48.254003+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:15:47.413461+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:15:46.313960+0000', 'dead_epoch': 197}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:15:38.724804+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:14:44.557445+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:15:04.768039+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T03:14:54.343043+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [4, 1, 7]}], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.7', 'osds': [6, 5]}, {'pgid': '10.c', 'osds': [4, 7, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:16:04.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:16:04.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:04.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:04.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:04 smithi192 ceph-mon[33110]: osdmap e384: 8 total, 8 up, 6 in 2024-03-26T03:16:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:04 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 1506/108906 objects degraded (1.383%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:04.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:04.840 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:04 smithi066 ceph-mon[29666]: osdmap e384: 8 total, 8 up, 6 in 2024-03-26T03:16:04.840 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:04 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 1506/108906 objects degraded (1.383%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:04.840 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:04 smithi066 ceph-mon[37920]: osdmap e384: 8 total, 8 up, 6 in 2024-03-26T03:16:04.840 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:04 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 1506/108906 objects degraded (1.383%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:04.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:04.914 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:04.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:05.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:05.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:05.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:05.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:05.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:16:05.254 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:05.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:05.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:05.352 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:05.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:05.484 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:05.586 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:05.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:05.736 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:05 smithi066 ceph-mon[29666]: pgmap v800: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 55 op/s; 1506/108237 objects degraded (1.391%); 3654/108237 objects misplaced (3.376%) 2024-03-26T03:16:05.736 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:05 smithi066 ceph-mon[37920]: pgmap v800: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 55 op/s; 1506/108237 objects degraded (1.391%); 3654/108237 objects misplaced (3.376%) 2024-03-26T03:16:05.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:05.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:05.900 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:05.965 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:05.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:05 smithi192 ceph-mon[33110]: pgmap v800: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 173 active+clean; 2.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 55 op/s; 1506/108237 objects degraded (1.391%); 3654/108237 objects misplaced (3.376%) 2024-03-26T03:16:06.067 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:06.110 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:06.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:06.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:06.349 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:06.393 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:06.495 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:06.535 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:06.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:06.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:06.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:06.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:06.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:06.952 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:06.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:07.054 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:07.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:07.202 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:07.235 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:07.337 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:07.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:07.493 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:07.536 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:07.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:07.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:07.775 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:07.825 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:07.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:07 smithi066 ceph-mon[29666]: pgmap v801: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 176 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 865 B/s rd, 0 B/s wr, 607 op/s; 1337/98139 objects degraded (1.362%); 2068/98139 objects misplaced (2.107%); 7.0 MiB/s, 112 objects/s recovering 2024-03-26T03:16:07.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:07 smithi066 ceph-mon[37920]: pgmap v801: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 176 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 865 B/s rd, 0 B/s wr, 607 op/s; 1337/98139 objects degraded (1.362%); 2068/98139 objects misplaced (2.107%); 7.0 MiB/s, 112 objects/s recovering 2024-03-26T03:16:07.927 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:07.969 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:07.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:07 smithi192 ceph-mon[33110]: pgmap v801: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 176 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 865 B/s rd, 0 B/s wr, 607 op/s; 1337/98139 objects degraded (1.362%); 2068/98139 objects misplaced (2.107%); 7.0 MiB/s, 112 objects/s recovering 2024-03-26T03:16:08.070 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:08.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:08.217 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:08.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:08.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:08.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:08.498 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.542 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:08.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:08.643 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:08.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:08.790 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:08.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:08.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:08.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:08.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:09.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:09.240 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:09.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:09.266 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:09.270 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:09.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:09.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:09.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:09.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:09.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:09.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:09.613 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:09.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:09.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:09.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:09.780 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:09 smithi066 ceph-mon[29666]: pgmap v802: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 176 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 739 B/s rd, 0 B/s wr, 519 op/s; 1337/98139 objects degraded (1.362%); 2068/98139 objects misplaced (2.107%); 6.0 MiB/s, 95 objects/s recovering 2024-03-26T03:16:09.781 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:09 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 1337/98139 objects degraded (1.362%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:09.781 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:09 smithi066 ceph-mon[37920]: pgmap v802: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 176 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 739 B/s rd, 0 B/s wr, 519 op/s; 1337/98139 objects degraded (1.362%); 2068/98139 objects misplaced (2.107%); 6.0 MiB/s, 95 objects/s recovering 2024-03-26T03:16:09.781 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:09 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 1337/98139 objects degraded (1.362%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:09.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:09.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:09.948 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:09.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:09.957 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:09 smithi192 ceph-mon[33110]: pgmap v802: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 176 active+clean; 2.0 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 739 B/s rd, 0 B/s wr, 519 op/s; 1337/98139 objects degraded (1.362%); 2068/98139 objects misplaced (2.107%); 6.0 MiB/s, 95 objects/s recovering 2024-03-26T03:16:09.957 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:09 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 1337/98139 objects degraded (1.362%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:10.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:10.094 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:10.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:10.195 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:10.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:16:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:16:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:10.278 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:10.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:10.382 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:10.427 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:10.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:10.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:10.587 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:10.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:10.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:10.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:10.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:10.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:10.810 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:10.823 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:10.865 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:10.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:10.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:11.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:11.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:11.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:11.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:11.242 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:11.268 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:16:11.313 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:16:11.330 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:11.362 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:16:11.410 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:16:11.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:11.458 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:16:11.473 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:11.498 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:16:11.529 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:16:11.542 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:16:11.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:11.585 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:16:11.589 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:16:11.589 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:16:11.589 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:16:11.590 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:16:11.590 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:16:11.590 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:16:11.590 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:16:11.590 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:16:11.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:11.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: pgmap v803: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 776/76458 objects degraded (1.015%); 1610/76458 objects misplaced (2.106%); 7.3 MiB/s, 117 objects/s recovering 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: osdmap e385: 8 total, 8 up, 6 in 2024-03-26T03:16:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1242476943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2609027161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: pgmap v803: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 776/76458 objects degraded (1.015%); 1610/76458 objects misplaced (2.106%); 7.3 MiB/s, 117 objects/s recovering 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: osdmap e385: 8 total, 8 up, 6 in 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1242476943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2609027161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:16:11.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:11.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:11.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:11.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: pgmap v803: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 776/76458 objects degraded (1.015%); 1610/76458 objects misplaced (2.106%); 7.3 MiB/s, 117 objects/s recovering 2024-03-26T03:16:11.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:16:11.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:16:11.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:16:11.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: osdmap e385: 8 total, 8 up, 6 in 2024-03-26T03:16:11.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1242476943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:11.890 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2609027161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:16:11.894 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:11.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:11.996 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:12.039 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:12.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:12.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:12.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:12.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:12.414 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.456 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:12.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:12.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:12.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:12.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:12.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:12.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:12 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:16:12.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2609027161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:16:12.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:12 smithi066 ceph-mon[29666]: osdmap e386: 8 total, 8 up, 6 in 2024-03-26T03:16:12.829 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:12 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:16:12.829 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2609027161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:16:12.829 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:12 smithi066 ceph-mon[37920]: osdmap e386: 8 total, 8 up, 6 in 2024-03-26T03:16:12.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:12.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:12.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:12.973 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:12 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:16:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2609027161' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:16:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:12 smithi192 ceph-mon[33110]: osdmap e386: 8 total, 8 up, 6 in 2024-03-26T03:16:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.017 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:13.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:13.118 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:13.163 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:13.215 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:16:13.221 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:16:13.265 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.316 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:13.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:13.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:13.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:13.556 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:13.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:13.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:13.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:13.796 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:13 smithi066 ceph-mon[29666]: pgmap v806: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 776/76458 objects degraded (1.015%); 1610/76458 objects misplaced (2.106%); 7.3 MiB/s, 117 objects/s recovering 2024-03-26T03:16:13.797 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:13 smithi066 ceph-mon[37920]: pgmap v806: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 776/76458 objects degraded (1.015%); 1610/76458 objects misplaced (2.106%); 7.3 MiB/s, 117 objects/s recovering 2024-03-26T03:16:13.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:13.873 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:13.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:13.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:13 smithi192 ceph-mon[33110]: pgmap v806: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 776/76458 objects degraded (1.015%); 1610/76458 objects misplaced (2.106%); 7.3 MiB/s, 117 objects/s recovering 2024-03-26T03:16:14.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:16:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:16:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:16:14.113 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:14.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:14.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:14.298 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:14.361 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-03-26T03:16:12.381372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1655987031,"num_objects":25486,"num_object_clones":0,"num_object_copies":76458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":1610,"num_objects_unfound":0,"num_objects_dirty":25486,"num_whiteouts":0,"num_read":2497,"num_read_kb":2242,"num_write":224992,"num_write_kb":7977865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90366,"num_bytes_recovered":5906620099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67060,"ondisk_log_size":67060,"up":534,"acting":533,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":540,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9998304,"kb_used_data":6177392,"kb_used_omap":12,"kb_used_meta":3820787,"kb_avail":552472608,"statfs":{"total":575970213888,"available":565731950592,"internally_reserved":0,"allocated":6325649408,"data_stored":6318567398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12419,"internal_metadata":3912486781},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-694222848,"num_objects":-10593,"num_object_clones":0,"num_object_copies":-31779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-730,"num_objects_misplaced":-2044,"num_objects_unfound":0,"num_objects_dirty":-10593,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10593,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":61538304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007166"},"pg_stats":[{"pgid":"10.6","version":"384'14002","reported_seq":26906,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.610393+0000","last_change":"2024-03-26T03:15:01.535150+0000","last_active":"2024-03-26T03:16:09.610393+0000","last_peered":"2024-03-26T03:16:09.610393+0000","last_clean":"2024-03-26T03:16:09.610393+0000","last_became_active":"2024-03-26T03:15:01.516632+0000","last_became_peered":"2024-03-26T03:15:01.516632+0000","last_unstale":"2024-03-26T03:16:09.610393+0000","last_undegraded":"2024-03-26T03:16:09.610393+0000","last_fullsized":"2024-03-26T03:16:09.610393+0000","mapping_epoch":351,"log_start":"345'8759","ondisk_log_start":"345'8759","created":289,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":5243,"log_dups_size":0,"ondisk_log_size":5243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105119744,"num_objects":1604,"num_object_clones":0,"num_object_copies":4812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14002,"num_write_kb":499392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6339,"num_bytes_recovered":415432704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":614,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488313+0000","last_change":"2024-03-26T03:14:40.174777+0000","last_active":"2024-03-26T03:16:03.488313+0000","last_peered":"2024-03-26T03:16:03.488313+0000","last_clean":"2024-03-26T03:16:03.488313+0000","last_became_active":"2024-03-26T03:14:40.174462+0000","last_became_peered":"2024-03-26T03:14:40.174462+0000","last_unstale":"2024-03-26T03:16:03.488313+0000","last_undegraded":"2024-03-26T03:16:03.488313+0000","last_fullsized":"2024-03-26T03:16:03.488313+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.054032+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.054032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:08.303128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"342'10","reported_seq":273,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512383+0000","last_change":"2024-03-26T03:14:37.309821+0000","last_active":"2024-03-26T03:16:03.512383+0000","last_peered":"2024-03-26T03:16:03.512383+0000","last_clean":"2024-03-26T03:16:03.512383+0000","last_became_active":"2024-03-26T03:14:37.273000+0000","last_became_peered":"2024-03-26T03:14:37.273000+0000","last_unstale":"2024-03-26T03:16:03.512383+0000","last_undegraded":"2024-03-26T03:16:03.512383+0000","last_fullsized":"2024-03-26T03:16:03.512383+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'9","last_scrub_stamp":"2024-03-26T03:13:34.936561+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:13:34.936561+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.862205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":818,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.491307+0000","last_change":"2024-03-26T03:14:37.321937+0000","last_active":"2024-03-26T03:16:03.491307+0000","last_peered":"2024-03-26T03:16:03.491307+0000","last_clean":"2024-03-26T03:16:03.491307+0000","last_became_active":"2024-03-26T03:14:37.267205+0000","last_became_peered":"2024-03-26T03:14:37.267205+0000","last_unstale":"2024-03-26T03:16:03.491307+0000","last_undegraded":"2024-03-26T03:16:03.491307+0000","last_fullsized":"2024-03-26T03:16:03.491307+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:13:25.646227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:25.646227+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":124,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.487259+0000","last_change":"2024-03-26T03:14:37.266094+0000","last_active":"2024-03-26T03:16:03.487259+0000","last_peered":"2024-03-26T03:16:03.487259+0000","last_clean":"2024-03-26T03:16:03.487259+0000","last_became_active":"2024-03-26T03:14:37.264771+0000","last_became_peered":"2024-03-26T03:14:37.264771+0000","last_unstale":"2024-03-26T03:16:03.487259+0000","last_undegraded":"2024-03-26T03:16:03.487259+0000","last_fullsized":"2024-03-26T03:16:03.487259+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:23.378694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:23.378694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":529,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512404+0000","last_change":"2024-03-26T03:13:35.270141+0000","last_active":"2024-03-26T03:16:03.512404+0000","last_peered":"2024-03-26T03:16:03.512404+0000","last_clean":"2024-03-26T03:16:03.512404+0000","last_became_active":"2024-03-26T03:13:35.269469+0000","last_became_peered":"2024-03-26T03:13:35.269469+0000","last_unstale":"2024-03-26T03:16:03.512404+0000","last_undegraded":"2024-03-26T03:16:03.512404+0000","last_fullsized":"2024-03-26T03:16:03.512404+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:33.324601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:33.324601+0000","last_clean_scrub_stamp":"2024-03-26T03:13:33.324601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918620+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"384'14160","reported_seq":32466,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:10.217845+0000","last_change":"2024-03-26T03:16:03.507485+0000","last_active":"2024-03-26T03:16:10.217845+0000","last_peered":"2024-03-26T03:16:10.217845+0000","last_clean":"2024-03-26T03:16:10.217845+0000","last_became_active":"2024-03-26T03:16:03.488606+0000","last_became_peered":"2024-03-26T03:16:03.488606+0000","last_unstale":"2024-03-26T03:16:10.217845+0000","last_undegraded":"2024-03-26T03:16:10.217845+0000","last_fullsized":"2024-03-26T03:16:10.217845+0000","mapping_epoch":383,"log_start":"358'10772","ondisk_log_start":"358'10772","created":289,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:13.273041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97648640,"num_objects":1490,"num_object_clones":0,"num_object_copies":4470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14160,"num_write_kb":500800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7744,"num_bytes_recovered":507510784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":373,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484267+0000","last_change":"2024-03-26T03:15:05.168111+0000","last_active":"2024-03-26T03:16:03.484267+0000","last_peered":"2024-03-26T03:16:03.484267+0000","last_clean":"2024-03-26T03:16:03.484267+0000","last_became_active":"2024-03-26T03:15:05.167703+0000","last_became_peered":"2024-03-26T03:15:05.167703+0000","last_unstale":"2024-03-26T03:16:03.484267+0000","last_undegraded":"2024-03-26T03:16:03.484267+0000","last_fullsized":"2024-03-26T03:16:03.484267+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:17.097284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.097284+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.782524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":594,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513640+0000","last_change":"2024-03-26T03:14:22.682587+0000","last_active":"2024-03-26T03:16:03.513640+0000","last_peered":"2024-03-26T03:16:03.513640+0000","last_clean":"2024-03-26T03:16:03.513640+0000","last_became_active":"2024-03-26T03:14:22.579044+0000","last_became_peered":"2024-03-26T03:14:22.579044+0000","last_unstale":"2024-03-26T03:16:03.513640+0000","last_undegraded":"2024-03-26T03:16:03.513640+0000","last_fullsized":"2024-03-26T03:16:03.513640+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:12:30.521522+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.521522+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":705,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514785+0000","last_change":"2024-03-26T03:15:16.933135+0000","last_active":"2024-03-26T03:16:03.514785+0000","last_peered":"2024-03-26T03:16:03.514785+0000","last_clean":"2024-03-26T03:16:03.514785+0000","last_became_active":"2024-03-26T03:15:02.534067+0000","last_became_peered":"2024-03-26T03:15:02.534067+0000","last_unstale":"2024-03-26T03:16:03.514785+0000","last_undegraded":"2024-03-26T03:16:03.514785+0000","last_fullsized":"2024-03-26T03:16:03.514785+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:16.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:15:16.933002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:40.265687+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":359,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484158+0000","last_change":"2024-03-26T03:15:36.802685+0000","last_active":"2024-03-26T03:16:03.484158+0000","last_peered":"2024-03-26T03:16:03.484158+0000","last_clean":"2024-03-26T03:16:03.484158+0000","last_became_active":"2024-03-26T03:15:36.802306+0000","last_became_peered":"2024-03-26T03:15:36.802306+0000","last_unstale":"2024-03-26T03:16:03.484158+0000","last_undegraded":"2024-03-26T03:16:03.484158+0000","last_fullsized":"2024-03-26T03:16:03.484158+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:36.837729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:36.837729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":142,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486251+0000","last_change":"2024-03-26T03:14:37.250277+0000","last_active":"2024-03-26T03:16:03.486251+0000","last_peered":"2024-03-26T03:16:03.486251+0000","last_clean":"2024-03-26T03:16:03.486251+0000","last_became_active":"2024-03-26T03:14:37.249782+0000","last_became_peered":"2024-03-26T03:14:37.249782+0000","last_unstale":"2024-03-26T03:16:03.486251+0000","last_undegraded":"2024-03-26T03:16:03.486251+0000","last_fullsized":"2024-03-26T03:16:03.486251+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:26.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:13:26.722546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"384'13771","reported_seq":32543,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.586396+0000","last_change":"2024-03-26T03:15:45.898414+0000","last_active":"2024-03-26T03:16:09.586396+0000","last_peered":"2024-03-26T03:16:09.586396+0000","last_clean":"2024-03-26T03:16:09.586396+0000","last_became_active":"2024-03-26T03:15:45.859028+0000","last_became_peered":"2024-03-26T03:15:45.859028+0000","last_unstale":"2024-03-26T03:16:09.586396+0000","last_undegraded":"2024-03-26T03:16:09.586396+0000","last_fullsized":"2024-03-26T03:16:09.586396+0000","mapping_epoch":371,"log_start":"345'8563","ondisk_log_start":"345'8563","created":289,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":5208,"log_dups_size":0,"ondisk_log_size":5208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13771,"num_write_kb":490944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9050,"num_bytes_recovered":593100800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":790,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513225+0000","last_change":"2024-03-26T03:12:40.141914+0000","last_active":"2024-03-26T03:16:03.513225+0000","last_peered":"2024-03-26T03:16:03.513225+0000","last_clean":"2024-03-26T03:16:03.513225+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:16:03.513225+0000","last_undegraded":"2024-03-26T03:16:03.513225+0000","last_fullsized":"2024-03-26T03:16:03.513225+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:40.141823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:40.141823+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":833,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512640+0000","last_change":"2024-03-26T03:14:37.326277+0000","last_active":"2024-03-26T03:16:03.512640+0000","last_peered":"2024-03-26T03:16:03.512640+0000","last_clean":"2024-03-26T03:16:03.512640+0000","last_became_active":"2024-03-26T03:14:37.266473+0000","last_became_peered":"2024-03-26T03:14:37.266473+0000","last_unstale":"2024-03-26T03:16:03.512640+0000","last_undegraded":"2024-03-26T03:16:03.512640+0000","last_fullsized":"2024-03-26T03:16:03.512640+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:13:18.294384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:18.294384+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":736,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483926+0000","last_change":"2024-03-26T03:13:35.267136+0000","last_active":"2024-03-26T03:16:03.483926+0000","last_peered":"2024-03-26T03:16:03.483926+0000","last_clean":"2024-03-26T03:16:03.483926+0000","last_became_active":"2024-03-26T03:13:35.266710+0000","last_became_peered":"2024-03-26T03:13:35.266710+0000","last_unstale":"2024-03-26T03:16:03.483926+0000","last_undegraded":"2024-03-26T03:16:03.483926+0000","last_fullsized":"2024-03-26T03:16:03.483926+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:22.061526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:13:22.061526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":726,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483686+0000","last_change":"2024-03-26T03:14:37.244560+0000","last_active":"2024-03-26T03:16:03.483686+0000","last_peered":"2024-03-26T03:16:03.483686+0000","last_clean":"2024-03-26T03:16:03.483686+0000","last_became_active":"2024-03-26T03:14:37.242217+0000","last_became_peered":"2024-03-26T03:14:37.242217+0000","last_unstale":"2024-03-26T03:16:03.483686+0000","last_undegraded":"2024-03-26T03:16:03.483686+0000","last_fullsized":"2024-03-26T03:16:03.483686+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:20.030941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:26.426122+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.030941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":654,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483485+0000","last_change":"2024-03-26T03:14:37.272004+0000","last_active":"2024-03-26T03:16:03.483485+0000","last_peered":"2024-03-26T03:16:03.483485+0000","last_clean":"2024-03-26T03:16:03.483485+0000","last_became_active":"2024-03-26T03:14:37.270129+0000","last_became_peered":"2024-03-26T03:14:37.270129+0000","last_unstale":"2024-03-26T03:16:03.483485+0000","last_undegraded":"2024-03-26T03:16:03.483485+0000","last_fullsized":"2024-03-26T03:16:03.483485+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:41.640503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.640503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"384'14198","reported_seq":22288,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.262427+0000","last_change":"2024-03-26T03:15:44.903780+0000","last_active":"2024-03-26T03:16:09.262427+0000","last_peered":"2024-03-26T03:16:09.262427+0000","last_clean":"2024-03-26T03:16:09.262427+0000","last_became_active":"2024-03-26T03:15:44.859013+0000","last_became_peered":"2024-03-26T03:15:44.859013+0000","last_unstale":"2024-03-26T03:16:09.262427+0000","last_undegraded":"2024-03-26T03:16:09.262427+0000","last_fullsized":"2024-03-26T03:16:09.262427+0000","mapping_epoch":370,"log_start":"358'11266","ondisk_log_start":"358'11266","created":289,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":2932,"log_dups_size":1642,"ondisk_log_size":2932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:09.865787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110886912,"num_objects":1692,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14198,"num_write_kb":508480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3936,"num_bytes_recovered":257949696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":523,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484758+0000","last_change":"2024-03-26T03:15:02.595953+0000","last_active":"2024-03-26T03:16:03.484758+0000","last_peered":"2024-03-26T03:16:03.484758+0000","last_clean":"2024-03-26T03:16:03.484758+0000","last_became_active":"2024-03-26T03:15:02.575671+0000","last_became_peered":"2024-03-26T03:15:02.575671+0000","last_unstale":"2024-03-26T03:16:03.484758+0000","last_undegraded":"2024-03-26T03:16:03.484758+0000","last_fullsized":"2024-03-26T03:16:03.484758+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:12:30.037403+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.037403+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":50,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489441+0000","last_change":"2024-03-26T03:15:36.789979+0000","last_active":"2024-03-26T03:16:03.489441+0000","last_peered":"2024-03-26T03:16:03.489441+0000","last_clean":"2024-03-26T03:16:03.489441+0000","last_became_active":"2024-03-26T03:15:36.789660+0000","last_became_peered":"2024-03-26T03:15:36.789660+0000","last_unstale":"2024-03-26T03:16:03.489441+0000","last_undegraded":"2024-03-26T03:16:03.489441+0000","last_fullsized":"2024-03-26T03:16:03.489441+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:41.872620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.872620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":704,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514444+0000","last_change":"2024-03-26T03:13:35.277539+0000","last_active":"2024-03-26T03:16:03.514444+0000","last_peered":"2024-03-26T03:16:03.514444+0000","last_clean":"2024-03-26T03:16:03.514444+0000","last_became_active":"2024-03-26T03:13:35.277164+0000","last_became_peered":"2024-03-26T03:13:35.277164+0000","last_unstale":"2024-03-26T03:16:03.514444+0000","last_undegraded":"2024-03-26T03:16:03.514444+0000","last_fullsized":"2024-03-26T03:16:03.514444+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:23.358630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.358630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":118,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485184+0000","last_change":"2024-03-26T03:14:21.588254+0000","last_active":"2024-03-26T03:16:03.485184+0000","last_peered":"2024-03-26T03:16:03.485184+0000","last_clean":"2024-03-26T03:16:03.485184+0000","last_became_active":"2024-03-26T03:14:21.586859+0000","last_became_peered":"2024-03-26T03:14:21.586859+0000","last_unstale":"2024-03-26T03:16:03.485184+0000","last_undegraded":"2024-03-26T03:16:03.485184+0000","last_fullsized":"2024-03-26T03:16:03.485184+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":412,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483069+0000","last_change":"2024-03-26T03:14:21.591046+0000","last_active":"2024-03-26T03:16:03.483069+0000","last_peered":"2024-03-26T03:16:03.483069+0000","last_clean":"2024-03-26T03:16:03.483069+0000","last_became_active":"2024-03-26T03:14:21.590629+0000","last_became_peered":"2024-03-26T03:14:21.590629+0000","last_unstale":"2024-03-26T03:16:03.483069+0000","last_undegraded":"2024-03-26T03:16:03.483069+0000","last_fullsized":"2024-03-26T03:16:03.483069+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.715072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.715072+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.782499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"384'14068","reported_seq":20533,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:10.151734+0000","last_change":"2024-03-26T03:15:59.209808+0000","last_active":"2024-03-26T03:16:10.151734+0000","last_peered":"2024-03-26T03:16:10.151734+0000","last_clean":"2024-03-26T03:16:10.151734+0000","last_became_active":"2024-03-26T03:15:59.185916+0000","last_became_peered":"2024-03-26T03:15:59.185916+0000","last_unstale":"2024-03-26T03:16:10.151734+0000","last_undegraded":"2024-03-26T03:16:10.151734+0000","last_fullsized":"2024-03-26T03:16:10.151734+0000","mapping_epoch":379,"log_start":"358'10841","ondisk_log_start":"358'10841","created":289,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100925440,"num_objects":1540,"num_object_clones":0,"num_object_copies":4620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14068,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3113,"num_bytes_recovered":204013568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":579,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484017+0000","last_change":"2024-03-26T03:14:40.169783+0000","last_active":"2024-03-26T03:16:03.484017+0000","last_peered":"2024-03-26T03:16:03.484017+0000","last_clean":"2024-03-26T03:16:03.484017+0000","last_became_active":"2024-03-26T03:14:40.169332+0000","last_became_peered":"2024-03-26T03:14:40.169332+0000","last_unstale":"2024-03-26T03:16:03.484017+0000","last_undegraded":"2024-03-26T03:16:03.484017+0000","last_fullsized":"2024-03-26T03:16:03.484017+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:21.705365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:14:21.705365+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.826154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":8,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":728,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488392+0000","last_change":"2024-03-26T03:14:21.700172+0000","last_active":"2024-03-26T03:16:03.488392+0000","last_peered":"2024-03-26T03:16:03.488392+0000","last_clean":"2024-03-26T03:16:03.488392+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:16:03.488392+0000","last_undegraded":"2024-03-26T03:16:03.488392+0000","last_fullsized":"2024-03-26T03:16:03.488392+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'10","last_scrub_stamp":"2024-03-26T03:06:13.978278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:06:13.978278+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":697,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513508+0000","last_change":"2024-03-26T03:15:09.812769+0000","last_active":"2024-03-26T03:16:03.513508+0000","last_peered":"2024-03-26T03:16:03.513508+0000","last_clean":"2024-03-26T03:16:03.513508+0000","last_became_active":"2024-03-26T03:14:37.251633+0000","last_became_peered":"2024-03-26T03:14:37.251633+0000","last_unstale":"2024-03-26T03:16:03.513508+0000","last_undegraded":"2024-03-26T03:16:03.513508+0000","last_fullsized":"2024-03-26T03:16:03.513508+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:09.812652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:15:09.812652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:17.999037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"342'10","reported_seq":940,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484005+0000","last_change":"2024-03-26T03:14:37.376007+0000","last_active":"2024-03-26T03:16:03.484005+0000","last_peered":"2024-03-26T03:16:03.484005+0000","last_clean":"2024-03-26T03:16:03.484005+0000","last_became_active":"2024-03-26T03:14:37.369407+0000","last_became_peered":"2024-03-26T03:14:37.369407+0000","last_unstale":"2024-03-26T03:16:03.484005+0000","last_undegraded":"2024-03-26T03:16:03.484005+0000","last_fullsized":"2024-03-26T03:16:03.484005+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:12:28.903038+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.903038+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":48,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.515183+0000","last_change":"2024-03-26T03:15:36.801327+0000","last_active":"2024-03-26T03:16:03.515183+0000","last_peered":"2024-03-26T03:16:03.515183+0000","last_clean":"2024-03-26T03:16:03.515183+0000","last_became_active":"2024-03-26T03:15:36.796119+0000","last_became_peered":"2024-03-26T03:15:36.796119+0000","last_unstale":"2024-03-26T03:16:03.515183+0000","last_undegraded":"2024-03-26T03:16:03.515183+0000","last_fullsized":"2024-03-26T03:16:03.515183+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:25.078426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:25.078426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"384'14217","reported_seq":30590,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.602144+0000","last_change":"2024-03-26T03:16:02.516851+0000","last_active":"2024-03-26T03:16:09.602144+0000","last_peered":"2024-03-26T03:16:09.602144+0000","last_clean":"2024-03-26T03:16:09.602144+0000","last_became_active":"2024-03-26T03:16:02.489444+0000","last_became_peered":"2024-03-26T03:16:02.489444+0000","last_unstale":"2024-03-26T03:16:09.602144+0000","last_undegraded":"2024-03-26T03:16:09.602144+0000","last_fullsized":"2024-03-26T03:16:09.602144+0000","mapping_epoch":382,"log_start":"346'9016","ondisk_log_start":"346'9016","created":289,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.276551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":4653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14217,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7616,"num_bytes_recovered":499122176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":809,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485117+0000","last_change":"2024-03-26T03:13:15.986954+0000","last_active":"2024-03-26T03:16:03.485117+0000","last_peered":"2024-03-26T03:16:03.485117+0000","last_clean":"2024-03-26T03:16:03.485117+0000","last_became_active":"2024-03-26T03:13:15.495827+0000","last_became_peered":"2024-03-26T03:13:15.495827+0000","last_unstale":"2024-03-26T03:16:03.485117+0000","last_undegraded":"2024-03-26T03:16:03.485117+0000","last_fullsized":"2024-03-26T03:16:03.485117+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:15.986791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.986791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":753,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.515015+0000","last_change":"2024-03-26T03:15:15.903248+0000","last_active":"2024-03-26T03:16:03.515015+0000","last_peered":"2024-03-26T03:16:03.515015+0000","last_clean":"2024-03-26T03:16:03.515015+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:16:03.515015+0000","last_undegraded":"2024-03-26T03:16:03.515015+0000","last_fullsized":"2024-03-26T03:16:03.515015+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'27","last_scrub_stamp":"2024-03-26T03:15:15.903078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:15:15.903078+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:22.533167+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":697,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514498+0000","last_change":"2024-03-26T03:14:39.896972+0000","last_active":"2024-03-26T03:16:03.514498+0000","last_peered":"2024-03-26T03:16:03.514498+0000","last_clean":"2024-03-26T03:16:03.514498+0000","last_became_active":"2024-03-26T03:14:37.249790+0000","last_became_peered":"2024-03-26T03:14:37.249790+0000","last_unstale":"2024-03-26T03:16:03.514498+0000","last_undegraded":"2024-03-26T03:16:03.514498+0000","last_fullsized":"2024-03-26T03:16:03.514498+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:39.896823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:14:39.896823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.863147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"353'10","reported_seq":836,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486844+0000","last_change":"2024-03-26T03:15:02.582374+0000","last_active":"2024-03-26T03:16:03.486844+0000","last_peered":"2024-03-26T03:16:03.486844+0000","last_clean":"2024-03-26T03:16:03.486844+0000","last_became_active":"2024-03-26T03:15:02.573407+0000","last_became_peered":"2024-03-26T03:15:02.573407+0000","last_unstale":"2024-03-26T03:16:03.486844+0000","last_undegraded":"2024-03-26T03:16:03.486844+0000","last_fullsized":"2024-03-26T03:16:03.486844+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:13:30.083400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:30.083400+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":513,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514320+0000","last_change":"2024-03-26T03:13:35.276632+0000","last_active":"2024-03-26T03:16:03.514320+0000","last_peered":"2024-03-26T03:16:03.514320+0000","last_clean":"2024-03-26T03:16:03.514320+0000","last_became_active":"2024-03-26T03:13:35.275396+0000","last_became_peered":"2024-03-26T03:13:35.275396+0000","last_unstale":"2024-03-26T03:16:03.514320+0000","last_undegraded":"2024-03-26T03:16:03.514320+0000","last_fullsized":"2024-03-26T03:16:03.514320+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:18.063340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:13:18.063340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862711+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"384'13982","reported_seq":23371,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.650865+0000","last_change":"2024-03-26T03:15:33.859689+0000","last_active":"2024-03-26T03:16:09.650865+0000","last_peered":"2024-03-26T03:16:09.650865+0000","last_clean":"2024-03-26T03:16:09.650865+0000","last_became_active":"2024-03-26T03:15:33.809325+0000","last_became_peered":"2024-03-26T03:15:33.809325+0000","last_unstale":"2024-03-26T03:16:09.650865+0000","last_undegraded":"2024-03-26T03:16:09.650865+0000","last_fullsized":"2024-03-26T03:16:09.650865+0000","mapping_epoch":363,"log_start":"358'10625","ondisk_log_start":"358'10625","created":289,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.861845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103940096,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13982,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4585,"num_bytes_recovered":300417024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":830,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513341+0000","last_change":"2024-03-26T03:13:38.315128+0000","last_active":"2024-03-26T03:16:03.513341+0000","last_peered":"2024-03-26T03:16:03.513341+0000","last_clean":"2024-03-26T03:16:03.513341+0000","last_became_active":"2024-03-26T03:13:38.314531+0000","last_became_peered":"2024-03-26T03:13:38.314531+0000","last_unstale":"2024-03-26T03:16:03.513341+0000","last_undegraded":"2024-03-26T03:16:03.513341+0000","last_fullsized":"2024-03-26T03:16:03.513341+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:15.894405+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.894405+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"291'31","reported_seq":757,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489811+0000","last_change":"2024-03-26T03:12:37.144959+0000","last_active":"2024-03-26T03:16:03.489811+0000","last_peered":"2024-03-26T03:16:03.489811+0000","last_clean":"2024-03-26T03:16:03.489811+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:16:03.489811+0000","last_undegraded":"2024-03-26T03:16:03.489811+0000","last_fullsized":"2024-03-26T03:16:03.489811+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:12:37.144848+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269078+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":666,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513131+0000","last_change":"2024-03-26T03:14:37.253086+0000","last_active":"2024-03-26T03:16:03.513131+0000","last_peered":"2024-03-26T03:16:03.513131+0000","last_clean":"2024-03-26T03:16:03.513131+0000","last_became_active":"2024-03-26T03:14:37.252745+0000","last_became_peered":"2024-03-26T03:14:37.252745+0000","last_unstale":"2024-03-26T03:16:03.513131+0000","last_undegraded":"2024-03-26T03:16:03.513131+0000","last_fullsized":"2024-03-26T03:16:03.513131+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:31.980008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.980008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.863086+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":510,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490637+0000","last_change":"2024-03-26T03:14:37.250209+0000","last_active":"2024-03-26T03:16:03.490637+0000","last_peered":"2024-03-26T03:16:03.490637+0000","last_clean":"2024-03-26T03:16:03.490637+0000","last_became_active":"2024-03-26T03:14:37.249741+0000","last_became_peered":"2024-03-26T03:14:37.249741+0000","last_unstale":"2024-03-26T03:16:03.490637+0000","last_undegraded":"2024-03-26T03:16:03.490637+0000","last_fullsized":"2024-03-26T03:16:03.490637+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.965732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.965732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":331,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485371+0000","last_change":"2024-03-26T03:14:21.591922+0000","last_active":"2024-03-26T03:16:03.485371+0000","last_peered":"2024-03-26T03:16:03.485371+0000","last_clean":"2024-03-26T03:16:03.485371+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:16:03.485371+0000","last_undegraded":"2024-03-26T03:16:03.485371+0000","last_fullsized":"2024-03-26T03:16:03.485371+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:35.720776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:04:35.720776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"384'14080","reported_seq":27642,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.652979+0000","last_change":"2024-03-26T03:15:04.183518+0000","last_active":"2024-03-26T03:16:09.652979+0000","last_peered":"2024-03-26T03:16:09.652979+0000","last_clean":"2024-03-26T03:16:09.652979+0000","last_became_active":"2024-03-26T03:15:04.159900+0000","last_became_peered":"2024-03-26T03:15:04.159900+0000","last_unstale":"2024-03-26T03:16:09.652979+0000","last_undegraded":"2024-03-26T03:16:09.652979+0000","last_fullsized":"2024-03-26T03:16:09.652979+0000","mapping_epoch":354,"log_start":"358'10773","ondisk_log_start":"358'10773","created":289,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.862076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":1602,"num_object_clones":0,"num_object_copies":4806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14080,"num_write_kb":501824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6674,"num_bytes_recovered":437387264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":479,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483137+0000","last_change":"2024-03-26T03:14:21.586686+0000","last_active":"2024-03-26T03:16:03.483137+0000","last_peered":"2024-03-26T03:16:03.483137+0000","last_clean":"2024-03-26T03:16:03.483137+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:16:03.483137+0000","last_undegraded":"2024-03-26T03:16:03.483137+0000","last_fullsized":"2024-03-26T03:16:03.483137+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-26T03:05:15.906148+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.906148+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:09.863267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":763,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489451+0000","last_change":"2024-03-26T03:12:22.041053+0000","last_active":"2024-03-26T03:16:03.489451+0000","last_peered":"2024-03-26T03:16:03.489451+0000","last_clean":"2024-03-26T03:16:03.489451+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:16:03.489451+0000","last_undegraded":"2024-03-26T03:16:03.489451+0000","last_fullsized":"2024-03-26T03:16:03.489451+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:12:22.040942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.040942+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513564+0000","last_change":"2024-03-26T03:14:37.276400+0000","last_active":"2024-03-26T03:16:03.513564+0000","last_peered":"2024-03-26T03:16:03.513564+0000","last_clean":"2024-03-26T03:16:03.513564+0000","last_became_active":"2024-03-26T03:14:37.276008+0000","last_became_peered":"2024-03-26T03:14:37.276008+0000","last_unstale":"2024-03-26T03:16:03.513564+0000","last_undegraded":"2024-03-26T03:16:03.513564+0000","last_fullsized":"2024-03-26T03:16:03.513564+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:32.227383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:12:32.227383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":96,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513174+0000","last_change":"2024-03-26T03:14:37.249283+0000","last_active":"2024-03-26T03:16:03.513174+0000","last_peered":"2024-03-26T03:16:03.513174+0000","last_clean":"2024-03-26T03:16:03.513174+0000","last_became_active":"2024-03-26T03:14:37.248955+0000","last_became_peered":"2024-03-26T03:14:37.248955+0000","last_unstale":"2024-03-26T03:16:03.513174+0000","last_undegraded":"2024-03-26T03:16:03.513174+0000","last_fullsized":"2024-03-26T03:16:03.513174+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:11:49.921407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":501,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486426+0000","last_change":"2024-03-26T03:14:40.166895+0000","last_active":"2024-03-26T03:16:03.486426+0000","last_peered":"2024-03-26T03:16:03.486426+0000","last_clean":"2024-03-26T03:16:03.486426+0000","last_became_active":"2024-03-26T03:14:40.166583+0000","last_became_peered":"2024-03-26T03:14:40.166583+0000","last_unstale":"2024-03-26T03:16:03.486426+0000","last_undegraded":"2024-03-26T03:16:03.486426+0000","last_fullsized":"2024-03-26T03:16:03.486426+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:06:16.123772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:06:16.123772+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.826133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"384'14097","reported_seq":14297,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.608788+0000","last_change":"2024-03-26T03:12:36.109906+0000","last_active":"2024-03-26T03:16:09.608788+0000","last_peered":"2024-03-26T03:16:09.608788+0000","last_clean":"2024-03-26T03:16:09.608788+0000","last_became_active":"2024-03-26T03:12:36.109323+0000","last_became_peered":"2024-03-26T03:12:36.109323+0000","last_unstale":"2024-03-26T03:16:09.608788+0000","last_undegraded":"2024-03-26T03:16:09.608788+0000","last_fullsized":"2024-03-26T03:16:09.608788+0000","mapping_epoch":289,"log_start":"346'8919","ondisk_log_start":"346'8919","created":289,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104660992,"num_objects":1597,"num_object_clones":0,"num_object_copies":4791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14097,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":680,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489641+0000","last_change":"2024-03-26T03:14:37.261571+0000","last_active":"2024-03-26T03:16:03.489641+0000","last_peered":"2024-03-26T03:16:03.489641+0000","last_clean":"2024-03-26T03:16:03.489641+0000","last_became_active":"2024-03-26T03:14:37.257004+0000","last_became_peered":"2024-03-26T03:14:37.257004+0000","last_unstale":"2024-03-26T03:16:03.489641+0000","last_undegraded":"2024-03-26T03:16:03.489641+0000","last_fullsized":"2024-03-26T03:16:03.489641+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.889978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.889978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":495,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483605+0000","last_change":"2024-03-26T03:14:37.343688+0000","last_active":"2024-03-26T03:16:03.483605+0000","last_peered":"2024-03-26T03:16:03.483605+0000","last_clean":"2024-03-26T03:16:03.483605+0000","last_became_active":"2024-03-26T03:14:37.278227+0000","last_became_peered":"2024-03-26T03:14:37.278227+0000","last_unstale":"2024-03-26T03:16:03.483605+0000","last_undegraded":"2024-03-26T03:16:03.483605+0000","last_fullsized":"2024-03-26T03:16:03.483605+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'42","last_scrub_stamp":"2024-03-26T03:01:36.768795+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:01:36.768795+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514865+0000","last_change":"2024-03-26T03:13:29.030426+0000","last_active":"2024-03-26T03:16:03.514865+0000","last_peered":"2024-03-26T03:16:03.514865+0000","last_clean":"2024-03-26T03:16:03.514865+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:16:03.514865+0000","last_undegraded":"2024-03-26T03:16:03.514865+0000","last_fullsized":"2024-03-26T03:16:03.514865+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:13:29.030298+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.030298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862810+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":630,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485375+0000","last_change":"2024-03-26T03:15:02.578357+0000","last_active":"2024-03-26T03:16:03.485375+0000","last_peered":"2024-03-26T03:16:03.485375+0000","last_clean":"2024-03-26T03:16:03.485375+0000","last_became_active":"2024-03-26T03:15:02.577864+0000","last_became_peered":"2024-03-26T03:15:02.577864+0000","last_unstale":"2024-03-26T03:16:03.485375+0000","last_undegraded":"2024-03-26T03:16:03.485375+0000","last_fullsized":"2024-03-26T03:16:03.485375+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.691223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.691223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":813,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488412+0000","last_change":"2024-03-26T03:13:16.068275+0000","last_active":"2024-03-26T03:16:03.488412+0000","last_peered":"2024-03-26T03:16:03.488412+0000","last_clean":"2024-03-26T03:16:03.488412+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:16:03.488412+0000","last_undegraded":"2024-03-26T03:16:03.488412+0000","last_fullsized":"2024-03-26T03:16:03.488412+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.068035+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.068035+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"384'13996","reported_seq":23187,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.621129+0000","last_change":"2024-03-26T03:15:46.888311+0000","last_active":"2024-03-26T03:16:09.621129+0000","last_peered":"2024-03-26T03:16:09.621129+0000","last_clean":"2024-03-26T03:16:09.621129+0000","last_became_active":"2024-03-26T03:15:46.858625+0000","last_became_peered":"2024-03-26T03:15:46.858625+0000","last_unstale":"2024-03-26T03:16:09.621129+0000","last_undegraded":"2024-03-26T03:16:09.621129+0000","last_fullsized":"2024-03-26T03:16:09.621129+0000","mapping_epoch":372,"log_start":"358'10695","ondisk_log_start":"358'10695","created":289,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3301,"log_dups_size":0,"ondisk_log_size":3301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.860064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219224,"num_objects":1576,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13996,"num_write_kb":498241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4477,"num_bytes_recovered":293339160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":713,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512890+0000","last_change":"2024-03-26T03:13:35.270135+0000","last_active":"2024-03-26T03:16:03.512890+0000","last_peered":"2024-03-26T03:16:03.512890+0000","last_clean":"2024-03-26T03:16:03.512890+0000","last_became_active":"2024-03-26T03:13:35.269721+0000","last_became_peered":"2024-03-26T03:13:35.269721+0000","last_unstale":"2024-03-26T03:16:03.512890+0000","last_undegraded":"2024-03-26T03:16:03.512890+0000","last_fullsized":"2024-03-26T03:16:03.512890+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.211706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.211706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":645,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489560+0000","last_change":"2024-03-26T03:14:37.311935+0000","last_active":"2024-03-26T03:16:03.489560+0000","last_peered":"2024-03-26T03:16:03.489560+0000","last_clean":"2024-03-26T03:16:03.489560+0000","last_became_active":"2024-03-26T03:14:37.258677+0000","last_became_peered":"2024-03-26T03:14:37.258677+0000","last_unstale":"2024-03-26T03:16:03.489560+0000","last_undegraded":"2024-03-26T03:16:03.489560+0000","last_fullsized":"2024-03-26T03:16:03.489560+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:13:31.527048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.527048+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633725+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":651,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488536+0000","last_change":"2024-03-26T03:14:37.257762+0000","last_active":"2024-03-26T03:16:03.488536+0000","last_peered":"2024-03-26T03:16:03.488536+0000","last_clean":"2024-03-26T03:16:03.488536+0000","last_became_active":"2024-03-26T03:14:37.257422+0000","last_became_peered":"2024-03-26T03:14:37.257422+0000","last_unstale":"2024-03-26T03:16:03.488536+0000","last_undegraded":"2024-03-26T03:16:03.488536+0000","last_fullsized":"2024-03-26T03:16:03.488536+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:35.954172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:35.954172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269147+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":706,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514622+0000","last_change":"2024-03-26T03:13:24.055056+0000","last_active":"2024-03-26T03:16:03.514622+0000","last_peered":"2024-03-26T03:16:03.514622+0000","last_clean":"2024-03-26T03:16:03.514622+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:16:03.514622+0000","last_undegraded":"2024-03-26T03:16:03.514622+0000","last_fullsized":"2024-03-26T03:16:03.514622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:24.054721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:13:24.054721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862776+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":783,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513207+0000","last_change":"2024-03-26T03:14:22.570585+0000","last_active":"2024-03-26T03:16:03.513207+0000","last_peered":"2024-03-26T03:16:03.513207+0000","last_clean":"2024-03-26T03:16:03.513207+0000","last_became_active":"2024-03-26T03:14:22.569851+0000","last_became_peered":"2024-03-26T03:14:22.569851+0000","last_unstale":"2024-03-26T03:16:03.513207+0000","last_undegraded":"2024-03-26T03:16:03.513207+0000","last_fullsized":"2024-03-26T03:16:03.513207+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:43.811912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:43.811912+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.942225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"384'14074","reported_seq":38510,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:10.218785+0000","last_change":"2024-03-26T03:15:22.511038+0000","last_active":"2024-03-26T03:16:10.218785+0000","last_peered":"2024-03-26T03:16:10.218785+0000","last_clean":"2024-03-26T03:16:10.218785+0000","last_became_active":"2024-03-26T03:15:22.486543+0000","last_became_peered":"2024-03-26T03:15:22.486543+0000","last_unstale":"2024-03-26T03:16:10.218785+0000","last_undegraded":"2024-03-26T03:16:10.218785+0000","last_fullsized":"2024-03-26T03:16:10.218785+0000","mapping_epoch":359,"log_start":"358'10679","ondisk_log_start":"358'10679","created":289,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3395,"log_dups_size":0,"ondisk_log_size":3395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:13.272960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100139008,"num_objects":1528,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14074,"num_write_kb":499264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12101,"num_bytes_recovered":793051136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":727,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.491017+0000","last_change":"2024-03-26T03:14:36.687488+0000","last_active":"2024-03-26T03:16:03.491017+0000","last_peered":"2024-03-26T03:16:03.491017+0000","last_clean":"2024-03-26T03:16:03.491017+0000","last_became_active":"2024-03-26T03:13:35.271824+0000","last_became_peered":"2024-03-26T03:13:35.271824+0000","last_unstale":"2024-03-26T03:16:03.491017+0000","last_undegraded":"2024-03-26T03:16:03.491017+0000","last_fullsized":"2024-03-26T03:16:03.491017+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.687375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.687375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":610,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.515064+0000","last_change":"2024-03-26T03:14:37.277935+0000","last_active":"2024-03-26T03:16:03.515064+0000","last_peered":"2024-03-26T03:16:03.515064+0000","last_clean":"2024-03-26T03:16:03.515064+0000","last_became_active":"2024-03-26T03:14:37.246803+0000","last_became_peered":"2024-03-26T03:14:37.246803+0000","last_unstale":"2024-03-26T03:16:03.515064+0000","last_undegraded":"2024-03-26T03:16:03.515064+0000","last_fullsized":"2024-03-26T03:16:03.515064+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:13:15.467975+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.467975+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862918+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":632,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513236+0000","last_change":"2024-03-26T03:14:37.266069+0000","last_active":"2024-03-26T03:16:03.513236+0000","last_peered":"2024-03-26T03:16:03.513236+0000","last_clean":"2024-03-26T03:16:03.513236+0000","last_became_active":"2024-03-26T03:14:37.265520+0000","last_became_peered":"2024-03-26T03:14:37.265520+0000","last_unstale":"2024-03-26T03:16:03.513236+0000","last_undegraded":"2024-03-26T03:16:03.513236+0000","last_fullsized":"2024-03-26T03:16:03.513236+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.461667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.461667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":431,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483476+0000","last_change":"2024-03-26T03:15:02.570929+0000","last_active":"2024-03-26T03:16:03.483476+0000","last_peered":"2024-03-26T03:16:03.483476+0000","last_clean":"2024-03-26T03:16:03.483476+0000","last_became_active":"2024-03-26T03:15:02.570488+0000","last_became_peered":"2024-03-26T03:15:02.570488+0000","last_unstale":"2024-03-26T03:16:03.483476+0000","last_undegraded":"2024-03-26T03:16:03.483476+0000","last_fullsized":"2024-03-26T03:16:03.483476+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":688,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514542+0000","last_change":"2024-03-26T03:15:17.827310+0000","last_active":"2024-03-26T03:16:03.514542+0000","last_peered":"2024-03-26T03:16:03.514542+0000","last_clean":"2024-03-26T03:16:03.514542+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:16:03.514542+0000","last_undegraded":"2024-03-26T03:16:03.514542+0000","last_fullsized":"2024-03-26T03:16:03.514542+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:17.827186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:15:17.827186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:25.879914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":810,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513444+0000","last_change":"2024-03-26T03:15:39.816877+0000","last_active":"2024-03-26T03:16:03.513444+0000","last_peered":"2024-03-26T03:16:03.513444+0000","last_clean":"2024-03-26T03:16:03.513444+0000","last_became_active":"2024-03-26T03:15:39.816407+0000","last_became_peered":"2024-03-26T03:15:39.816407+0000","last_unstale":"2024-03-26T03:16:03.513444+0000","last_undegraded":"2024-03-26T03:16:03.513444+0000","last_fullsized":"2024-03-26T03:16:03.513444+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:29.489954+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:29.489954+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"384'14223","reported_seq":33600,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.643450+0000","last_change":"2024-03-26T03:16:01.487006+0000","last_active":"2024-03-26T03:16:09.643450+0000","last_peered":"2024-03-26T03:16:09.643450+0000","last_clean":"2024-03-26T03:16:09.643450+0000","last_became_active":"2024-03-26T03:16:01.468608+0000","last_became_peered":"2024-03-26T03:16:01.468608+0000","last_unstale":"2024-03-26T03:16:09.643450+0000","last_undegraded":"2024-03-26T03:16:09.643450+0000","last_fullsized":"2024-03-26T03:16:09.643450+0000","mapping_epoch":381,"log_start":"358'10864","ondisk_log_start":"358'10864","created":289,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3359,"log_dups_size":0,"ondisk_log_size":3359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109510656,"num_objects":1671,"num_object_clones":0,"num_object_copies":5013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14223,"num_write_kb":508608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9574,"num_bytes_recovered":627441664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":96,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483701+0000","last_change":"2024-03-26T03:14:37.235429+0000","last_active":"2024-03-26T03:16:03.483701+0000","last_peered":"2024-03-26T03:16:03.483701+0000","last_clean":"2024-03-26T03:16:03.483701+0000","last_became_active":"2024-03-26T03:14:37.234721+0000","last_became_peered":"2024-03-26T03:14:37.234721+0000","last_unstale":"2024-03-26T03:16:03.483701+0000","last_undegraded":"2024-03-26T03:16:03.483701+0000","last_fullsized":"2024-03-26T03:16:03.483701+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:47.792700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:47.792700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"342'29","reported_seq":982,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513528+0000","last_change":"2024-03-26T03:14:37.320989+0000","last_active":"2024-03-26T03:16:03.513528+0000","last_peered":"2024-03-26T03:16:03.513528+0000","last_clean":"2024-03-26T03:16:03.513528+0000","last_became_active":"2024-03-26T03:14:37.265178+0000","last_became_peered":"2024-03-26T03:14:37.265178+0000","last_unstale":"2024-03-26T03:16:03.513528+0000","last_undegraded":"2024-03-26T03:16:03.513528+0000","last_fullsized":"2024-03-26T03:16:03.513528+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'28","last_scrub_stamp":"2024-03-26T03:12:20.180126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:12:20.180126+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.860099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":391,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512292+0000","last_change":"2024-03-26T03:14:40.181184+0000","last_active":"2024-03-26T03:16:03.512292+0000","last_peered":"2024-03-26T03:16:03.512292+0000","last_clean":"2024-03-26T03:16:03.512292+0000","last_became_active":"2024-03-26T03:14:40.180729+0000","last_became_peered":"2024-03-26T03:14:40.180729+0000","last_unstale":"2024-03-26T03:16:03.512292+0000","last_undegraded":"2024-03-26T03:16:03.512292+0000","last_fullsized":"2024-03-26T03:16:03.512292+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:23.126521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:23.126521+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.942361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":742,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489535+0000","last_change":"2024-03-26T03:14:37.275488+0000","last_active":"2024-03-26T03:16:03.489535+0000","last_peered":"2024-03-26T03:16:03.489535+0000","last_clean":"2024-03-26T03:16:03.489535+0000","last_became_active":"2024-03-26T03:14:37.274568+0000","last_became_peered":"2024-03-26T03:14:37.274568+0000","last_unstale":"2024-03-26T03:16:03.489535+0000","last_undegraded":"2024-03-26T03:16:03.489535+0000","last_fullsized":"2024-03-26T03:16:03.489535+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:24.846895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:12:24.846895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":393,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485272+0000","last_change":"2024-03-26T03:14:21.657126+0000","last_active":"2024-03-26T03:16:03.485272+0000","last_peered":"2024-03-26T03:16:03.485272+0000","last_clean":"2024-03-26T03:16:03.485272+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:16:03.485272+0000","last_undegraded":"2024-03-26T03:16:03.485272+0000","last_fullsized":"2024-03-26T03:16:03.485272+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":686,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.491106+0000","last_change":"2024-03-26T03:14:26.812522+0000","last_active":"2024-03-26T03:16:03.491106+0000","last_peered":"2024-03-26T03:16:03.491106+0000","last_clean":"2024-03-26T03:16:03.491106+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:16:03.491106+0000","last_undegraded":"2024-03-26T03:16:03.491106+0000","last_fullsized":"2024-03-26T03:16:03.491106+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:26.812402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:26.812402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633951+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"384'13918","reported_seq":20409,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.713135+0000","last_change":"2024-03-26T03:15:54.220759+0000","last_active":"2024-03-26T03:16:09.713135+0000","last_peered":"2024-03-26T03:16:09.713135+0000","last_clean":"2024-03-26T03:16:09.713135+0000","last_became_active":"2024-03-26T03:15:54.194397+0000","last_became_peered":"2024-03-26T03:15:54.194397+0000","last_unstale":"2024-03-26T03:16:09.713135+0000","last_undegraded":"2024-03-26T03:16:09.713135+0000","last_fullsized":"2024-03-26T03:16:09.713135+0000","mapping_epoch":377,"log_start":"358'10925","ondisk_log_start":"358'10925","created":289,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7742","last_scrub_stamp":"2024-03-26T03:14:22.245901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:14:22.245901+0000","objects_scrubbed":7742,"log_size":2993,"log_dups_size":1740,"ondisk_log_size":2993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:13.526271+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104071168,"num_objects":1588,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13918,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3091,"num_bytes_recovered":202571776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.487036+0000","last_change":"2024-03-26T03:14:37.239265+0000","last_active":"2024-03-26T03:16:03.487036+0000","last_peered":"2024-03-26T03:16:03.487036+0000","last_clean":"2024-03-26T03:16:03.487036+0000","last_became_active":"2024-03-26T03:14:37.238907+0000","last_became_peered":"2024-03-26T03:14:37.238907+0000","last_unstale":"2024-03-26T03:16:03.487036+0000","last_undegraded":"2024-03-26T03:16:03.487036+0000","last_fullsized":"2024-03-26T03:16:03.487036+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:16.992128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.992128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":795,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.515098+0000","last_change":"2024-03-26T03:14:37.294831+0000","last_active":"2024-03-26T03:16:03.515098+0000","last_peered":"2024-03-26T03:16:03.515098+0000","last_clean":"2024-03-26T03:16:03.515098+0000","last_became_active":"2024-03-26T03:14:37.272426+0000","last_became_peered":"2024-03-26T03:14:37.272426+0000","last_unstale":"2024-03-26T03:16:03.515098+0000","last_undegraded":"2024-03-26T03:16:03.515098+0000","last_fullsized":"2024-03-26T03:16:03.515098+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:13:23.013434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:23.013434+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.863022+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":696,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486357+0000","last_change":"2024-03-26T03:14:22.581508+0000","last_active":"2024-03-26T03:16:03.486357+0000","last_peered":"2024-03-26T03:16:03.486357+0000","last_clean":"2024-03-26T03:16:03.486357+0000","last_became_active":"2024-03-26T03:14:22.580051+0000","last_became_peered":"2024-03-26T03:14:22.580051+0000","last_unstale":"2024-03-26T03:16:03.486357+0000","last_undegraded":"2024-03-26T03:16:03.486357+0000","last_fullsized":"2024-03-26T03:16:03.486357+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.456991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.456991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":1755,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:08.662416+0000","last_change":"2024-03-26T03:13:31.075605+0000","last_active":"2024-03-26T03:16:08.662416+0000","last_peered":"2024-03-26T03:16:08.662416+0000","last_clean":"2024-03-26T03:16:08.662416+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:16:08.662416+0000","last_undegraded":"2024-03-26T03:16:08.662416+0000","last_fullsized":"2024-03-26T03:16:08.662416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:13:31.074419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.074419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1022,"num_read_kb":1017,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":662,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513330+0000","last_change":"2024-03-26T03:14:37.374055+0000","last_active":"2024-03-26T03:16:03.513330+0000","last_peered":"2024-03-26T03:16:03.513330+0000","last_clean":"2024-03-26T03:16:03.513330+0000","last_became_active":"2024-03-26T03:14:37.373684+0000","last_became_peered":"2024-03-26T03:14:37.373684+0000","last_unstale":"2024-03-26T03:16:03.513330+0000","last_undegraded":"2024-03-26T03:16:03.513330+0000","last_fullsized":"2024-03-26T03:16:03.513330+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:43.562063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.562063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917043+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":917,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485350+0000","last_change":"2024-03-26T03:15:05.189723+0000","last_active":"2024-03-26T03:16:03.485350+0000","last_peered":"2024-03-26T03:16:03.485350+0000","last_clean":"2024-03-26T03:16:03.485350+0000","last_became_active":"2024-03-26T03:15:05.189328+0000","last_became_peered":"2024-03-26T03:15:05.189328+0000","last_unstale":"2024-03-26T03:16:03.485350+0000","last_undegraded":"2024-03-26T03:16:03.485350+0000","last_fullsized":"2024-03-26T03:16:03.485350+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:15.093551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.093551+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"384'13726","reported_seq":26614,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:09.671081+0000","last_change":"2024-03-26T03:14:23.571704+0000","last_active":"2024-03-26T03:16:09.671081+0000","last_peered":"2024-03-26T03:16:09.671081+0000","last_clean":"2024-03-26T03:16:09.671081+0000","last_became_active":"2024-03-26T03:14:23.562719+0000","last_became_peered":"2024-03-26T03:14:23.562719+0000","last_unstale":"2024-03-26T03:16:09.671081+0000","last_undegraded":"2024-03-26T03:16:09.671081+0000","last_fullsized":"2024-03-26T03:16:09.671081+0000","mapping_epoch":333,"log_start":"358'10373","ondisk_log_start":"358'10373","created":289,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.862016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":4686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13726,"num_write_kb":489216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6335,"num_bytes_recovered":415170560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":159,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514512+0000","last_change":"2024-03-26T03:15:02.574062+0000","last_active":"2024-03-26T03:16:03.514512+0000","last_peered":"2024-03-26T03:16:03.514512+0000","last_clean":"2024-03-26T03:16:03.514512+0000","last_became_active":"2024-03-26T03:15:02.572307+0000","last_became_peered":"2024-03-26T03:15:02.572307+0000","last_unstale":"2024-03-26T03:16:03.514512+0000","last_undegraded":"2024-03-26T03:16:03.514512+0000","last_fullsized":"2024-03-26T03:16:03.514512+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:17.442291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.442291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"332'59","reported_seq":1008,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514184+0000","last_change":"2024-03-26T03:15:12.789448+0000","last_active":"2024-03-26T03:16:03.514184+0000","last_peered":"2024-03-26T03:16:03.514184+0000","last_clean":"2024-03-26T03:16:03.514184+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:16:03.514184+0000","last_undegraded":"2024-03-26T03:16:03.514184+0000","last_fullsized":"2024-03-26T03:16:03.514184+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'59","last_scrub_stamp":"2024-03-26T03:15:12.789148+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:15:12.789148+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:28.893144+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":507,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513255+0000","last_change":"2024-03-26T03:12:30.465375+0000","last_active":"2024-03-26T03:16:03.513255+0000","last_peered":"2024-03-26T03:16:03.513255+0000","last_clean":"2024-03-26T03:16:03.513255+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:16:03.513255+0000","last_undegraded":"2024-03-26T03:16:03.513255+0000","last_fullsized":"2024-03-26T03:16:03.513255+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:30.465292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.465292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862403+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":405,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.487007+0000","last_change":"2024-03-26T03:14:37.276021+0000","last_active":"2024-03-26T03:16:03.487007+0000","last_peered":"2024-03-26T03:16:03.487007+0000","last_clean":"2024-03-26T03:16:03.487007+0000","last_became_active":"2024-03-26T03:14:37.274135+0000","last_became_peered":"2024-03-26T03:14:37.274135+0000","last_unstale":"2024-03-26T03:16:03.487007+0000","last_undegraded":"2024-03-26T03:16:03.487007+0000","last_fullsized":"2024-03-26T03:16:03.487007+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":432,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489180+0000","last_change":"2024-03-26T03:13:56.450297+0000","last_active":"2024-03-26T03:16:03.489180+0000","last_peered":"2024-03-26T03:16:03.489180+0000","last_clean":"2024-03-26T03:16:03.489180+0000","last_became_active":"2024-03-26T03:13:56.449984+0000","last_became_peered":"2024-03-26T03:13:56.449984+0000","last_unstale":"2024-03-26T03:16:03.489180+0000","last_undegraded":"2024-03-26T03:16:03.489180+0000","last_fullsized":"2024-03-26T03:16:03.489180+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:21.520759+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:12:21.520759+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"384'13745","reported_seq":22735,"reported_epoch":384,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T03:16:09.629375+0000","last_change":"2024-03-26T03:15:39.889656+0000","last_active":"2024-03-26T03:16:09.629375+0000","last_peered":"2024-03-26T03:16:09.629375+0000","last_clean":"2024-03-26T03:14:20.504480+0000","last_became_active":"2024-03-26T03:14:38.307615+0000","last_became_peered":"2024-03-26T03:14:38.307615+0000","last_unstale":"2024-03-26T03:16:09.629375+0000","last_undegraded":"2024-03-26T03:16:09.629375+0000","last_fullsized":"2024-03-26T03:16:09.629375+0000","mapping_epoch":342,"log_start":"346'8532","ondisk_log_start":"346'8532","created":289,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103350272,"num_objects":1577,"num_object_clones":0,"num_object_copies":4731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1610,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13745,"num_write_kb":490304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2920,"num_bytes_recovered":191299584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":1577}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"333'12","reported_seq":222,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513119+0000","last_change":"2024-03-26T03:14:22.595140+0000","last_active":"2024-03-26T03:16:03.513119+0000","last_peered":"2024-03-26T03:16:03.513119+0000","last_clean":"2024-03-26T03:16:03.513119+0000","last_became_active":"2024-03-26T03:14:22.574243+0000","last_became_peered":"2024-03-26T03:14:22.574243+0000","last_unstale":"2024-03-26T03:16:03.513119+0000","last_undegraded":"2024-03-26T03:16:03.513119+0000","last_fullsized":"2024-03-26T03:16:03.513119+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:12:39.462209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:39.462209+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.862110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":774,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513840+0000","last_change":"2024-03-26T03:15:36.813531+0000","last_active":"2024-03-26T03:16:03.513840+0000","last_peered":"2024-03-26T03:16:03.513840+0000","last_clean":"2024-03-26T03:16:03.513840+0000","last_became_active":"2024-03-26T03:15:36.801545+0000","last_became_peered":"2024-03-26T03:15:36.801545+0000","last_unstale":"2024-03-26T03:16:03.513840+0000","last_undegraded":"2024-03-26T03:16:03.513840+0000","last_fullsized":"2024-03-26T03:16:03.513840+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:12:28.539083+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.539083+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":727,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514238+0000","last_change":"2024-03-26T03:14:37.252086+0000","last_active":"2024-03-26T03:16:03.514238+0000","last_peered":"2024-03-26T03:16:03.514238+0000","last_clean":"2024-03-26T03:16:03.514238+0000","last_became_active":"2024-03-26T03:14:37.251746+0000","last_became_peered":"2024-03-26T03:14:37.251746+0000","last_unstale":"2024-03-26T03:16:03.514238+0000","last_undegraded":"2024-03-26T03:16:03.514238+0000","last_fullsized":"2024-03-26T03:16:03.514238+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:19.981413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.981413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":116,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483218+0000","last_change":"2024-03-26T03:14:22.567168+0000","last_active":"2024-03-26T03:16:03.483218+0000","last_peered":"2024-03-26T03:16:03.483218+0000","last_clean":"2024-03-26T03:16:03.483218+0000","last_became_active":"2024-03-26T03:14:22.566823+0000","last_became_peered":"2024-03-26T03:14:22.566823+0000","last_unstale":"2024-03-26T03:16:03.483218+0000","last_undegraded":"2024-03-26T03:16:03.483218+0000","last_fullsized":"2024-03-26T03:16:03.483218+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":784,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513211+0000","last_change":"2024-03-26T03:14:40.180344+0000","last_active":"2024-03-26T03:16:03.513211+0000","last_peered":"2024-03-26T03:16:03.513211+0000","last_clean":"2024-03-26T03:16:03.513211+0000","last_became_active":"2024-03-26T03:14:40.178828+0000","last_became_peered":"2024-03-26T03:14:40.178828+0000","last_unstale":"2024-03-26T03:16:03.513211+0000","last_undegraded":"2024-03-26T03:16:03.513211+0000","last_fullsized":"2024-03-26T03:16:03.513211+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:04:42.846152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:04:42.846152+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.942328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"384'13777","reported_seq":19116,"reported_epoch":384,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-03-26T03:16:09.668592+0000","last_change":"2024-03-26T03:16:07.322613+0000","last_active":"2024-03-26T03:16:09.668592+0000","last_peered":"2024-03-26T03:16:09.668592+0000","last_clean":"2024-03-26T03:12:35.089656+0000","last_became_active":"2024-03-26T03:14:22.698063+0000","last_became_peered":"2024-03-26T03:14:22.698063+0000","last_unstale":"2024-03-26T03:16:09.668592+0000","last_undegraded":"2024-03-26T03:14:22.579770+0000","last_fullsized":"2024-03-26T03:14:22.563495+0000","mapping_epoch":332,"log_start":"358'10396","ondisk_log_start":"358'10396","created":289,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:12:35.089656+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.089656+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99680256,"num_objects":1521,"num_object_clones":0,"num_object_copies":4563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13777,"num_write_kb":489536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2560,"num_bytes_recovered":167772160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1521}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":503,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485039+0000","last_change":"2024-03-26T03:15:39.823867+0000","last_active":"2024-03-26T03:16:03.485039+0000","last_peered":"2024-03-26T03:16:03.485039+0000","last_clean":"2024-03-26T03:16:03.485039+0000","last_became_active":"2024-03-26T03:15:39.823405+0000","last_became_peered":"2024-03-26T03:15:39.823405+0000","last_unstale":"2024-03-26T03:16:03.485039+0000","last_undegraded":"2024-03-26T03:16:03.485039+0000","last_fullsized":"2024-03-26T03:16:03.485039+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.474559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.474559+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":728,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483498+0000","last_change":"2024-03-26T03:13:35.281909+0000","last_active":"2024-03-26T03:16:03.483498+0000","last_peered":"2024-03-26T03:16:03.483498+0000","last_clean":"2024-03-26T03:16:03.483498+0000","last_became_active":"2024-03-26T03:13:35.281453+0000","last_became_peered":"2024-03-26T03:13:35.281453+0000","last_unstale":"2024-03-26T03:16:03.483498+0000","last_undegraded":"2024-03-26T03:16:03.483498+0000","last_fullsized":"2024-03-26T03:16:03.483498+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:26.091566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:26.091566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":772,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489546+0000","last_change":"2024-03-26T03:14:37.290458+0000","last_active":"2024-03-26T03:16:03.489546+0000","last_peered":"2024-03-26T03:16:03.489546+0000","last_clean":"2024-03-26T03:16:03.489546+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:16:03.489546+0000","last_undegraded":"2024-03-26T03:16:03.489546+0000","last_fullsized":"2024-03-26T03:16:03.489546+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:14:37.290313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:14:37.290313+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269196+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":772,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490560+0000","last_change":"2024-03-26T03:14:37.285721+0000","last_active":"2024-03-26T03:16:03.490560+0000","last_peered":"2024-03-26T03:16:03.490560+0000","last_clean":"2024-03-26T03:16:03.490560+0000","last_became_active":"2024-03-26T03:14:37.249021+0000","last_became_peered":"2024-03-26T03:14:37.249021+0000","last_unstale":"2024-03-26T03:16:03.490560+0000","last_undegraded":"2024-03-26T03:16:03.490560+0000","last_fullsized":"2024-03-26T03:16:03.490560+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:12:33.067512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:33.067512+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269060+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":377,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485793+0000","last_change":"2024-03-26T03:14:37.265797+0000","last_active":"2024-03-26T03:16:03.485793+0000","last_peered":"2024-03-26T03:16:03.485793+0000","last_clean":"2024-03-26T03:16:03.485793+0000","last_became_active":"2024-03-26T03:14:37.265041+0000","last_became_peered":"2024-03-26T03:14:37.265041+0000","last_unstale":"2024-03-26T03:16:03.485793+0000","last_undegraded":"2024-03-26T03:16:03.485793+0000","last_fullsized":"2024-03-26T03:16:03.485793+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":822,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513221+0000","last_change":"2024-03-26T03:15:05.183541+0000","last_active":"2024-03-26T03:16:03.513221+0000","last_peered":"2024-03-26T03:16:03.513221+0000","last_clean":"2024-03-26T03:16:03.513221+0000","last_became_active":"2024-03-26T03:15:05.183000+0000","last_became_peered":"2024-03-26T03:15:05.183000+0000","last_unstale":"2024-03-26T03:16:03.513221+0000","last_undegraded":"2024-03-26T03:16:03.513221+0000","last_fullsized":"2024-03-26T03:16:03.513221+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:22.342472+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:12:22.342472+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":202,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486798+0000","last_change":"2024-03-26T03:14:37.323020+0000","last_active":"2024-03-26T03:16:03.486798+0000","last_peered":"2024-03-26T03:16:03.486798+0000","last_clean":"2024-03-26T03:16:03.486798+0000","last_became_active":"2024-03-26T03:14:37.269153+0000","last_became_peered":"2024-03-26T03:14:37.269153+0000","last_unstale":"2024-03-26T03:16:03.486798+0000","last_undegraded":"2024-03-26T03:16:03.486798+0000","last_fullsized":"2024-03-26T03:16:03.486798+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":748,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483340+0000","last_change":"2024-03-26T03:15:36.803800+0000","last_active":"2024-03-26T03:16:03.483340+0000","last_peered":"2024-03-26T03:16:03.483340+0000","last_clean":"2024-03-26T03:16:03.483340+0000","last_became_active":"2024-03-26T03:15:36.797246+0000","last_became_peered":"2024-03-26T03:15:36.797246+0000","last_unstale":"2024-03-26T03:16:03.483340+0000","last_undegraded":"2024-03-26T03:16:03.483340+0000","last_fullsized":"2024-03-26T03:16:03.483340+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:12:23.943885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:23.943885+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":397,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485453+0000","last_change":"2024-03-26T03:14:37.274074+0000","last_active":"2024-03-26T03:16:03.485453+0000","last_peered":"2024-03-26T03:16:03.485453+0000","last_clean":"2024-03-26T03:16:03.485453+0000","last_became_active":"2024-03-26T03:14:37.273658+0000","last_became_peered":"2024-03-26T03:14:37.273658+0000","last_unstale":"2024-03-26T03:16:03.485453+0000","last_undegraded":"2024-03-26T03:16:03.485453+0000","last_fullsized":"2024-03-26T03:16:03.485453+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":716,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484310+0000","last_change":"2024-03-26T03:14:37.239442+0000","last_active":"2024-03-26T03:16:03.484310+0000","last_peered":"2024-03-26T03:16:03.484310+0000","last_clean":"2024-03-26T03:16:03.484310+0000","last_became_active":"2024-03-26T03:14:37.239105+0000","last_became_peered":"2024-03-26T03:14:37.239105+0000","last_unstale":"2024-03-26T03:16:03.484310+0000","last_undegraded":"2024-03-26T03:16:03.484310+0000","last_fullsized":"2024-03-26T03:16:03.484310+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:17.998402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.998402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":588,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489373+0000","last_change":"2024-03-26T03:13:28.546432+0000","last_active":"2024-03-26T03:16:03.489373+0000","last_peered":"2024-03-26T03:16:03.489373+0000","last_clean":"2024-03-26T03:16:03.489373+0000","last_became_active":"2024-03-26T03:10:44.085377+0000","last_became_peered":"2024-03-26T03:10:44.085377+0000","last_unstale":"2024-03-26T03:16:03.489373+0000","last_undegraded":"2024-03-26T03:16:03.489373+0000","last_fullsized":"2024-03-26T03:16:03.489373+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:28.546187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:28.546187+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":462,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484899+0000","last_change":"2024-03-26T03:14:37.319032+0000","last_active":"2024-03-26T03:16:03.484899+0000","last_peered":"2024-03-26T03:16:03.484899+0000","last_clean":"2024-03-26T03:16:03.484899+0000","last_became_active":"2024-03-26T03:14:37.269408+0000","last_became_peered":"2024-03-26T03:14:37.269408+0000","last_unstale":"2024-03-26T03:16:03.484899+0000","last_undegraded":"2024-03-26T03:16:03.484899+0000","last_fullsized":"2024-03-26T03:16:03.484899+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'12","last_scrub_stamp":"2024-03-26T03:01:50.876023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:50.876023+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512137+0000","last_change":"2024-03-26T03:15:02.576361+0000","last_active":"2024-03-26T03:16:03.512137+0000","last_peered":"2024-03-26T03:16:03.512137+0000","last_clean":"2024-03-26T03:16:03.512137+0000","last_became_active":"2024-03-26T03:15:02.574289+0000","last_became_peered":"2024-03-26T03:15:02.574289+0000","last_unstale":"2024-03-26T03:16:03.512137+0000","last_undegraded":"2024-03-26T03:16:03.512137+0000","last_fullsized":"2024-03-26T03:16:03.512137+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.094677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.094677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":690,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490332+0000","last_change":"2024-03-26T03:14:30.639018+0000","last_active":"2024-03-26T03:16:03.490332+0000","last_peered":"2024-03-26T03:16:03.490332+0000","last_clean":"2024-03-26T03:16:03.490332+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:16:03.490332+0000","last_undegraded":"2024-03-26T03:16:03.490332+0000","last_fullsized":"2024-03-26T03:16:03.490332+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:30.638868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:30.638868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.634011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":96,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513338+0000","last_change":"2024-03-26T03:14:37.333977+0000","last_active":"2024-03-26T03:16:03.513338+0000","last_peered":"2024-03-26T03:16:03.513338+0000","last_clean":"2024-03-26T03:16:03.513338+0000","last_became_active":"2024-03-26T03:14:37.333430+0000","last_became_peered":"2024-03-26T03:14:37.333430+0000","last_unstale":"2024-03-26T03:16:03.513338+0000","last_undegraded":"2024-03-26T03:16:03.513338+0000","last_fullsized":"2024-03-26T03:16:03.513338+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:45.964365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:45.964365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":467,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488583+0000","last_change":"2024-03-26T03:14:24.781366+0000","last_active":"2024-03-26T03:16:03.488583+0000","last_peered":"2024-03-26T03:16:03.488583+0000","last_clean":"2024-03-26T03:16:03.488583+0000","last_became_active":"2024-03-26T03:14:24.567937+0000","last_became_peered":"2024-03-26T03:14:24.567937+0000","last_unstale":"2024-03-26T03:16:03.488583+0000","last_undegraded":"2024-03-26T03:16:03.488583+0000","last_fullsized":"2024-03-26T03:16:03.488583+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:24.781232+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:14:24.781232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633885+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":479,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485260+0000","last_change":"2024-03-26T03:14:37.287298+0000","last_active":"2024-03-26T03:16:03.485260+0000","last_peered":"2024-03-26T03:16:03.485260+0000","last_clean":"2024-03-26T03:16:03.485260+0000","last_became_active":"2024-03-26T03:14:37.265466+0000","last_became_peered":"2024-03-26T03:14:37.265466+0000","last_unstale":"2024-03-26T03:16:03.485260+0000","last_undegraded":"2024-03-26T03:16:03.485260+0000","last_fullsized":"2024-03-26T03:16:03.485260+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'29","last_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_deep_scrub":"151'29","last_deep_scrub_stamp":"2024-03-26T03:03:52.392020+0000","last_clean_scrub_stamp":"2024-03-26T03:03:52.392020+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":710,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489652+0000","last_change":"2024-03-26T03:15:36.787524+0000","last_active":"2024-03-26T03:16:03.489652+0000","last_peered":"2024-03-26T03:16:03.489652+0000","last_clean":"2024-03-26T03:16:03.489652+0000","last_became_active":"2024-03-26T03:15:36.786930+0000","last_became_peered":"2024-03-26T03:15:36.786930+0000","last_unstale":"2024-03-26T03:16:03.489652+0000","last_undegraded":"2024-03-26T03:16:03.489652+0000","last_fullsized":"2024-03-26T03:16:03.489652+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:17.582921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:06:17.582921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633453+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"333'13","reported_seq":945,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490300+0000","last_change":"2024-03-26T03:14:22.675905+0000","last_active":"2024-03-26T03:16:03.490300+0000","last_peered":"2024-03-26T03:16:03.490300+0000","last_clean":"2024-03-26T03:16:03.490300+0000","last_became_active":"2024-03-26T03:14:22.577229+0000","last_became_peered":"2024-03-26T03:14:22.577229+0000","last_unstale":"2024-03-26T03:16:03.490300+0000","last_undegraded":"2024-03-26T03:16:03.490300+0000","last_fullsized":"2024-03-26T03:16:03.490300+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"201'9","last_scrub_stamp":"2024-03-26T03:13:19.552025+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.552025+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":434,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486909+0000","last_change":"2024-03-26T03:14:37.275365+0000","last_active":"2024-03-26T03:16:03.486909+0000","last_peered":"2024-03-26T03:16:03.486909+0000","last_clean":"2024-03-26T03:16:03.486909+0000","last_became_active":"2024-03-26T03:14:37.273988+0000","last_became_peered":"2024-03-26T03:14:37.273988+0000","last_unstale":"2024-03-26T03:16:03.486909+0000","last_undegraded":"2024-03-26T03:16:03.486909+0000","last_fullsized":"2024-03-26T03:16:03.486909+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":391,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512036+0000","last_change":"2024-03-26T03:14:40.173591+0000","last_active":"2024-03-26T03:16:03.512036+0000","last_peered":"2024-03-26T03:16:03.512036+0000","last_clean":"2024-03-26T03:16:03.512036+0000","last_became_active":"2024-03-26T03:14:40.173243+0000","last_became_peered":"2024-03-26T03:14:40.173243+0000","last_unstale":"2024-03-26T03:16:03.512036+0000","last_undegraded":"2024-03-26T03:16:03.512036+0000","last_fullsized":"2024-03-26T03:16:03.512036+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:18.008606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:18.008606+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.942263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":527,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484682+0000","last_change":"2024-03-26T03:14:37.341736+0000","last_active":"2024-03-26T03:16:03.484682+0000","last_peered":"2024-03-26T03:16:03.484682+0000","last_clean":"2024-03-26T03:16:03.484682+0000","last_became_active":"2024-03-26T03:14:37.333426+0000","last_became_peered":"2024-03-26T03:14:37.333426+0000","last_unstale":"2024-03-26T03:16:03.484682+0000","last_undegraded":"2024-03-26T03:16:03.484682+0000","last_fullsized":"2024-03-26T03:16:03.484682+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:02.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:02.283801+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":460,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488050+0000","last_change":"2024-03-26T03:14:37.263218+0000","last_active":"2024-03-26T03:16:03.488050+0000","last_peered":"2024-03-26T03:16:03.488050+0000","last_clean":"2024-03-26T03:16:03.488050+0000","last_became_active":"2024-03-26T03:14:37.262766+0000","last_became_peered":"2024-03-26T03:14:37.262766+0000","last_unstale":"2024-03-26T03:16:03.488050+0000","last_undegraded":"2024-03-26T03:16:03.488050+0000","last_fullsized":"2024-03-26T03:16:03.488050+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:04:46.666837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268780+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"342'11","reported_seq":459,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485600+0000","last_change":"2024-03-26T03:14:37.279513+0000","last_active":"2024-03-26T03:16:03.485600+0000","last_peered":"2024-03-26T03:16:03.485600+0000","last_clean":"2024-03-26T03:16:03.485600+0000","last_became_active":"2024-03-26T03:14:37.264759+0000","last_became_peered":"2024-03-26T03:14:37.264759+0000","last_unstale":"2024-03-26T03:16:03.485600+0000","last_undegraded":"2024-03-26T03:16:03.485600+0000","last_fullsized":"2024-03-26T03:16:03.485600+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:04:44.796644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.796644+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:09.863200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":693,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490644+0000","last_change":"2024-03-26T03:13:19.308432+0000","last_active":"2024-03-26T03:16:03.490644+0000","last_peered":"2024-03-26T03:16:03.490644+0000","last_clean":"2024-03-26T03:16:03.490644+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:16:03.490644+0000","last_undegraded":"2024-03-26T03:16:03.490644+0000","last_fullsized":"2024-03-26T03:16:03.490644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:13:19.308309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.308309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269113+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":796,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490365+0000","last_change":"2024-03-26T03:14:40.171431+0000","last_active":"2024-03-26T03:16:03.490365+0000","last_peered":"2024-03-26T03:16:03.490365+0000","last_clean":"2024-03-26T03:16:03.490365+0000","last_became_active":"2024-03-26T03:14:40.170537+0000","last_became_peered":"2024-03-26T03:14:40.170537+0000","last_unstale":"2024-03-26T03:16:03.490365+0000","last_undegraded":"2024-03-26T03:16:03.490365+0000","last_fullsized":"2024-03-26T03:16:03.490365+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:22.652846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:22.652846+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633586+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":739,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513595+0000","last_change":"2024-03-26T03:13:33.092136+0000","last_active":"2024-03-26T03:16:03.513595+0000","last_peered":"2024-03-26T03:16:03.513595+0000","last_clean":"2024-03-26T03:16:03.513595+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:16:03.513595+0000","last_undegraded":"2024-03-26T03:16:03.513595+0000","last_fullsized":"2024-03-26T03:16:03.513595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:13:33.091975+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:13:33.091975+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862745+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":691,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512511+0000","last_change":"2024-03-26T03:14:21.589026+0000","last_active":"2024-03-26T03:16:03.512511+0000","last_peered":"2024-03-26T03:16:03.512511+0000","last_clean":"2024-03-26T03:16:03.512511+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:16:03.512511+0000","last_undegraded":"2024-03-26T03:16:03.512511+0000","last_fullsized":"2024-03-26T03:16:03.512511+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:15.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:15.802110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":380,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485535+0000","last_change":"2024-03-26T03:14:22.573582+0000","last_active":"2024-03-26T03:16:03.485535+0000","last_peered":"2024-03-26T03:16:03.485535+0000","last_clean":"2024-03-26T03:16:03.485535+0000","last_became_active":"2024-03-26T03:14:22.573232+0000","last_became_peered":"2024-03-26T03:14:22.573232+0000","last_unstale":"2024-03-26T03:16:03.485535+0000","last_undegraded":"2024-03-26T03:16:03.485535+0000","last_fullsized":"2024-03-26T03:16:03.485535+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:03:44.286053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:03:44.286053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":154,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512942+0000","last_change":"2024-03-26T03:13:35.280094+0000","last_active":"2024-03-26T03:16:03.512942+0000","last_peered":"2024-03-26T03:16:03.512942+0000","last_clean":"2024-03-26T03:16:03.512942+0000","last_became_active":"2024-03-26T03:13:35.278675+0000","last_became_peered":"2024-03-26T03:13:35.278675+0000","last_unstale":"2024-03-26T03:16:03.512942+0000","last_undegraded":"2024-03-26T03:16:03.512942+0000","last_fullsized":"2024-03-26T03:16:03.512942+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:38.484038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:38.484038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":480,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514841+0000","last_change":"2024-03-26T03:14:22.578069+0000","last_active":"2024-03-26T03:16:03.514841+0000","last_peered":"2024-03-26T03:16:03.514841+0000","last_clean":"2024-03-26T03:16:03.514841+0000","last_became_active":"2024-03-26T03:14:22.577614+0000","last_became_peered":"2024-03-26T03:14:22.577614+0000","last_unstale":"2024-03-26T03:16:03.514841+0000","last_undegraded":"2024-03-26T03:16:03.514841+0000","last_fullsized":"2024-03-26T03:16:03.514841+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.940687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.940687+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862953+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":731,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.487927+0000","last_change":"2024-03-26T03:11:42.830091+0000","last_active":"2024-03-26T03:16:03.487927+0000","last_peered":"2024-03-26T03:16:03.487927+0000","last_clean":"2024-03-26T03:16:03.487927+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:16:03.487927+0000","last_undegraded":"2024-03-26T03:16:03.487927+0000","last_fullsized":"2024-03-26T03:16:03.487927+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:11:42.829912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:42.829912+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268893+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":720,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514669+0000","last_change":"2024-03-26T03:13:35.275554+0000","last_active":"2024-03-26T03:16:03.514669+0000","last_peered":"2024-03-26T03:16:03.514669+0000","last_clean":"2024-03-26T03:16:03.514669+0000","last_became_active":"2024-03-26T03:13:35.274830+0000","last_became_peered":"2024-03-26T03:13:35.274830+0000","last_unstale":"2024-03-26T03:16:03.514669+0000","last_undegraded":"2024-03-26T03:16:03.514669+0000","last_fullsized":"2024-03-26T03:16:03.514669+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.319189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.319189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":683,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513952+0000","last_change":"2024-03-26T03:15:14.862264+0000","last_active":"2024-03-26T03:16:03.513952+0000","last_peered":"2024-03-26T03:16:03.513952+0000","last_clean":"2024-03-26T03:16:03.513952+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:16:03.513952+0000","last_undegraded":"2024-03-26T03:16:03.513952+0000","last_fullsized":"2024-03-26T03:16:03.513952+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:14.862148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:14.862148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:41.652065+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":392,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483713+0000","last_change":"2024-03-26T03:14:22.577571+0000","last_active":"2024-03-26T03:16:03.483713+0000","last_peered":"2024-03-26T03:16:03.483713+0000","last_clean":"2024-03-26T03:16:03.483713+0000","last_became_active":"2024-03-26T03:14:22.577265+0000","last_became_peered":"2024-03-26T03:14:22.577265+0000","last_unstale":"2024-03-26T03:16:03.483713+0000","last_undegraded":"2024-03-26T03:16:03.483713+0000","last_fullsized":"2024-03-26T03:16:03.483713+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:19.903253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:19.903253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":403,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514034+0000","last_change":"2024-03-26T03:14:40.950126+0000","last_active":"2024-03-26T03:16:03.514034+0000","last_peered":"2024-03-26T03:16:03.514034+0000","last_clean":"2024-03-26T03:16:03.514034+0000","last_became_active":"2024-03-26T03:14:40.166745+0000","last_became_peered":"2024-03-26T03:14:40.166745+0000","last_unstale":"2024-03-26T03:16:03.514034+0000","last_undegraded":"2024-03-26T03:16:03.514034+0000","last_fullsized":"2024-03-26T03:16:03.514034+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:40.950003+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:14:40.950003+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.362422+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":401,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483316+0000","last_change":"2024-03-26T03:14:21.689604+0000","last_active":"2024-03-26T03:16:03.483316+0000","last_peered":"2024-03-26T03:16:03.483316+0000","last_clean":"2024-03-26T03:16:03.483316+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:16:03.483316+0000","last_undegraded":"2024-03-26T03:16:03.483316+0000","last_fullsized":"2024-03-26T03:16:03.483316+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'4","last_scrub_stamp":"2024-03-26T03:01:37.776542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:01:37.776542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":729,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490422+0000","last_change":"2024-03-26T03:13:34.684610+0000","last_active":"2024-03-26T03:16:03.490422+0000","last_peered":"2024-03-26T03:16:03.490422+0000","last_clean":"2024-03-26T03:16:03.490422+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:16:03.490422+0000","last_undegraded":"2024-03-26T03:16:03.490422+0000","last_fullsized":"2024-03-26T03:16:03.490422+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:34.684526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:13:34.684526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":726,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512133+0000","last_change":"2024-03-26T03:15:02.574249+0000","last_active":"2024-03-26T03:16:03.512133+0000","last_peered":"2024-03-26T03:16:03.512133+0000","last_clean":"2024-03-26T03:16:03.512133+0000","last_became_active":"2024-03-26T03:15:02.573859+0000","last_became_peered":"2024-03-26T03:15:02.573859+0000","last_unstale":"2024-03-26T03:16:03.512133+0000","last_undegraded":"2024-03-26T03:16:03.512133+0000","last_fullsized":"2024-03-26T03:16:03.512133+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.300020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918533+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":96,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488776+0000","last_change":"2024-03-26T03:14:37.261464+0000","last_active":"2024-03-26T03:16:03.488776+0000","last_peered":"2024-03-26T03:16:03.488776+0000","last_clean":"2024-03-26T03:16:03.488776+0000","last_became_active":"2024-03-26T03:14:37.260920+0000","last_became_peered":"2024-03-26T03:14:37.260920+0000","last_unstale":"2024-03-26T03:16:03.488776+0000","last_undegraded":"2024-03-26T03:16:03.488776+0000","last_fullsized":"2024-03-26T03:16:03.488776+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:01.548673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:11:01.548673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":600,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483491+0000","last_change":"2024-03-26T03:15:05.189270+0000","last_active":"2024-03-26T03:16:03.483491+0000","last_peered":"2024-03-26T03:16:03.483491+0000","last_clean":"2024-03-26T03:16:03.483491+0000","last_became_active":"2024-03-26T03:15:05.188632+0000","last_became_peered":"2024-03-26T03:15:05.188632+0000","last_unstale":"2024-03-26T03:16:03.483491+0000","last_undegraded":"2024-03-26T03:16:03.483491+0000","last_fullsized":"2024-03-26T03:16:03.483491+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:12:21.953552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:12:21.953552+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":464,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483913+0000","last_change":"2024-03-26T03:14:37.321944+0000","last_active":"2024-03-26T03:16:03.483913+0000","last_peered":"2024-03-26T03:16:03.483913+0000","last_clean":"2024-03-26T03:16:03.483913+0000","last_became_active":"2024-03-26T03:14:37.273380+0000","last_became_peered":"2024-03-26T03:14:37.273380+0000","last_unstale":"2024-03-26T03:16:03.483913+0000","last_undegraded":"2024-03-26T03:16:03.483913+0000","last_fullsized":"2024-03-26T03:16:03.483913+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'23","last_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_deep_scrub":"150'23","last_deep_scrub_stamp":"2024-03-26T03:03:50.330297+0000","last_clean_scrub_stamp":"2024-03-26T03:03:50.330297+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":714,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513446+0000","last_change":"2024-03-26T03:13:30.335583+0000","last_active":"2024-03-26T03:16:03.513446+0000","last_peered":"2024-03-26T03:16:03.513446+0000","last_clean":"2024-03-26T03:16:03.513446+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:16:03.513446+0000","last_undegraded":"2024-03-26T03:16:03.513446+0000","last_fullsized":"2024-03-26T03:16:03.513446+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:30.334068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:30.334068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918560+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":96,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489787+0000","last_change":"2024-03-26T03:14:37.249712+0000","last_active":"2024-03-26T03:16:03.489787+0000","last_peered":"2024-03-26T03:16:03.489787+0000","last_clean":"2024-03-26T03:16:03.489787+0000","last_became_active":"2024-03-26T03:14:37.249377+0000","last_became_peered":"2024-03-26T03:14:37.249377+0000","last_unstale":"2024-03-26T03:16:03.489787+0000","last_undegraded":"2024-03-26T03:16:03.489787+0000","last_fullsized":"2024-03-26T03:16:03.489787+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:30.019226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:12:30.019226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":378,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485256+0000","last_change":"2024-03-26T03:14:22.603550+0000","last_active":"2024-03-26T03:16:03.485256+0000","last_peered":"2024-03-26T03:16:03.485256+0000","last_clean":"2024-03-26T03:16:03.485256+0000","last_became_active":"2024-03-26T03:14:22.603198+0000","last_became_peered":"2024-03-26T03:14:22.603198+0000","last_unstale":"2024-03-26T03:16:03.485256+0000","last_undegraded":"2024-03-26T03:16:03.485256+0000","last_fullsized":"2024-03-26T03:16:03.485256+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":707,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489262+0000","last_change":"2024-03-26T03:13:38.310162+0000","last_active":"2024-03-26T03:16:03.489262+0000","last_peered":"2024-03-26T03:16:03.489262+0000","last_clean":"2024-03-26T03:16:03.489262+0000","last_became_active":"2024-03-26T03:13:38.309830+0000","last_became_peered":"2024-03-26T03:13:38.309830+0000","last_unstale":"2024-03-26T03:16:03.489262+0000","last_undegraded":"2024-03-26T03:16:03.489262+0000","last_fullsized":"2024-03-26T03:16:03.489262+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:21.641797+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:13:21.641797+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633655+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":583,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.486690+0000","last_change":"2024-03-26T03:14:37.333804+0000","last_active":"2024-03-26T03:16:03.486690+0000","last_peered":"2024-03-26T03:16:03.486690+0000","last_clean":"2024-03-26T03:16:03.486690+0000","last_became_active":"2024-03-26T03:14:37.274931+0000","last_became_peered":"2024-03-26T03:14:37.274931+0000","last_unstale":"2024-03-26T03:16:03.486690+0000","last_undegraded":"2024-03-26T03:16:03.486690+0000","last_fullsized":"2024-03-26T03:16:03.486690+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":735,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.491365+0000","last_change":"2024-03-26T03:14:37.267711+0000","last_active":"2024-03-26T03:16:03.491365+0000","last_peered":"2024-03-26T03:16:03.491365+0000","last_clean":"2024-03-26T03:16:03.491365+0000","last_became_active":"2024-03-26T03:14:37.266519+0000","last_became_peered":"2024-03-26T03:14:37.266519+0000","last_unstale":"2024-03-26T03:16:03.491365+0000","last_undegraded":"2024-03-26T03:16:03.491365+0000","last_fullsized":"2024-03-26T03:16:03.491365+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:35.673504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:13:35.673504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"353'10","reported_seq":945,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513623+0000","last_change":"2024-03-26T03:15:02.586070+0000","last_active":"2024-03-26T03:16:03.513623+0000","last_peered":"2024-03-26T03:16:03.513623+0000","last_clean":"2024-03-26T03:16:03.513623+0000","last_became_active":"2024-03-26T03:15:02.579403+0000","last_became_peered":"2024-03-26T03:15:02.579403+0000","last_unstale":"2024-03-26T03:16:03.513623+0000","last_undegraded":"2024-03-26T03:16:03.513623+0000","last_fullsized":"2024-03-26T03:16:03.513623+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:12:38.100984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:38.100984+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:10.861880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":607,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483387+0000","last_change":"2024-03-26T03:14:37.289992+0000","last_active":"2024-03-26T03:16:03.483387+0000","last_peered":"2024-03-26T03:16:03.483387+0000","last_clean":"2024-03-26T03:16:03.483387+0000","last_became_active":"2024-03-26T03:14:37.252165+0000","last_became_peered":"2024-03-26T03:14:37.252165+0000","last_unstale":"2024-03-26T03:16:03.483387+0000","last_undegraded":"2024-03-26T03:16:03.483387+0000","last_fullsized":"2024-03-26T03:16:03.483387+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.141917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":690,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513084+0000","last_change":"2024-03-26T03:15:39.825446+0000","last_active":"2024-03-26T03:16:03.513084+0000","last_peered":"2024-03-26T03:16:03.513084+0000","last_clean":"2024-03-26T03:16:03.513084+0000","last_became_active":"2024-03-26T03:15:39.824782+0000","last_became_peered":"2024-03-26T03:15:39.824782+0000","last_unstale":"2024-03-26T03:16:03.513084+0000","last_undegraded":"2024-03-26T03:16:03.513084+0000","last_fullsized":"2024-03-26T03:16:03.513084+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:21.061456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:21.061456+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.924414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":754,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513652+0000","last_change":"2024-03-26T03:13:32.336994+0000","last_active":"2024-03-26T03:16:03.513652+0000","last_peered":"2024-03-26T03:16:03.513652+0000","last_clean":"2024-03-26T03:16:03.513652+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:16:03.513652+0000","last_undegraded":"2024-03-26T03:16:03.513652+0000","last_fullsized":"2024-03-26T03:16:03.513652+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:13:32.336748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:32.336748+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918649+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":351,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483313+0000","last_change":"2024-03-26T03:14:22.577227+0000","last_active":"2024-03-26T03:16:03.483313+0000","last_peered":"2024-03-26T03:16:03.483313+0000","last_clean":"2024-03-26T03:16:03.483313+0000","last_became_active":"2024-03-26T03:14:22.576816+0000","last_became_peered":"2024-03-26T03:14:22.576816+0000","last_unstale":"2024-03-26T03:16:03.483313+0000","last_undegraded":"2024-03-26T03:16:03.483313+0000","last_fullsized":"2024-03-26T03:16:03.483313+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:04:40.740040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:04:40.740040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":99,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.515032+0000","last_change":"2024-03-26T03:14:37.255762+0000","last_active":"2024-03-26T03:16:03.515032+0000","last_peered":"2024-03-26T03:16:03.515032+0000","last_clean":"2024-03-26T03:16:03.515032+0000","last_became_active":"2024-03-26T03:14:37.255400+0000","last_became_peered":"2024-03-26T03:14:37.255400+0000","last_unstale":"2024-03-26T03:16:03.515032+0000","last_undegraded":"2024-03-26T03:16:03.515032+0000","last_fullsized":"2024-03-26T03:16:03.515032+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:25.571608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:13:25.571608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.863056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514414+0000","last_change":"2024-03-26T03:15:36.798920+0000","last_active":"2024-03-26T03:16:03.514414+0000","last_peered":"2024-03-26T03:16:03.514414+0000","last_clean":"2024-03-26T03:16:03.514414+0000","last_became_active":"2024-03-26T03:15:36.796445+0000","last_became_peered":"2024-03-26T03:15:36.796445+0000","last_unstale":"2024-03-26T03:16:03.514414+0000","last_undegraded":"2024-03-26T03:16:03.514414+0000","last_fullsized":"2024-03-26T03:16:03.514414+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:36.651720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:12:36.651720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862433+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":604,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483228+0000","last_change":"2024-03-26T03:14:40.162652+0000","last_active":"2024-03-26T03:16:03.483228+0000","last_peered":"2024-03-26T03:16:03.483228+0000","last_clean":"2024-03-26T03:16:03.483228+0000","last_became_active":"2024-03-26T03:14:40.162258+0000","last_became_peered":"2024-03-26T03:14:40.162258+0000","last_unstale":"2024-03-26T03:16:03.483228+0000","last_undegraded":"2024-03-26T03:16:03.483228+0000","last_fullsized":"2024-03-26T03:16:03.483228+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.546156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.546156+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":744,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484542+0000","last_change":"2024-03-26T03:14:21.684103+0000","last_active":"2024-03-26T03:16:03.484542+0000","last_peered":"2024-03-26T03:16:03.484542+0000","last_clean":"2024-03-26T03:16:03.484542+0000","last_became_active":"2024-03-26T03:14:21.586487+0000","last_became_peered":"2024-03-26T03:14:21.586487+0000","last_unstale":"2024-03-26T03:16:03.484542+0000","last_undegraded":"2024-03-26T03:16:03.484542+0000","last_fullsized":"2024-03-26T03:16:03.484542+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'15","last_scrub_stamp":"2024-03-26T03:04:44.684413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:44.684413+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":727,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512690+0000","last_change":"2024-03-26T03:14:37.249362+0000","last_active":"2024-03-26T03:16:03.512690+0000","last_peered":"2024-03-26T03:16:03.512690+0000","last_clean":"2024-03-26T03:16:03.512690+0000","last_became_active":"2024-03-26T03:14:37.249066+0000","last_became_peered":"2024-03-26T03:14:37.249066+0000","last_unstale":"2024-03-26T03:16:03.512690+0000","last_undegraded":"2024-03-26T03:16:03.512690+0000","last_fullsized":"2024-03-26T03:16:03.512690+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:22.309995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:22.309995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918471+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":717,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.515030+0000","last_change":"2024-03-26T03:15:02.570298+0000","last_active":"2024-03-26T03:16:03.515030+0000","last_peered":"2024-03-26T03:16:03.515030+0000","last_clean":"2024-03-26T03:16:03.515030+0000","last_became_active":"2024-03-26T03:15:02.569904+0000","last_became_peered":"2024-03-26T03:15:02.569904+0000","last_unstale":"2024-03-26T03:16:03.515030+0000","last_undegraded":"2024-03-26T03:16:03.515030+0000","last_fullsized":"2024-03-26T03:16:03.515030+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:35.484358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:12:35.484358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":701,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490001+0000","last_change":"2024-03-26T03:13:29.068724+0000","last_active":"2024-03-26T03:16:03.490001+0000","last_peered":"2024-03-26T03:16:03.490001+0000","last_clean":"2024-03-26T03:16:03.490001+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:16:03.490001+0000","last_undegraded":"2024-03-26T03:16:03.490001+0000","last_fullsized":"2024-03-26T03:16:03.490001+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.068502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.068502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"82'128","reported_seq":491,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.491469+0000","last_change":"2024-03-26T03:14:40.162986+0000","last_active":"2024-03-26T03:16:03.491469+0000","last_peered":"2024-03-26T03:16:03.491469+0000","last_clean":"2024-03-26T03:16:03.491469+0000","last_became_active":"2024-03-26T03:14:40.162583+0000","last_became_peered":"2024-03-26T03:14:40.162583+0000","last_unstale":"2024-03-26T03:16:03.491469+0000","last_undegraded":"2024-03-26T03:16:03.491469+0000","last_fullsized":"2024-03-26T03:16:03.491469+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.604164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.604164+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633622+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"231'24","reported_seq":436,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485028+0000","last_change":"2024-03-26T03:14:21.678553+0000","last_active":"2024-03-26T03:16:03.485028+0000","last_peered":"2024-03-26T03:16:03.485028+0000","last_clean":"2024-03-26T03:16:03.485028+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:16:03.485028+0000","last_undegraded":"2024-03-26T03:16:03.485028+0000","last_fullsized":"2024-03-26T03:16:03.485028+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'18","last_scrub_stamp":"2024-03-26T03:03:47.301480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:03:47.301480+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":409,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485633+0000","last_change":"2024-03-26T03:14:37.275602+0000","last_active":"2024-03-26T03:16:03.485633+0000","last_peered":"2024-03-26T03:16:03.485633+0000","last_clean":"2024-03-26T03:16:03.485633+0000","last_became_active":"2024-03-26T03:14:37.275278+0000","last_became_peered":"2024-03-26T03:14:37.275278+0000","last_unstale":"2024-03-26T03:16:03.485633+0000","last_undegraded":"2024-03-26T03:16:03.485633+0000","last_fullsized":"2024-03-26T03:16:03.485633+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:01:49.896449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:01:49.896449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":680,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488303+0000","last_change":"2024-03-26T03:14:36.271942+0000","last_active":"2024-03-26T03:16:03.488303+0000","last_peered":"2024-03-26T03:16:03.488303+0000","last_clean":"2024-03-26T03:16:03.488303+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:16:03.488303+0000","last_undegraded":"2024-03-26T03:16:03.488303+0000","last_fullsized":"2024-03-26T03:16:03.488303+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.271810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.271810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269179+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":372,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485291+0000","last_change":"2024-03-26T03:14:21.603222+0000","last_active":"2024-03-26T03:16:03.485291+0000","last_peered":"2024-03-26T03:16:03.485291+0000","last_clean":"2024-03-26T03:16:03.485291+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:16:03.485291+0000","last_undegraded":"2024-03-26T03:16:03.485291+0000","last_fullsized":"2024-03-26T03:16:03.485291+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":735,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483815+0000","last_change":"2024-03-26T03:15:02.593108+0000","last_active":"2024-03-26T03:16:03.483815+0000","last_peered":"2024-03-26T03:16:03.483815+0000","last_clean":"2024-03-26T03:16:03.483815+0000","last_became_active":"2024-03-26T03:15:02.577360+0000","last_became_peered":"2024-03-26T03:15:02.577360+0000","last_unstale":"2024-03-26T03:16:03.483815+0000","last_undegraded":"2024-03-26T03:16:03.483815+0000","last_fullsized":"2024-03-26T03:16:03.483815+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'27","last_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:05:37.637787+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":95,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512794+0000","last_change":"2024-03-26T03:14:37.244018+0000","last_active":"2024-03-26T03:16:03.512794+0000","last_peered":"2024-03-26T03:16:03.512794+0000","last_clean":"2024-03-26T03:16:03.512794+0000","last_became_active":"2024-03-26T03:14:37.243678+0000","last_became_peered":"2024-03-26T03:14:37.243678+0000","last_unstale":"2024-03-26T03:16:03.512794+0000","last_undegraded":"2024-03-26T03:16:03.512794+0000","last_fullsized":"2024-03-26T03:16:03.512794+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:18.930944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:12:18.930944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":730,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489389+0000","last_change":"2024-03-26T03:15:02.580228+0000","last_active":"2024-03-26T03:16:03.489389+0000","last_peered":"2024-03-26T03:16:03.489389+0000","last_clean":"2024-03-26T03:16:03.489389+0000","last_became_active":"2024-03-26T03:15:02.579424+0000","last_became_peered":"2024-03-26T03:15:02.579424+0000","last_unstale":"2024-03-26T03:16:03.489389+0000","last_undegraded":"2024-03-26T03:16:03.489389+0000","last_fullsized":"2024-03-26T03:16:03.489389+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:12:34.667060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.667060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.633551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":166,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512817+0000","last_change":"2024-03-26T03:15:02.570245+0000","last_active":"2024-03-26T03:16:03.512817+0000","last_peered":"2024-03-26T03:16:03.512817+0000","last_clean":"2024-03-26T03:16:03.512817+0000","last_became_active":"2024-03-26T03:15:02.569812+0000","last_became_peered":"2024-03-26T03:15:02.569812+0000","last_unstale":"2024-03-26T03:16:03.512817+0000","last_undegraded":"2024-03-26T03:16:03.512817+0000","last_fullsized":"2024-03-26T03:16:03.512817+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.660527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.660527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.863118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":429,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484121+0000","last_change":"2024-03-26T03:15:39.820123+0000","last_active":"2024-03-26T03:16:03.484121+0000","last_peered":"2024-03-26T03:16:03.484121+0000","last_clean":"2024-03-26T03:16:03.484121+0000","last_became_active":"2024-03-26T03:15:39.819755+0000","last_became_peered":"2024-03-26T03:15:39.819755+0000","last_unstale":"2024-03-26T03:16:03.484121+0000","last_undegraded":"2024-03-26T03:16:03.484121+0000","last_fullsized":"2024-03-26T03:16:03.484121+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.305762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.305762+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.826173+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":671,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.485421+0000","last_change":"2024-03-26T03:15:02.542849+0000","last_active":"2024-03-26T03:16:03.485421+0000","last_peered":"2024-03-26T03:16:03.485421+0000","last_clean":"2024-03-26T03:16:03.485421+0000","last_became_active":"2024-03-26T03:15:02.535174+0000","last_became_peered":"2024-03-26T03:15:02.535174+0000","last_unstale":"2024-03-26T03:16:03.485421+0000","last_undegraded":"2024-03-26T03:16:03.485421+0000","last_fullsized":"2024-03-26T03:16:03.485421+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'11","last_scrub_stamp":"2024-03-26T03:04:41.069080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:04:41.069080+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":700,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.513174+0000","last_change":"2024-03-26T03:14:37.373867+0000","last_active":"2024-03-26T03:16:03.513174+0000","last_peered":"2024-03-26T03:16:03.513174+0000","last_clean":"2024-03-26T03:16:03.513174+0000","last_became_active":"2024-03-26T03:14:37.373600+0000","last_became_peered":"2024-03-26T03:14:37.373600+0000","last_unstale":"2024-03-26T03:16:03.513174+0000","last_undegraded":"2024-03-26T03:16:03.513174+0000","last_fullsized":"2024-03-26T03:16:03.513174+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:16.999769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:11:16.999769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.917069+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":96,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.489023+0000","last_change":"2024-03-26T03:14:37.254552+0000","last_active":"2024-03-26T03:16:03.489023+0000","last_peered":"2024-03-26T03:16:03.489023+0000","last_clean":"2024-03-26T03:16:03.489023+0000","last_became_active":"2024-03-26T03:14:37.253707+0000","last_became_peered":"2024-03-26T03:14:37.253707+0000","last_unstale":"2024-03-26T03:16:03.489023+0000","last_undegraded":"2024-03-26T03:16:03.489023+0000","last_fullsized":"2024-03-26T03:16:03.489023+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:48.834753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:11:48.834753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":686,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488908+0000","last_change":"2024-03-26T03:14:33.772956+0000","last_active":"2024-03-26T03:16:03.488908+0000","last_peered":"2024-03-26T03:16:03.488908+0000","last_clean":"2024-03-26T03:16:03.488908+0000","last_became_active":"2024-03-26T03:14:21.596806+0000","last_became_peered":"2024-03-26T03:14:21.596806+0000","last_unstale":"2024-03-26T03:16:03.488908+0000","last_undegraded":"2024-03-26T03:16:03.488908+0000","last_fullsized":"2024-03-26T03:16:03.488908+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:33.772842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:33.772842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.634036+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":490,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514211+0000","last_change":"2024-03-26T03:15:39.825036+0000","last_active":"2024-03-26T03:16:03.514211+0000","last_peered":"2024-03-26T03:16:03.514211+0000","last_clean":"2024-03-26T03:16:03.514211+0000","last_became_active":"2024-03-26T03:15:39.824448+0000","last_became_peered":"2024-03-26T03:15:39.824448+0000","last_unstale":"2024-03-26T03:16:03.514211+0000","last_undegraded":"2024-03-26T03:16:03.514211+0000","last_fullsized":"2024-03-26T03:16:03.514211+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:18.946791+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:13:18.946791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.862679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":737,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.487673+0000","last_change":"2024-03-26T03:11:41.830501+0000","last_active":"2024-03-26T03:16:03.487673+0000","last_peered":"2024-03-26T03:16:03.487673+0000","last_clean":"2024-03-26T03:16:03.487673+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:16:03.487673+0000","last_undegraded":"2024-03-26T03:16:03.487673+0000","last_fullsized":"2024-03-26T03:16:03.487673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:11:41.830357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:11:41.830357+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268846+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":381,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.487195+0000","last_change":"2024-03-26T03:14:22.572664+0000","last_active":"2024-03-26T03:16:03.487195+0000","last_peered":"2024-03-26T03:16:03.487195+0000","last_clean":"2024-03-26T03:16:03.487195+0000","last_became_active":"2024-03-26T03:14:22.571857+0000","last_became_peered":"2024-03-26T03:14:22.571857+0000","last_unstale":"2024-03-26T03:16:03.487195+0000","last_undegraded":"2024-03-26T03:16:03.487195+0000","last_fullsized":"2024-03-26T03:16:03.487195+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":712,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.514597+0000","last_change":"2024-03-26T03:15:20.809430+0000","last_active":"2024-03-26T03:16:03.514597+0000","last_peered":"2024-03-26T03:16:03.514597+0000","last_clean":"2024-03-26T03:16:03.514597+0000","last_became_active":"2024-03-26T03:14:37.242540+0000","last_became_peered":"2024-03-26T03:14:37.242540+0000","last_unstale":"2024-03-26T03:16:03.514597+0000","last_undegraded":"2024-03-26T03:16:03.514597+0000","last_fullsized":"2024-03-26T03:16:03.514597+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:20.809176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:20.809176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:41.279137+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":699,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.488871+0000","last_change":"2024-03-26T03:13:32.119587+0000","last_active":"2024-03-26T03:16:03.488871+0000","last_peered":"2024-03-26T03:16:03.488871+0000","last_clean":"2024-03-26T03:16:03.488871+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:16:03.488871+0000","last_undegraded":"2024-03-26T03:16:03.488871+0000","last_fullsized":"2024-03-26T03:16:03.488871+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:32.119468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:13:32.119468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.269095+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":395,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512609+0000","last_change":"2024-03-26T03:14:40.174890+0000","last_active":"2024-03-26T03:16:03.512609+0000","last_peered":"2024-03-26T03:16:03.512609+0000","last_clean":"2024-03-26T03:16:03.512609+0000","last_became_active":"2024-03-26T03:14:40.174111+0000","last_became_peered":"2024-03-26T03:14:40.174111+0000","last_unstale":"2024-03-26T03:16:03.512609+0000","last_undegraded":"2024-03-26T03:16:03.512609+0000","last_fullsized":"2024-03-26T03:16:03.512609+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:22.084825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:22.084825+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:05.942295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8945,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":3,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":470,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.483342+0000","last_change":"2024-03-26T03:14:22.625030+0000","last_active":"2024-03-26T03:16:03.483342+0000","last_peered":"2024-03-26T03:16:03.483342+0000","last_clean":"2024-03-26T03:16:03.483342+0000","last_became_active":"2024-03-26T03:14:22.576815+0000","last_became_peered":"2024-03-26T03:14:22.576815+0000","last_unstale":"2024-03-26T03:16:03.483342+0000","last_undegraded":"2024-03-26T03:16:03.483342+0000","last_fullsized":"2024-03-26T03:16:03.483342+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":65,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":99,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.490467+0000","last_change":"2024-03-26T03:14:37.249386+0000","last_active":"2024-03-26T03:16:03.490467+0000","last_peered":"2024-03-26T03:16:03.490467+0000","last_clean":"2024-03-26T03:16:03.490467+0000","last_became_active":"2024-03-26T03:14:37.248588+0000","last_became_peered":"2024-03-26T03:14:37.248588+0000","last_unstale":"2024-03-26T03:16:03.490467+0000","last_undegraded":"2024-03-26T03:16:03.490467+0000","last_fullsized":"2024-03-26T03:16:03.490467+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:11:51.537921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:11:51.537921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:07.268860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":75,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.512497+0000","last_change":"2024-03-26T03:15:02.574553+0000","last_active":"2024-03-26T03:16:03.512497+0000","last_peered":"2024-03-26T03:16:03.512497+0000","last_clean":"2024-03-26T03:16:03.512497+0000","last_became_active":"2024-03-26T03:15:02.574237+0000","last_became_peered":"2024-03-26T03:15:02.574237+0000","last_unstale":"2024-03-26T03:16:03.512497+0000","last_undegraded":"2024-03-26T03:16:03.512497+0000","last_fullsized":"2024-03-26T03:16:03.512497+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:15.111677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.111677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:16:04.918441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":373,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-03-26T03:16:03.484227+0000","last_change":"2024-03-26T03:14:37.269042+0000","last_active":"2024-03-26T03:16:03.484227+0000","last_peered":"2024-03-26T03:16:03.484227+0000","last_clean":"2024-03-26T03:16:03.484227+0000","last_became_active":"2024-03-26T03:14:37.265472+0000","last_became_peered":"2024-03-26T03:14:37.265472+0000","last_unstale":"2024-03-26T03:16:03.484227+0000","last_undegraded":"2024-03-26T03:16:03.484227+0000","last_fullsized":"2024-03-26T03:16:03.484227+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1655111704,"num_objects":25256,"num_object_clones":0,"num_object_copies":75768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":1610,"num_objects_unfound":0,"num_objects_dirty":25256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224034,"num_write_kb":7977217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90115,"num_bytes_recovered":5905580056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7522500608,"data_stored":7522484320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62036,"ondisk_log_size":62036,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":869303,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":74,"num_write":44,"num_write_kb":603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1033143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3968,"ondisk_log_size":3968,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1022,"num_read_kb":1017,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1401,"num_read_kb":1126,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":3124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430423,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1402796,"kb_used_data":798180,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92342356,"statfs":{"total":95995035648,"available":94558572544,"internally_reserved":0,"allocated":817336320,"data_stored":816160222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":619051462},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1416004,"kb_used_data":612924,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":92329148,"statfs":{"total":95995035648,"available":94545047552,"internally_reserved":0,"allocated":627634176,"data_stored":626434513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738613,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411562,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1659688,"kb_used_data":938528,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92085464,"statfs":{"total":95995035648,"available":94295515136,"internally_reserved":0,"allocated":961052672,"data_stored":959881074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634174999,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305728,"kb_used_data":2082360,"kb_used_omap":4,"kb_used_meta":223355,"kb_avail":91439424,"statfs":{"total":95995035648,"available":93633970176,"internally_reserved":0,"allocated":2132336640,"data_stored":2131139576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":228716195},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986401,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1581144,"kb_used_data":928016,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92164008,"statfs":{"total":95995035648,"available":94375944192,"internally_reserved":0,"allocated":950288384,"data_stored":949131194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692061,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1632944,"kb_used_data":817384,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92112208,"statfs":{"total":95995035648,"available":94322900992,"internally_reserved":0,"allocated":837001216,"data_stored":835820819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2039549952,"data_stored":2039545880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":609157120,"data_stored":609157120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":907476992,"data_stored":907476992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":736825344,"data_stored":736821272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":932909056,"data_stored":932904984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":824639488,"data_stored":824639488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":778309632,"data_stored":778305560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":693633024,"data_stored":693633024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:16:14.368 INFO:tasks.thrashosds.thrasher:Setting 1.8 to [3, 5, 4, 2, 0, 1] 2024-03-26T03:16:14.369 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.8', '3', '5', '4', '2', '0', '1'] 2024-03-26T03:16:14.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 1.8 3 5 4 2 0 1 2024-03-26T03:16:14.400 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:14.456 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:14.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:14.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:14.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:14 smithi066 ceph-mon[29666]: from='client.25711 v1:172.21.15.66:0/3785285683' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:16:14.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:14 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 776/76458 objects degraded (1.015%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:14.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:14 smithi066 ceph-mon[37920]: from='client.25711 v1:172.21.15.66:0/3785285683' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:16:14.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:14 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 776/76458 objects degraded (1.015%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:14.692 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:14.732 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:14.834 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:14.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:14.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:14 smithi192 ceph-mon[33110]: from='client.25711 v1:172.21.15.66:0/3785285683' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:16:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:14 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 776/76458 objects degraded (1.015%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:14.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:15.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:15.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:15.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:15.200 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:16:15.269 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.317 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:15.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:15.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:15.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:15.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:15.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:15.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:15.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:15.828 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:15.857 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:15.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:15 smithi066 ceph-mon[29666]: pgmap v807: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s; 776/75099 objects degraded (1.033%); 1610/75099 objects misplaced (2.144%); 2.2 MiB/s, 34 objects/s recovering 2024-03-26T03:16:15.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:15 smithi066 ceph-mon[37920]: pgmap v807: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s; 776/75099 objects degraded (1.033%); 1610/75099 objects misplaced (2.144%); 2.2 MiB/s, 34 objects/s recovering 2024-03-26T03:16:15.960 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:15.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:15 smithi192 ceph-mon[33110]: pgmap v807: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.5 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 959 op/s; 776/75099 objects degraded (1.033%); 1610/75099 objects misplaced (2.144%); 2.2 MiB/s, 34 objects/s recovering 2024-03-26T03:16:15.988 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:15.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:16.089 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:16.107 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:16.107 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.119 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:16.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:16.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:16.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:16.346 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:16.374 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:16.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.504 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:16.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:16.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:16.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:16.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:16.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:16 smithi066 ceph-mon[29666]: osdmap e387: 8 total, 8 up, 6 in 2024-03-26T03:16:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:16 smithi066 ceph-mon[29666]: pgmap v809: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 44/55296 objects degraded (0.080%); 1210/55296 objects misplaced (2.188%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T03:16:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:16 smithi066 ceph-mon[37920]: osdmap e387: 8 total, 8 up, 6 in 2024-03-26T03:16:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:16 smithi066 ceph-mon[37920]: pgmap v809: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 44/55296 objects degraded (0.080%); 1210/55296 objects misplaced (2.188%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T03:16:16.879 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:16.928 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:16.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:16 smithi192 ceph-mon[33110]: osdmap e387: 8 total, 8 up, 6 in 2024-03-26T03:16:16.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:16 smithi192 ceph-mon[33110]: pgmap v809: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 44/55296 objects degraded (0.080%); 1210/55296 objects misplaced (2.188%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T03:16:17.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.070 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:17.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:17.171 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:17.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:17.313 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.354 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:17.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:17.455 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.511 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:17.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:17.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:16:17.612 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:17.696 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:17 smithi066 ceph-mon[29666]: osdmap e388: 8 total, 8 up, 6 in 2024-03-26T03:16:17.696 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:17 smithi066 ceph-mon[37920]: osdmap e388: 8 total, 8 up, 6 in 2024-03-26T03:16:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.703 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:17.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:17.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:17.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:17.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:17.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:17 smithi192 ceph-mon[33110]: osdmap e388: 8 total, 8 up, 6 in 2024-03-26T03:16:17.995 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:17.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:18.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:18.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:18.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:18.236 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:18.299 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:18.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:124423 10.3 10.bdd83133 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e331)", 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:20.565202+0000", 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 117.745991983, 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0004394190000001, 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.318 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 124423 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565202+0000", 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.319 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565202+0000", 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565204+0000", 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6249999999999999e-06 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565276+0000", 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1457000000000004e-05 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.320 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565286+0000", 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0421000000000001e-05 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565297+0000", 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1032999999999999e-05 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578757+0000", 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013459891999999999 2024-03-26T03:16:18.321 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578765+0000", 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4100000000000008e-06 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565630+0000", 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98686430599999997 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.322 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565641+0000", 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1275e-05 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:124424 10.3 10.9f3a05a3 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e331)", 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:20.565322+0000", 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 117.74587195300001, 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0003774160000001, 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:16:18.323 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 124424 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565322+0000", 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565322+0000", 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.324 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565323+0000", 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0100000000000003e-07 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565373+0000", 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0773999999999999e-05 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565385+0000", 2024-03-26T03:16:18.325 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1430000000000001e-05 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565392+0000", 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.52e-06 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578791+0000", 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013398558 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.326 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578797+0000", 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2029999999999997e-06 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565691+0000", 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.986893927 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565699+0000", 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1030000000000002e-06 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.327 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198816 10.3 10:c18f3cf7:::benchmark_data_smithi066_148944_object74169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.912953+0000", 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.398240455, 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99607143899999995, 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198816 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.328 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912953+0000", 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912953+0000", 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912956+0000", 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4339999999999999e-06 2024-03-26T03:16:18.329 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912961+0000", 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7570000000000001e-06 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912975+0000", 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4042e-05 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912983+0000", 2024-03-26T03:16:18.330 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3769999999999998e-06 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.913057+0000", 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3811000000000002e-05 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.913066+0000", 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5920000000000007e-06 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.895282+0000", 2024-03-26T03:16:18.331 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98221624799999996 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.895410+0000", 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012716499999999999 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.895706+0000", 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00029644900000000001 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.332 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904010+0000", 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0083038139999999996 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904057+0000", 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7370999999999998e-05 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.908933+0000", 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004875379 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.333 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.908988+0000", 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5380000000000002e-05 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909025+0000", 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6619999999999998e-05 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198822 10.3 10:c95efd72:::benchmark_data_smithi066_148944_object74175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.920307+0000", 2024-03-26T03:16:18.334 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.390886164999998, 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98900118000000004, 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198822 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920307+0000", 2024-03-26T03:16:18.335 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920307+0000", 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920309+0000", 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920314+0000", 2024-03-26T03:16:18.336 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6290000000000001e-06 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920323+0000", 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8859999999999993e-06 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920329+0000", 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4320000000000004e-06 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.337 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920387+0000", 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7651000000000001e-05 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920395+0000", 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3510000000000002e-06 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904091+0000", 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.983696126 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.338 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904231+0000", 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013951499999999999 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904352+0000", 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012101 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.908917+0000", 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0045653120000000002 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.339 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909121+0000", 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020395100000000001 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909200+0000", 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9015000000000002e-05 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909259+0000", 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8956000000000003e-05 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.340 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909309+0000", 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9746000000000001e-05 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198826 10.3 10:cbb6a365:::benchmark_data_smithi066_148944_object74179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.922987+0000", 2024-03-26T03:16:18.341 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.388206580000002, 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98656156500000003, 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198826 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922987+0000", 2024-03-26T03:16:18.342 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922987+0000", 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922989+0000", 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6270000000000001e-06 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.343 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922994+0000", 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7690000000000002e-06 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923004+0000", 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9180000000000006e-06 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923012+0000", 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7510000000000008e-06 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.344 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923946+0000", 2024-03-26T03:16:18.345 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00093349999999999998 2024-03-26T03:16:18.345 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.345 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.345 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.345 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923956+0000", 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0709e-05 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.905833+0000", 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.981876741 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.905933+0000", 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9852999999999994e-05 2024-03-26T03:16:18.346 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.906046+0000", 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000112899 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909072+0000", 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030257830000000002 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909442+0000", 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00037073599999999998 2024-03-26T03:16:18.347 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909458+0000", 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5773999999999998e-05 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909506+0000", 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7905000000000003e-05 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.348 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909549+0000", 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2599999999999999e-05 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198829 10.3 10:cb612e94:::benchmark_data_smithi066_148944_object74182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.924268+0000", 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.386925177999998, 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99669433600000001, 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.349 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198829 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924268+0000", 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.350 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924268+0000", 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924270+0000", 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.176e-06 2024-03-26T03:16:18.351 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.353 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924284+0000", 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4351e-05 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924292+0000", 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4840000000000004e-06 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924299+0000", 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0480000000000001e-06 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.354 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.927171+0000", 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028710929999999999 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.927180+0000", 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1689999999999998e-06 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.907599+0000", 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98041955000000003 2024-03-26T03:16:18.355 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.907696+0000", 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6552000000000007e-05 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.907762+0000", 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6438000000000003e-05 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.912805+0000", 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0050422770000000004 2024-03-26T03:16:18.356 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.912851+0000", 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6381999999999997e-05 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920853+0000", 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0080025549999999997 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920913+0000", 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9673999999999998e-05 2024-03-26T03:16:18.357 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920963+0000", 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9586999999999998e-05 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198856 10.3 10:ca535049:::benchmark_data_smithi066_148944_object74209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.951098+0000", 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.360095991000001, 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97358442499999998, 2024-03-26T03:16:18.358 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198856 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951098+0000", 2024-03-26T03:16:18.359 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951098+0000", 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951099+0000", 2024-03-26T03:16:18.360 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9120000000000001e-06 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951105+0000", 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8359999999999997e-06 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951120+0000", 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4851000000000001e-05 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951128+0000", 2024-03-26T03:16:18.361 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8420000000000005e-06 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951192+0000", 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3709999999999998e-05 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951201+0000", 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7070000000000004e-06 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.362 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909662+0000", 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95846033399999997 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909784+0000", 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012201699999999999 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909932+0000", 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014834599999999999 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.363 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.915982+0000", 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0060503120000000004 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923190+0000", 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0072076320000000003 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924590+0000", 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013998330000000001 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.364 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924639+0000", 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8955000000000001e-05 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924682+0000", 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3137999999999999e-05 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198874 10.3 10:c46ca9df:::benchmark_data_smithi066_148944_object74227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.365 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.964061+0000", 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.347132371999997, 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96077497499999998, 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198874 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964061+0000", 2024-03-26T03:16:18.366 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964061+0000", 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964063+0000", 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.125e-06 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964070+0000", 2024-03-26T03:16:18.367 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7950000000000002e-06 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964083+0000", 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3063e-05 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.368 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964092+0000", 2024-03-26T03:16:18.369 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0620000000000007e-06 2024-03-26T03:16:18.369 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.369 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.369 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964171+0000", 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8604999999999997e-05 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964181+0000", 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9119999999999993e-06 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.911186+0000", 2024-03-26T03:16:18.370 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.94700484399999996 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.911293+0000", 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000107865 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.911405+0000", 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011191 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.371 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923197+0000", 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011791585 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924720+0000", 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001523429 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924758+0000", 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7191999999999999e-05 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.372 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924802+0000", 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4503999999999999e-05 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924836+0000", 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4084000000000001e-05 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.373 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198895 10.3 10:c9006454:::benchmark_data_smithi066_148944_object74248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.990762+0000", 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.320431065000001, 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93426770699999995, 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198895 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.374 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990762+0000", 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990762+0000", 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990765+0000", 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1449999999999998e-06 2024-03-26T03:16:18.375 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990770+0000", 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8359999999999997e-06 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990784+0000", 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3998e-05 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990793+0000", 2024-03-26T03:16:18.376 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3909999999999998e-06 2024-03-26T03:16:18.377 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.377 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.377 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.377 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990853+0000", 2024-03-26T03:16:18.377 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0543e-05 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990862+0000", 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0089999999999996e-06 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.912898+0000", 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.922035194 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.378 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.913027+0000", 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012908 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.913163+0000", 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013621799999999999 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923202+0000", 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010039078 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.379 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924872+0000", 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001669554 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924947+0000", 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5858000000000005e-05 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924993+0000", 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5778000000000001e-05 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.380 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925030+0000", 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7024999999999999e-05 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198918 10.3 10:cc0a2071:::benchmark_data_smithi066_148944_object74271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.009004+0000", 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.302189374999998, 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91617897400000003, 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.381 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198918 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009004+0000", 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009004+0000", 2024-03-26T03:16:18.382 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009006+0000", 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2510000000000001e-06 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009013+0000", 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.041e-06 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009028+0000", 2024-03-26T03:16:18.383 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4673e-05 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009038+0000", 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0336999999999999e-05 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009098+0000", 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9861999999999997e-05 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.384 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009109+0000", 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0465000000000001e-05 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.914465+0000", 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.90535576699999998 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.914578+0000", 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113625 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.385 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.914697+0000", 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118663 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923207+0000", 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0085102230000000008 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925068+0000", 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018610460000000001 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.386 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925106+0000", 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7373999999999998e-05 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925147+0000", 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1285000000000002e-05 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925183+0000", 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6362000000000002e-05 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.387 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198919 10.3 10:c4ec0153:::benchmark_data_smithi066_148944_object74272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.010698+0000", 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.300495314000003, 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91463146900000003, 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198919 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.388 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010698+0000", 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010698+0000", 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010700+0000", 2024-03-26T03:16:18.389 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6130000000000001e-06 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010705+0000", 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3070000000000002e-06 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010716+0000", 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0533000000000001e-05 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.390 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010723+0000", 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5669999999999997e-06 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010781+0000", 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8150999999999999e-05 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.010790+0000", 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5590000000000006e-06 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.391 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.916023+0000", 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.90523299000000002 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.916119+0000", 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5909999999999995e-05 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.916234+0000", 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011520500000000001 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.392 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923212+0000", 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069780720000000001 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925219+0000", 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020066569999999998 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925254+0000", 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.5055999999999998e-05 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.393 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925298+0000", 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3689999999999997e-05 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925330+0000", 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2159000000000002e-05 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198945 10.3 10:c758caba:::benchmark_data_smithi066_148944_object74298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.039200+0000", 2024-03-26T03:16:18.394 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.271993137999999, 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.88989614399999994, 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198945 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039200+0000", 2024-03-26T03:16:18.395 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039200+0000", 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039203+0000", 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.5019999999999999e-06 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039209+0000", 2024-03-26T03:16:18.396 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1310000000000003e-06 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039224+0000", 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5102e-05 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039233+0000", 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5199999999999997e-06 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.397 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039298+0000", 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5411000000000001e-05 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.039308+0000", 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9760000000000007e-06 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.917572+0000", 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.87826442400000004 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.398 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.917679+0000", 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010602 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.917784+0000", 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000105708 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925365+0000", 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0075811799999999999 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.399 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925401+0000", 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.5182999999999998e-05 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929002+0000", 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036016580000000002 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929052+0000", 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.994e-05 2024-03-26T03:16:18.400 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929097+0000", 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4388999999999998e-05 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198949 10.3 10:c7fa8b40:::benchmark_data_smithi066_148944_object74302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.042206+0000", 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.268988002999997, 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.88696962899999998, 2024-03-26T03:16:18.401 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198949 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042206+0000", 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.402 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042206+0000", 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042208+0000", 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.029e-06 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042222+0000", 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4778e-05 2024-03-26T03:16:18.403 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.404 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.404 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042255+0000", 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2645000000000003e-05 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042293+0000", 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7731000000000002e-05 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.405 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042501+0000", 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020845799999999999 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.042521+0000", 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9817e-05 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.919197+0000", 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.87667641299999999 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.406 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.919302+0000", 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010431500000000001 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.919435+0000", 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013336700000000001 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925438+0000", 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0060025260000000002 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.407 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925472+0000", 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4588999999999997e-05 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929104+0000", 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036316690000000001 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929141+0000", 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7270000000000001e-05 2024-03-26T03:16:18.408 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929175+0000", 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4022000000000001e-05 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198957 10.3 10:ce5ac58c:::benchmark_data_smithi066_148944_object74310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.054132+0000", 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.257061123, 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.87517225399999998, 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.409 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198957 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054132+0000", 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.410 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054132+0000", 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054135+0000", 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0810000000000001e-06 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054142+0000", 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3880000000000002e-06 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.411 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054151+0000", 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4469999999999995e-06 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054158+0000", 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9310000000000002e-06 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054240+0000", 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1707999999999998e-05 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.412 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.054250+0000", 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0020999999999999e-05 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920993+0000", 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.86674263399999996 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.413 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.921093+0000", 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100203 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.921204+0000", 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000110931 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925507+0000", 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043031279999999998 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.414 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925543+0000", 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6118999999999998e-05 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929236+0000", 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036925769999999998 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929272+0000", 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6089e-05 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929305+0000", 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.2997000000000003e-05 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198967 10.3 10:c38e5238:::benchmark_data_smithi066_148944_object74320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.065696+0000", 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.245497565999997, 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.86368315900000003, 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.416 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198967 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065696+0000", 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.417 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065696+0000", 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065703+0000", 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4540000000000001e-06 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065709+0000", 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7830000000000004e-06 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.418 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065719+0000", 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0030000000000001e-05 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065727+0000", 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8669999999999994e-06 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065799+0000", 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2082999999999994e-05 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.419 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.065808+0000", 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2010000000000005e-06 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923249+0000", 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.85744058499999998 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923323+0000", 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3584000000000006e-05 2024-03-26T03:16:18.420 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923406+0000", 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3601000000000001e-05 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925581+0000", 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002175125 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929224+0000", 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003642251 2024-03-26T03:16:18.421 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929310+0000", 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6687999999999994e-05 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929346+0000", 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.5818999999999997e-05 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929379+0000", 2024-03-26T03:16:18.422 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3087999999999999e-05 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198973 10.3 10:c619705c:::benchmark_data_smithi066_148944_object74326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.075602+0000", 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.235591552000002, 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.86293790100000001, 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.423 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198973 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075602+0000", 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075602+0000", 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.424 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075604+0000", 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8360000000000001e-06 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075610+0000", 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8540000000000002e-06 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075622+0000", 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2765000000000001e-05 2024-03-26T03:16:18.425 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075630+0000", 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.294e-06 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075687+0000", 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7037000000000003e-05 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.426 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.075702+0000", 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5158e-05 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925609+0000", 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.84990753699999999 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925717+0000", 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000107447 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.427 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.926005+0000", 2024-03-26T03:16:18.428 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00028779400000000002 2024-03-26T03:16:18.428 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.428 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.428 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.428 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929385+0000", 2024-03-26T03:16:18.428 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0033801130000000001 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.932990+0000", 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003605249 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.938459+0000", 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0054686259999999999 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.429 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.938492+0000", 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3599999999999997e-05 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.938540+0000", 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7590999999999999e-05 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198992 10.3 10:c3e1c5e0:::benchmark_data_smithi066_148944_object74345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.114686+0000", 2024-03-26T03:16:18.430 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.196507678000003, 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.82696633799999997, 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198992 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114686+0000", 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.431 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114686+0000", 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.432 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114688+0000", 2024-03-26T03:16:18.433 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.905e-06 2024-03-26T03:16:18.433 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.433 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.433 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.433 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114693+0000", 2024-03-26T03:16:18.433 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0529999999999997e-06 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114706+0000", 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2677000000000001e-05 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114714+0000", 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8920000000000006e-06 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114765+0000", 2024-03-26T03:16:18.434 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0587000000000002e-05 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.114772+0000", 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9210000000000003e-06 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.927359+0000", 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.81258694200000003 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.435 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.927469+0000", 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000110145 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.927588+0000", 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000119339 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.931212+0000", 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036239589999999999 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.436 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.936813+0000", 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0056007469999999997 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.941543+0000", 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047294579999999998 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.941607+0000", 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4003000000000004e-05 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.437 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.941652+0000", 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5710000000000001e-05 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:199014 10.3 10:c0cf2521:::benchmark_data_smithi066_148944_object74367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.238404+0000", 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 34.072789405999998, 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.70339241399999997, 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.438 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 199014 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238404+0000", 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238404+0000", 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.439 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238406+0000", 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6700000000000001e-06 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238411+0000", 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5890000000000002e-06 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238425+0000", 2024-03-26T03:16:18.440 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3957999999999999e-05 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238432+0000", 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.08e-06 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238965+0000", 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00053244299999999996 2024-03-26T03:16:18.441 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.238976+0000", 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0733e-05 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929412+0000", 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.69043608300000003 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929515+0000", 2024-03-26T03:16:18.442 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000103105 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.929649+0000", 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013408800000000001 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.933003+0000", 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003353642 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.938256+0000", 2024-03-26T03:16:18.443 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0052533980000000003 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.941710+0000", 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003453813 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.941756+0000", 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6492999999999997e-05 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.444 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.941797+0000", 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0318999999999998e-05 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:199317 10.3 10:c9f1079d:::benchmark_data_smithi066_148944_object74670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:45.976777+0000", 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 32.334416509999997, 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.099404300000000001, 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.445 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 199317 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976777+0000", 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976777+0000", 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.446 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976779+0000", 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2330000000000001e-06 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976786+0000", 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7159999999999997e-06 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976801+0000", 2024-03-26T03:16:18.447 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4541e-05 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976809+0000", 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1200000000000002e-06 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976863+0000", 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4693e-05 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.976997+0000", 2024-03-26T03:16:18.448 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000134141 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.977111+0000", 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113138 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.980600+0000", 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0034895690000000001 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:45.980626+0000", 2024-03-26T03:16:18.449 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6098999999999999e-05 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:46.076061+0000", 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.095434911999999997 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:46.076128+0000", 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6840000000000004e-05 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.450 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:46.076181+0000", 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3297999999999997e-05 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:206969 10.3 10:c893e739:::benchmark_data_smithi066_148944_object82322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:53.786592+0000", 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 24.524601865000001, 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.113187539, 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:18.451 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 206969 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786592+0000", 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786592+0000", 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.452 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786593+0000", 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.686e-06 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786599+0000", 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.22e-06 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786609+0000", 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0207000000000001e-05 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.453 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786616+0000", 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.148e-06 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786661+0000", 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5330999999999997e-05 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786779+0000", 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118112 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.454 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.786899+0000", 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000119665 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.788425+0000", 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0015258190000000001 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.789018+0000", 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00059349900000000004 2024-03-26T03:16:18.455 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.899496+0000", 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11047765399999999 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.899725+0000", 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000229445 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:53.899779+0000", 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3752999999999998e-05 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.456 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.457 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:18.457 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:18.457 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:16:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:18.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:18.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:18.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:18.569 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:16:18.618 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:16:18.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:18.619 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:18.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:18.657 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:16:18.700 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:16:18.721 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:18.744 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:16:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:18.765 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:18.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:18.784 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:16:18.825 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:16:18.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:18 smithi066 ceph-mon[29666]: pgmap v811: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 744 B/s rd, 0 B/s wr, 1.03k op/s; 44/55296 objects degraded (0.080%); 1210/55296 objects misplaced (2.188%); 5.6 MiB/s, 89 objects/s recovering 2024-03-26T03:16:18.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:18 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/523095955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:18.826 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:18 smithi066 ceph-mon[37920]: pgmap v811: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 744 B/s rd, 0 B/s wr, 1.03k op/s; 44/55296 objects degraded (0.080%); 1210/55296 objects misplaced (2.188%); 5.6 MiB/s, 89 objects/s recovering 2024-03-26T03:16:18.826 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:18 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/523095955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:18.864 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:16:18.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:18.873 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:16:18.874 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:16:18.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:18.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:18 smithi192 ceph-mon[33110]: pgmap v811: 178 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 176 active+clean; 1.1 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 744 B/s rd, 0 B/s wr, 1.03k op/s; 44/55296 objects degraded (0.080%); 1210/55296 objects misplaced (2.188%); 5.6 MiB/s, 89 objects/s recovering 2024-03-26T03:16:18.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:18 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/523095955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:19.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.049 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:19.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:19.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:19.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:19.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:19.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:19.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:19.475 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:19.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:19.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:19.717 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:19.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:19.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:19 smithi066 ceph-mon[29666]: Health check update: Degraded data redundancy: 44/55296 objects degraded (0.080%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:19 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:16:19.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:19 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:16:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:19 smithi066 ceph-mon[37920]: Health check update: Degraded data redundancy: 44/55296 objects degraded (0.080%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:19 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:16:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:19 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:16:19.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:19.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:19.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:19.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:19 smithi192 ceph-mon[33110]: Health check update: Degraded data redundancy: 44/55296 objects degraded (0.080%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:16:19.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:19 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:16:19.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:19 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:16:20.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:20.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:20.142 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:20.186 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:20.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:16:19 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:16:20.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:20.320 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:16:20.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:20.424 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:20.468 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:20.570 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:20.614 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:20.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:20.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:20.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:20.903 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:20.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:20 smithi192 ceph-mon[33110]: osdmap e389: 8 total, 8 up, 6 in 2024-03-26T03:16:20.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:20 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:20.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:20 smithi192 ceph-mon[33110]: pgmap v813: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 759 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.11k op/s; 868/37098 objects misplaced (2.340%); 6.9 MiB/s, 109 objects/s recovering 2024-03-26T03:16:21.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:21.052 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:21.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:20 smithi066 ceph-mon[29666]: osdmap e389: 8 total, 8 up, 6 in 2024-03-26T03:16:21.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:20 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:21.097 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:20 smithi066 ceph-mon[29666]: pgmap v813: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 759 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.11k op/s; 868/37098 objects misplaced (2.340%); 6.9 MiB/s, 109 objects/s recovering 2024-03-26T03:16:21.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:20 smithi066 ceph-mon[37920]: osdmap e389: 8 total, 8 up, 6 in 2024-03-26T03:16:21.098 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:20 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:21.098 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:20 smithi066 ceph-mon[37920]: pgmap v813: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 759 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.11k op/s; 868/37098 objects misplaced (2.340%); 6.9 MiB/s, 109 objects/s recovering 2024-03-26T03:16:21.154 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.198 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:21.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:21.299 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.361 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:21.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:21.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:21.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:21.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:21.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:21.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:21.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:21.877 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:21.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:21.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:21 smithi192 ceph-mon[33110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 44/55296 objects degraded (0.080%), 1 pg degraded, 1 pg undersized) 2024-03-26T03:16:22.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.061 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:22.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:22.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:21 smithi066 ceph-mon[29666]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 44/55296 objects degraded (0.080%), 1 pg degraded, 1 pg undersized) 2024-03-26T03:16:22.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:21 smithi066 ceph-mon[37920]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 44/55296 objects degraded (0.080%), 1 pg degraded, 1 pg undersized) 2024-03-26T03:16:22.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:22.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:22.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:22.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:22.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.468 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:22.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:22.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:22.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:22.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:22.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:22.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:22.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:22.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:22.878 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:22 smithi066 ceph-mon[29666]: pgmap v814: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 759 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 750 B/s rd, 0 B/s wr, 889 op/s; 868/37098 objects misplaced (2.340%); 403 KiB/s, 6 objects/s recovering 2024-03-26T03:16:22.879 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:22 smithi066 ceph-mon[37920]: pgmap v814: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 759 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 750 B/s rd, 0 B/s wr, 889 op/s; 868/37098 objects misplaced (2.340%); 403 KiB/s, 6 objects/s recovering 2024-03-26T03:16:22.976 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:22.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:22 smithi192 ceph-mon[33110]: pgmap v814: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 759 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 750 B/s rd, 0 B/s wr, 889 op/s; 868/37098 objects misplaced (2.340%); 403 KiB/s, 6 objects/s recovering 2024-03-26T03:16:23.018 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:23.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:23.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:23.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:23.298 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:23.400 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:23.443 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:23.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:23.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:23.666 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:23.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:23.699 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.744 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:23.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:23.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:23.860 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:16:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:16:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:16:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:23.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:23.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:23.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:24.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:24.118 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:24.161 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:24.263 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:24.309 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:24.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:16:24.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.467 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:24.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:24.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:24.622 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:24.727 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:24.775 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:24.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:24.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:24.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:25.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:25.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:25.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:25.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:25.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:25.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:25.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:25.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:25 smithi192 ceph-mon[33110]: osdmap e390: 8 total, 8 up, 6 in 2024-03-26T03:16:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:25 smithi192 ceph-mon[33110]: pgmap v816: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 730 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 654 B/s rd, 0 B/s wr, 835 op/s; 868/35712 objects misplaced (2.431%); 352 KiB/s, 5 objects/s recovering 2024-03-26T03:16:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:25 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:16:25.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:25 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:16:25.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:25.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[29666]: osdmap e390: 8 total, 8 up, 6 in 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[29666]: pgmap v816: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 730 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 654 B/s rd, 0 B/s wr, 835 op/s; 868/35712 objects misplaced (2.431%); 352 KiB/s, 5 objects/s recovering 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[37920]: osdmap e390: 8 total, 8 up, 6 in 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[37920]: pgmap v816: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 730 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 654 B/s rd, 0 B/s wr, 835 op/s; 868/35712 objects misplaced (2.431%); 352 KiB/s, 5 objects/s recovering 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:16:25.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:25 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:16:25.698 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.755 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:25.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:25.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:25.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:25.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:26.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:26.044 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:26.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:26.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:26.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:26.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:26.422 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.472 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:26.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:26.573 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.622 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:26.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:26.723 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:26.769 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:26.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:26.906 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:26.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:26.974 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:26 smithi066 ceph-mon[29666]: osdmap e391: 8 total, 8 up, 6 in 2024-03-26T03:16:26.974 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:26 smithi066 ceph-mon[29666]: pgmap v818: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 306 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.90k op/s; 360/15324 objects misplaced (2.349%); 406 KiB/s, 6 objects/s recovering 2024-03-26T03:16:26.974 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:26 smithi066 ceph-mon[37920]: osdmap e391: 8 total, 8 up, 6 in 2024-03-26T03:16:26.974 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:26 smithi066 ceph-mon[37920]: pgmap v818: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 306 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.90k op/s; 360/15324 objects misplaced (2.349%); 406 KiB/s, 6 objects/s recovering 2024-03-26T03:16:27.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:27.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:27.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:27.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:26 smithi192 ceph-mon[33110]: osdmap e391: 8 total, 8 up, 6 in 2024-03-26T03:16:27.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:26 smithi192 ceph-mon[33110]: pgmap v818: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 306 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.90k op/s; 360/15324 objects misplaced (2.349%); 406 KiB/s, 6 objects/s recovering 2024-03-26T03:16:27.284 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:27.331 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:27.433 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:27.476 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:27.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:27.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:27.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:27.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:27.754 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.777 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:27.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:27.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:27.891 INFO:teuthology.orchestra.run.smithi066.stderr:set 1.8 pg_upmap_items mapping to [3->5,4->2,0->1] 2024-03-26T03:16:27.891 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1709769761' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.8", "id": [3, 5, 4, 2, 0, 1]}]: dispatch 2024-03-26T03:16:27.891 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:27 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3451599835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:16:27.891 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:27 smithi066 ceph-mon[29666]: from='client.35780 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:16:27.892 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1709769761' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.8", "id": [3, 5, 4, 2, 0, 1]}]: dispatch 2024-03-26T03:16:27.892 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:27 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3451599835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:16:27.892 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:27 smithi066 ceph-mon[37920]: from='client.35780 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:16:27.905 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:16:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:27.936 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:27.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:28.037 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:28.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:28.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Removed 124644 objects 2024-03-26T03:16:28.139 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Clean up completed and total clean up time :123.944 2024-03-26T03:16:28.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.206 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:28.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:28.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1709769761' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.8", "id": [3, 5, 4, 2, 0, 1]}]: dispatch 2024-03-26T03:16:28.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:27 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3451599835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:16:28.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:27 smithi192 ceph-mon[33110]: from='client.35780 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:16:28.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:28.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:28.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:28.560 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:28.588 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:28.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:28.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:28.820 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.850 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:28.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:28.951 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:28.981 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:28.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:29.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.114 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:29.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:29.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:29.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:16:29.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1709769761' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.8", "id": [3, 5, 4, 2, 0, 1]}]': finished 2024-03-26T03:16:29.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: from='client.35780 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:16:29.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: osdmap e392: 8 total, 8 up, 6 in 2024-03-26T03:16:29.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: 1.6 scrub starts 2024-03-26T03:16:29.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: 1.6 scrub ok 2024-03-26T03:16:29.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:28 smithi192 ceph-mon[33110]: pgmap v820: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 306 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 360/15324 objects misplaced (2.349%) 2024-03-26T03:16:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:29.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:29.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:16:29.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1709769761' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.8", "id": [3, 5, 4, 2, 0, 1]}]': finished 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: from='client.35780 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: osdmap e392: 8 total, 8 up, 6 in 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: 1.6 scrub starts 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: 1.6 scrub ok 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[29666]: pgmap v820: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 306 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 360/15324 objects misplaced (2.349%) 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1709769761' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.8", "id": [3, 5, 4, 2, 0, 1]}]': finished 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: from='client.35780 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: osdmap e392: 8 total, 8 up, 6 in 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: 1.6 scrub starts 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: 1.6 scrub ok 2024-03-26T03:16:29.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:28 smithi066 ceph-mon[37920]: pgmap v820: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 306 MiB data, 5.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 360/15324 objects misplaced (2.349%) 2024-03-26T03:16:29.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:29.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:29.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:29.602 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:29.632 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:29.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:29.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:29.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:29.896 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:29.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:29.998 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.031 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:30.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:30.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:30.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:30.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 1.8 scrub starts 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 1.8 scrub ok 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 1.10 scrub starts 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 1.10 scrub ok 2024-03-26T03:16:30.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 2.13 scrub starts 2024-03-26T03:16:30.239 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:29 smithi192 ceph-mon[33110]: 2.13 scrub ok 2024-03-26T03:16:30.262 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:30.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:16:30.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 1.8 scrub starts 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 1.8 scrub ok 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 1.10 scrub starts 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 1.10 scrub ok 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 2.13 scrub starts 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[29666]: 2.13 scrub ok 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 1.8 scrub starts 2024-03-26T03:16:30.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 1.8 scrub ok 2024-03-26T03:16:30.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 1.10 scrub starts 2024-03-26T03:16:30.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 1.10 scrub ok 2024-03-26T03:16:30.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 2.13 scrub starts 2024-03-26T03:16:30.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:29 smithi066 ceph-mon[37920]: 2.13 scrub ok 2024-03-26T03:16:30.397 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:30.430 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:30.532 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:30.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:30.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:30.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:30.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:30.953 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:30.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:31.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:31.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:31.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.1a scrub starts 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.1a scrub ok 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.4 scrub starts 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.4 scrub ok 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.1c scrub starts 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.1c scrub ok 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[29666]: pgmap v821: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 700 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 1.96k op/s 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.1a scrub starts 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.1a scrub ok 2024-03-26T03:16:31.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.4 scrub starts 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.4 scrub ok 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.1c scrub starts 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.1c scrub ok 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:16:31.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:30 smithi066 ceph-mon[37920]: pgmap v821: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 700 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 1.96k op/s 2024-03-26T03:16:31.184 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:31.219 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:31.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.1a scrub starts 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.1a scrub ok 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.4 scrub starts 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.4 scrub ok 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.1c scrub starts 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.1c scrub ok 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:16:31.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:30 smithi192 ceph-mon[33110]: pgmap v821: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 700 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 1.96k op/s 2024-03-26T03:16:31.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:31.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:31.450 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:31.484 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:31.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:31.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:31.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:31.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:31.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:31.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:31.875 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:31.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:31.977 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.008 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:32.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:32.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:32.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:32.142 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:16:32.142 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:16:32.142 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 1.9 scrub starts 2024-03-26T03:16:32.142 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 1.9 scrub ok 2024-03-26T03:16:32.142 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 1.7 scrub starts 2024-03-26T03:16:32.142 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 1.7 scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 1.d scrub starts 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[29666]: 1.d scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 1.9 scrub starts 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 1.9 scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 1.7 scrub starts 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 1.7 scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:16:32.143 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 1.d scrub starts 2024-03-26T03:16:32.144 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:31 smithi066 ceph-mon[37920]: 1.d scrub ok 2024-03-26T03:16:32.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:16:32.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 1.9 scrub starts 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 1.9 scrub ok 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 1.7 scrub starts 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 1.7 scrub ok 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 1.d scrub starts 2024-03-26T03:16:32.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:31 smithi192 ceph-mon[33110]: 1.d scrub ok 2024-03-26T03:16:32.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:32.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:32.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:32.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:32.408 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:16:32.408 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:16:32.477 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-26T03:16:32.477 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-26T03:16:32.495 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.547 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:32.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:32.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.677 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:32.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:32.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:32.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:32.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:32.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:32.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:33.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:33.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:33.166 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:33.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:33.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:32 smithi192 ceph-mon[33110]: 1.e scrub starts 2024-03-26T03:16:33.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:32 smithi192 ceph-mon[33110]: 1.e scrub ok 2024-03-26T03:16:33.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:32 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:16:33.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:32 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:16:33.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:32 smithi192 ceph-mon[33110]: osdmap e393: 8 total, 8 up, 6 in 2024-03-26T03:16:33.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:32 smithi192 ceph-mon[33110]: pgmap v823: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 700 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 744 op/s 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[29666]: 1.e scrub starts 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[29666]: 1.e scrub ok 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[29666]: osdmap e393: 8 total, 8 up, 6 in 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[29666]: pgmap v823: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 700 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 744 op/s 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[37920]: 1.e scrub starts 2024-03-26T03:16:33.241 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[37920]: 1.e scrub ok 2024-03-26T03:16:33.242 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:16:33.242 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:16:33.242 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[37920]: osdmap e393: 8 total, 8 up, 6 in 2024-03-26T03:16:33.242 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:32 smithi066 ceph-mon[37920]: pgmap v823: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 700 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 744 op/s 2024-03-26T03:16:33.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:33.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:33.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:33.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:33.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:33.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:33.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:33.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:33.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:33.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:33.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:33.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:33.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:33.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:34.052 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:16:34.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:34.082 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[29666]: 1.1e scrub starts 2024-03-26T03:16:34.082 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[29666]: 1.1e scrub ok 2024-03-26T03:16:34.082 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:16:34.082 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:16:34.082 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:16:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:16:34.119 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[37920]: 1.1e scrub starts 2024-03-26T03:16:34.120 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[37920]: 1.1e scrub ok 2024-03-26T03:16:34.120 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:16:34.120 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:33 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:16:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.147 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:34.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:34.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:33 smithi192 ceph-mon[33110]: 1.1e scrub starts 2024-03-26T03:16:34.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:33 smithi192 ceph-mon[33110]: 1.1e scrub ok 2024-03-26T03:16:34.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:33 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:16:34.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:33 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:16:34.248 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.277 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:34.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:34.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:34.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:34.509 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.538 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:34.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:34.637 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:16:34.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.667 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:34.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:34.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:34.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:34.869 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:16:34.890 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:16:34.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:34.915 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:16:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:34.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:34.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:34.937 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:34.938 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:34.945 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:16:34.968 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:16:34.990 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:16:35.011 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:16:35.027 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:35.030 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:16:35.032 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:16:35.056 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:35.159 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.187 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:35.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: osdmap e394: 8 total, 8 up, 6 in 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: pgmap v825: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 644 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1981618703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:16:35.221 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:16:35.222 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:16:35.222 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: osdmap e394: 8 total, 8 up, 6 in 2024-03-26T03:16:35.222 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: pgmap v825: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 644 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s 2024-03-26T03:16:35.222 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:34 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1981618703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:35.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:16:35.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:16:35.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:16:35.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:16:35.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: osdmap e394: 8 total, 8 up, 6 in 2024-03-26T03:16:35.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: pgmap v825: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 644 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 745 op/s 2024-03-26T03:16:35.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:34 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1981618703' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:35.288 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.316 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:35.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:35.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:35.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:35.548 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.577 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:35.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:35.678 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:35.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:35.808 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:35.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.836 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:35.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:35.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:35.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:35.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:36.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:36.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:36.197 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:36.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:36.272 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:36 smithi066 ceph-mon[29666]: 4.10 scrub starts 2024-03-26T03:16:36.272 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:36 smithi066 ceph-mon[29666]: 4.10 scrub ok 2024-03-26T03:16:36.273 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:36 smithi066 ceph-mon[37920]: 4.10 scrub starts 2024-03-26T03:16:36.273 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:36 smithi066 ceph-mon[37920]: 4.10 scrub ok 2024-03-26T03:16:36.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.356 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:36.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:36.457 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.485 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:36.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:36 smithi192 ceph-mon[33110]: 4.10 scrub starts 2024-03-26T03:16:36.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:36 smithi192 ceph-mon[33110]: 4.10 scrub ok 2024-03-26T03:16:36.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:36.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:36.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.745 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:36.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:36.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:36.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:36.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:36.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:37.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:37.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:37.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:37.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:37.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:37.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[37920]: 5.1 scrub starts 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[37920]: 5.1 scrub ok 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[37920]: 4.1c scrub starts 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[37920]: 4.1c scrub ok 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[37920]: pgmap v826: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 604 op/s 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[29666]: 5.1 scrub starts 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[29666]: 5.1 scrub ok 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[29666]: 4.1c scrub starts 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[29666]: 4.1c scrub ok 2024-03-26T03:16:37.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:37 smithi066 ceph-mon[29666]: pgmap v826: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 604 op/s 2024-03-26T03:16:37.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.390 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:37.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:37 smithi192 ceph-mon[33110]: 5.1 scrub starts 2024-03-26T03:16:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:37 smithi192 ceph-mon[33110]: 5.1 scrub ok 2024-03-26T03:16:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:37 smithi192 ceph-mon[33110]: 4.1c scrub starts 2024-03-26T03:16:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:37 smithi192 ceph-mon[33110]: 4.1c scrub ok 2024-03-26T03:16:37.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:37 smithi192 ceph-mon[33110]: pgmap v826: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 604 op/s 2024-03-26T03:16:37.492 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.520 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:37.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:37.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:37.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:37.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:37.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:37.876 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:37.904 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:37.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:38.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:38.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:38.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:38.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:38.265 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:38.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:38.397 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.427 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:38.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:38.477 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:38 smithi066 ceph-mon[29666]: 6.5 scrub starts 2024-03-26T03:16:38.477 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:38 smithi066 ceph-mon[29666]: 6.5 scrub ok 2024-03-26T03:16:38.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:38 smithi066 ceph-mon[37920]: 6.5 scrub starts 2024-03-26T03:16:38.477 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:38 smithi066 ceph-mon[37920]: 6.5 scrub ok 2024-03-26T03:16:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:38 smithi192 ceph-mon[33110]: 6.5 scrub starts 2024-03-26T03:16:38.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:38 smithi192 ceph-mon[33110]: 6.5 scrub ok 2024-03-26T03:16:38.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:38.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:38.658 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:38.661 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:38.688 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:38.790 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:38.819 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:38.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:38.938 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:16:38.938 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:16:38.938 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:16:38.938 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:16:38.938 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:16:38.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:38.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:39.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.079 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:39.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:39.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:16:39.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:39.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:39.229 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:16:39.230 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:16:39.230 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T03:16:39.230 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.005167 2024-03-26T03:16:39.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 2 0.0051668981141794745 2024-03-26T03:16:39.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:39.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:39.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:39.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:39.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:39.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:16:39.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:16:39.481 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [1, 6]}]: dispatch 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "1.8"}]: dispatch 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: pgmap v827: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [1, 6]}]: dispatch 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "1.8"}]: dispatch 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: pgmap v827: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:39 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:16:39.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:16:39.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:16:39.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [1, 6]}]: dispatch 2024-03-26T03:16:39.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "1.8"}]: dispatch 2024-03-26T03:16:39.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: pgmap v827: 178 pgs: 1 active+remapped+backfill_toofull, 177 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:16:39.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:16:39.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:39.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:39.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:16:39.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:39.724 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:39.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:39.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:39.955 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:39.989 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:39.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:40.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:40.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:40.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:40.219 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:40.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:16:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: 4.0 scrub starts 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: 4.0 scrub ok 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [1, 6]}]': finished 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "1.8"}]': finished 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: osdmap e395: 8 total, 8 up, 6 in 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: 4.c scrub starts 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: 4.c scrub ok 2024-03-26T03:16:40.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:40.248 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:40.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:40.349 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:40.378 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:40.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:40.480 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:40.483 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: 4.0 scrub starts 2024-03-26T03:16:40.483 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: 4.0 scrub ok 2024-03-26T03:16:40.483 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [1, 6]}]': finished 2024-03-26T03:16:40.483 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "1.8"}]': finished 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: osdmap e395: 8 total, 8 up, 6 in 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: 4.c scrub starts 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: 4.c scrub ok 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: 4.0 scrub starts 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: 4.0 scrub ok 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1d", "id": [1, 6]}]': finished 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "1.8"}]': finished 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: osdmap e395: 8 total, 8 up, 6 in 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: 4.c scrub starts 2024-03-26T03:16:40.484 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: 4.c scrub ok 2024-03-26T03:16:40.485 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:40.510 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:40.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:40.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:40.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:40.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:40.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:40.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:40.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:40.871 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:40.999 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:41.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:41.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:41.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:41.236 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_3' removed 2024-03-26T03:16:41.295 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:41.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: 2.1b scrub starts 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: 2.1b scrub ok 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: osdmap e396: 8 total, 8 up, 6 in 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: pgmap v830: 178 pgs: 1 active+remapped+backfill_toofull, 1 peering, 176 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/319775056' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: 2.1b scrub starts 2024-03-26T03:16:41.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: 2.1b scrub ok 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: osdmap e396: 8 total, 8 up, 6 in 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: pgmap v830: 178 pgs: 1 active+remapped+backfill_toofull, 1 peering, 176 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/319775056' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:16:41.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:41 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:16:41.397 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.438 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:41.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: 2.1b scrub starts 2024-03-26T03:16:41.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: 2.1b scrub ok 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: osdmap e396: 8 total, 8 up, 6 in 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: pgmap v830: 178 pgs: 1 active+remapped+backfill_toofull, 1 peering, 176 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/319775056' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:16:41.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:41 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:16:41.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:41.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:41.664 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:41.692 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:41.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.822 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:41.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:41.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:41.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:41.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:42.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.077 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:42.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:42.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:42.202 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:42.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:42.213 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:16:42.214 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:16:42.214 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-26T03:16:42.214 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-26T03:16:42.313 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:42.344 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:42.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:42.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:16:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:16:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: 4.17 scrub starts 2024-03-26T03:16:42.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: 4.17 scrub ok 2024-03-26T03:16:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/319775056' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:16:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: osdmap e397: 8 total, 8 up, 6 in 2024-03-26T03:16:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: 2.1 scrub starts 2024-03-26T03:16:42.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:42 smithi192 ceph-mon[33110]: 2.1 scrub ok 2024-03-26T03:16:42.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:42.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: 4.17 scrub starts 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: 4.17 scrub ok 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/319775056' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: osdmap e397: 8 total, 8 up, 6 in 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: 2.1 scrub starts 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[29666]: 2.1 scrub ok 2024-03-26T03:16:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: 4.17 scrub starts 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: 4.17 scrub ok 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/319775056' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: osdmap e397: 8 total, 8 up, 6 in 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: 2.1 scrub starts 2024-03-26T03:16:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:42 smithi066 ceph-mon[37920]: 2.1 scrub ok 2024-03-26T03:16:42.701 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.731 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:42.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:42.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:42.864 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:42.869 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:42.870 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:42.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:42.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:42.999 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:43.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:43.134 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:43.236 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:43.265 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:43.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:16:43.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:16:43.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-mon[37920]: pgmap v832: 162 pgs: 1 peering, 161 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T03:16:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:16:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:16:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-mon[29666]: pgmap v832: 162 pgs: 1 peering, 161 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T03:16:43.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:43.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:43 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:16:43.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:43 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:16:43.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:43 smithi192 ceph-mon[33110]: pgmap v832: 162 pgs: 1 peering, 161 active+clean; 582 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T03:16:43.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:43.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:43.627 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.656 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:43.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:43.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.787 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:43.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:43.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:43.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:43.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:44.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:44.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:16:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:16:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:16:44.147 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.176 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:44.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:44.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:44.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:44.406 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:44.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:44 smithi192 ceph-mon[33110]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:16:44.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:44 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:16:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:44 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:16:44.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:44 smithi192 ceph-mon[33110]: osdmap e398: 8 total, 8 up, 6 in 2024-03-26T03:16:44.499 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[29666]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:16:44.499 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:16:44.499 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:16:44.499 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[29666]: osdmap e398: 8 total, 8 up, 6 in 2024-03-26T03:16:44.499 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[37920]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:16:44.499 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:16:44.500 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:16:44.500 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:44 smithi066 ceph-mon[37920]: osdmap e398: 8 total, 8 up, 6 in 2024-03-26T03:16:44.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:44.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:44.668 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:44.696 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:44.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:44.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:44.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:44.936 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:16:44.936 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:16:44.936 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:16:44.936 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:16:44.936 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:280126 9.5 9:a020b14b:::benchmark_data_smithi066_125102_object133825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:02.244060+0000", 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 282.68692372800001, 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.060487632999999999, 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 280126 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.937 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244060+0000", 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244060+0000", 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244062+0000", 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5910000000000001e-06 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244067+0000", 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6010000000000002e-06 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244084+0000", 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6582000000000001e-05 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244093+0000", 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2799999999999992e-06 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244154+0000", 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1308000000000003e-05 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.940 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244285+0000", 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000130147 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244402+0000", 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117142 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.251689+0000", 2024-03-26T03:16:44.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0072876080000000001 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.290229+0000", 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.038539963000000003 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304414+0000", 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014184275999999999 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304479+0000", 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5512000000000006e-05 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304548+0000", 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8622999999999997e-05 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:280129 9.5 9:adb46974:::benchmark_data_smithi066_125102_object133828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:16:44.943 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:02.249127+0000", 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 282.681856987, 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055577005999999998, 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 280129 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.944 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249127+0000", 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249127+0000", 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249129+0000", 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0999999999999998e-06 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249135+0000", 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3130000000000004e-06 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249147+0000", 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1406999999999999e-05 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249155+0000", 2024-03-26T03:16:44.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2579999999999995e-06 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249197+0000", 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2280000000000002e-05 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249297+0000", 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9832000000000002e-05 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249388+0000", 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1291000000000004e-05 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.251902+0000", 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002514124 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.296709+0000", 2024-03-26T03:16:44.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.044806954000000003 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304609+0000", 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0079000600000000004 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304662+0000", 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2311000000000001e-05 2024-03-26T03:16:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304704+0000", 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2076000000000002e-05 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281395 9.5 9:a71a042d:::benchmark_data_smithi066_125102_object135094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.201446+0000", 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 281.72953755899999, 2024-03-26T03:16:44.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.110794741, 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281395 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.951 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201446+0000", 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201446+0000", 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201448+0000", 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.415e-06 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201452+0000", 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6550000000000003e-06 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201461+0000", 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6589999999999996e-06 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201469+0000", 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8650000000000001e-06 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201508+0000", 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9436e-05 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201596+0000", 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7429999999999995e-05 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201675+0000", 2024-03-26T03:16:44.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9371999999999998e-05 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.204385+0000", 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0027099519999999998 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.209026+0000", 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046409779999999996 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.312123+0000", 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10309655199999999 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.312198+0000", 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5066000000000003e-05 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.312241+0000", 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3361e-05 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.956 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281402 9.5 9:a3d91319:::benchmark_data_smithi066_125102_object135101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.205723+0000", 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 281.725261171, 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11434588699999999, 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.957 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281402 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205723+0000", 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205723+0000", 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205724+0000", 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.124e-06 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205728+0000", 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2100000000000003e-06 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205737+0000", 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4949999999999997e-06 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205746+0000", 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1980000000000001e-06 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205801+0000", 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.486e-05 2024-03-26T03:16:44.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205984+0000", 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018371800000000001 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.206096+0000", 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011207800000000001 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.209983+0000", 2024-03-26T03:16:44.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038868790000000002 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214312+0000", 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043288220000000004 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.319873+0000", 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10556109900000001 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320005+0000", 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013166400000000001 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320069+0000", 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3739999999999996e-05 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281415 9.8 9:1daf5fd6:::benchmark_data_smithi066_125102_object135114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:16:44.963 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.211863+0000", 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 281.71912118, 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055859276999999999, 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281415 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211863+0000", 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211863+0000", 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211864+0000", 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.751e-06 2024-03-26T03:16:44.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211870+0000", 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4389999999999999e-06 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211885+0000", 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5216e-05 2024-03-26T03:16:44.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211898+0000", 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3339000000000001e-05 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211950+0000", 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1882000000000002e-05 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.212067+0000", 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011676300000000001 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5", 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.212531+0000", 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00046378700000000001 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.215891+0000", 2024-03-26T03:16:44.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0033606780000000002 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267589+0000", 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.051697369 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267623+0000", 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3893999999999998e-05 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267722+0000", 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9159000000000004e-05 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281419 9.b 9:d26b497b:::benchmark_data_smithi066_125102_object135118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.214285+0000", 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 281.716699199, 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.056169565999999997, 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.970 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.973 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281419 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214285+0000", 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214285+0000", 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214286+0000", 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.756e-06 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214292+0000", 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4489999999999998e-06 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214302+0000", 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7710000000000001e-06 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214310+0000", 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9389999999999997e-06 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214365+0000", 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5590999999999997e-05 2024-03-26T03:16:44.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214502+0000", 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000136797 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214645+0000", 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014285799999999999 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.217148+0000", 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0025030439999999998 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.217314+0000", 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016595299999999999 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270382+0000", 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.053067847000000001 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270417+0000", 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4962999999999997e-05 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270454+0000", 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7598000000000002e-05 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.979 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281425 9.5 9:abc5880a:::benchmark_data_smithi066_125102_object135124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.267780+0000", 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 281.663203694, 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052647581999999998, 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281425 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.980 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267780+0000", 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267780+0000", 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267781+0000", 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1260000000000001e-06 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267786+0000", 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6589999999999998e-06 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267797+0000", 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0679999999999999e-05 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267804+0000", 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6170000000000001e-06 2024-03-26T03:16:44.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267858+0000", 2024-03-26T03:16:44.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4046999999999999e-05 2024-03-26T03:16:44.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267964+0000", 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010581099999999999 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.268038+0000", 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3696999999999996e-05 2024-03-26T03:16:44.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270273+0000", 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022354649999999998 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270644+0000", 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00037056600000000002 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320315+0000", 2024-03-26T03:16:44.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.049671445000000002 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320380+0000", 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5005999999999994e-05 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320428+0000", 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7463000000000003e-05 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281777 9.5 9:a406715e:::benchmark_data_smithi066_125102_object135476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.547662+0000", 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 281.38332187999998, 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.054275181999999998, 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281777 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.989 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547662+0000", 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547662+0000", 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547663+0000", 2024-03-26T03:16:44.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2399999999999997e-07 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547666+0000", 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3500000000000001e-06 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547672+0000", 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2620000000000004e-06 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547677+0000", 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5349999999999998e-06 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.549501+0000", 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001823849 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.549581+0000", 2024-03-26T03:16:44.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0561000000000003e-05 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.549659+0000", 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8082000000000002e-05 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.551034+0000", 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013747799999999999 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.576143+0000", 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.025109267000000001 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.601865+0000", 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.025721895000000002 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.601910+0000", 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4332999999999997e-05 2024-03-26T03:16:44.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.601937+0000", 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7444e-05 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290173 9.9 9:90f0ff23:::benchmark_data_smithi066_125102_object143872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.532237+0000", 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 275.39874668900001, 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.061708225999999998, 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:44.995 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290173 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532237+0000", 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:44.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532237+0000", 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532239+0000", 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3920000000000001e-06 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532245+0000", 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8150000000000002e-06 2024-03-26T03:16:44.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532260+0000", 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4543e-05 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532268+0000", 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4710000000000007e-06 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532326+0000", 2024-03-26T03:16:44.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7299999999999997e-05 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532472+0000", 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014627999999999999 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532590+0000", 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011835199999999999 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:44.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539034+0000", 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0064440890000000001 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539158+0000", 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012363599999999999 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.593818+0000", 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.054660042999999998 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.593896+0000", 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7730000000000003e-05 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.593945+0000", 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9574999999999999e-05 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290174 9.9 9:971ef230:::benchmark_data_smithi066_125102_object143873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.532503+0000", 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 275.39848070400001, 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.062196251000000001, 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290174 2024-03-26T03:16:45.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532503+0000", 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532503+0000", 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532504+0000", 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3489999999999999e-06 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532509+0000", 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8459999999999999e-06 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532519+0000", 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7529999999999996e-06 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532527+0000", 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2220000000000002e-06 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535557+0000", 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030293690000000001 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535657+0000", 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100417 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:45.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535765+0000", 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010826 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539145+0000", 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003379309 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539287+0000", 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014264 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.594557+0000", 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055269791999999998 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.594652+0000", 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5237999999999999e-05 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.594699+0000", 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7055999999999998e-05 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.007 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290180 9.5 9:a6fd05c4:::benchmark_data_smithi066_125102_object143879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.535448+0000", 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 275.39553605200001, 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.056686799000000003, 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290180 2024-03-26T03:16:45.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535448+0000", 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535448+0000", 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535449+0000", 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0270000000000001e-06 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535453+0000", 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1899999999999997e-06 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535466+0000", 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3179e-05 2024-03-26T03:16:45.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535473+0000", 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3509999999999996e-06 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535530+0000", 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6192e-05 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535647+0000", 2024-03-26T03:16:45.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117118 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535765+0000", 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118508 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.538830+0000", 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030647019999999999 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.538974+0000", 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000144186 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.591996+0000", 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.053021336000000002 2024-03-26T03:16:45.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.592081+0000", 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5105999999999995e-05 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.592135+0000", 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3903999999999998e-05 2024-03-26T03:16:45.014 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290185 9.9 9:94769a65:::benchmark_data_smithi066_125102_object143884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.541601+0000", 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 275.38938309100001, 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.056043854999999997, 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:45.015 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290185 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541601+0000", 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541601+0000", 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541603+0000", 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.858e-06 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541608+0000", 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.04e-06 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541619+0000", 2024-03-26T03:16:45.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1257000000000001e-05 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541625+0000", 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3319999999999997e-06 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541676+0000", 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0759e-05 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.018 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541802+0000", 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012582000000000001 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541943+0000", 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014159599999999999 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.544680+0000", 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0027369769999999998 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.019 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.544784+0000", 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000103658 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.597572+0000", 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052788195000000003 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.597606+0000", 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3733999999999999e-05 2024-03-26T03:16:45.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.597645+0000", 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8628999999999999e-05 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:177 10.6 10:6dd0b257:::benchmark_data_smithi066_148944_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:53.173704+0000", 2024-03-26T03:16:45.021 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 231.75727956099999, 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.054475559999999999, 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 177 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.022 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173704+0000", 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173704+0000", 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173706+0000", 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5149999999999999e-06 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173726+0000", 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0684999999999999e-05 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173737+0000", 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0604999999999999e-05 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173745+0000", 2024-03-26T03:16:45.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9100000000000005e-06 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173789+0000", 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3627000000000001e-05 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173929+0000", 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014060900000000001 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5", 2024-03-26T03:16:45.025 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.174328+0000", 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00039881500000000001 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.174823+0000", 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00049515300000000002 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.175595+0000", 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00077191399999999996 2024-03-26T03:16:45.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.228066+0000", 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052470623000000001 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.228136+0000", 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0370999999999994e-05 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.228180+0000", 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3733000000000003e-05 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:62024 10.8 10:13aea929:::benchmark_data_smithi066_148944_object62023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:20.143956+0000", 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 204.787027628, 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055962244000000001, 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.028 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62024 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143956+0000", 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.029 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143956+0000", 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143958+0000", 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3039999999999999e-06 2024-03-26T03:16:45.030 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143997+0000", 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8958999999999997e-05 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144024+0000", 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6916000000000001e-05 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144034+0000", 2024-03-26T03:16:45.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2609999999999999e-06 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144090+0000", 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6790999999999999e-05 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144225+0000", 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000134707 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.191365+0000", 2024-03-26T03:16:45.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.047139970000000003 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.196677+0000", 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053120759999999998 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199505+0000", 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028275449999999999 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199813+0000", 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00030792800000000001 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199860+0000", 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7327000000000003e-05 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199918+0000", 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8459999999999999e-05 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.034 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:62027 10.8 10:1d6ddba7:::benchmark_data_smithi066_148944_object62026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:20.146816+0000", 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 204.784168115, 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.053254153999999998, 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62027 2024-03-26T03:16:45.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146816+0000", 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146816+0000", 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.036 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146818+0000", 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1610000000000001e-06 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146845+0000", 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7620999999999999e-05 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.191260+0000", 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.044414431999999997 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.191275+0000", 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4673e-05 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.194261+0000", 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0029868780000000001 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.194419+0000", 2024-03-26T03:16:45.038 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000157625 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.194607+0000", 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000188395 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199510+0000", 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004902689 2024-03-26T03:16:45.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199699+0000", 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018911399999999999 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199966+0000", 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00026690399999999999 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.200031+0000", 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4632000000000001e-05 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.200070+0000", 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9029999999999997e-05 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:88937 10.c e319/314 10:344b2b03:::benchmark_data_smithi066_148944_object88936:head v 319'5454, mlcod=319'5452)", 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.555823+0000", 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 182.37516089299999, 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.042912618, 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555823+0000", 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555823+0000", 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555824+0000", 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.468e-06 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555864+0000", 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9490000000000003e-05 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591050+0000", 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.035186533999999998 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591061+0000", 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1058e-05 2024-03-26T03:16:45.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591114+0000", 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2670000000000002e-05 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591136+0000", 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2340999999999998e-05 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.597711+0000", 2024-03-26T03:16:45.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.006574781 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.598736+0000", 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001024276 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:88939 10.4 e319/314 10:25b58809:::benchmark_data_smithi066_148944_object88938:head v 319'5508, mlcod=319'5506)", 2024-03-26T03:16:45.046 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.557928+0000", 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 182.37305538800001, 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.042445721999999998, 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557928+0000", 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557928+0000", 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557930+0000", 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.474e-06 2024-03-26T03:16:45.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557989+0000", 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9305000000000002e-05 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591131+0000", 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033141362000000001 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591141+0000", 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0163000000000001e-05 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591412+0000", 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00027127500000000002 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591437+0000", 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4924000000000001e-05 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.600112+0000", 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0086753009999999998 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.600374+0000", 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00026191799999999998 2024-03-26T03:16:45.051 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:188973 10.4 e363/331 10:28715c38:::benchmark_data_smithi066_148944_object64326:head v 363'11663, mlcod=362'11659)", 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:32.576404+0000", 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 72.354579927000003, 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.040717334000000001, 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.052 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576404+0000", 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576404+0000", 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576405+0000", 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.16e-07 2024-03-26T03:16:45.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576411+0000", 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2389999999999999e-06 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576418+0000", 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5590000000000002e-06 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576424+0000", 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4570000000000002e-06 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.580727+0000", 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043029330000000001 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.580790+0000", 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2564000000000006e-05 2024-03-26T03:16:45.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.616987+0000", 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.036197571999999997 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.617121+0000", 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013409399999999999 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:196368 10.4 e369/331 10:2df790bd:::benchmark_data_smithi066_148944_object71721:head v 369'12141, mlcod=369'12138)", 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.767420+0000", 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 63.163563594999999, 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.040578483999999998, 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767420+0000", 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767420+0000", 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767422+0000", 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8050000000000001e-06 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767430+0000", 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6699999999999994e-06 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767439+0000", 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6940000000000007e-06 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767447+0000", 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3969999999999997e-06 2024-03-26T03:16:45.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.767994+0000", 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00054754000000000001 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768017+0000", 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2603e-05 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.807950+0000", 2024-03-26T03:16:45.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.039933100999999999 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.807999+0000", 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8674000000000002e-05 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:196370 10.5 e369/341 10:ad493ff0:::benchmark_data_smithi066_148944_object71723:head v 369'12303, mlcod=369'12300)", 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:41.768477+0000", 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 63.162506804000003, 2024-03-26T03:16:45.061 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.040462897999999997, 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768477+0000", 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:16:45.062 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768477+0000", 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768478+0000", 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.42e-06 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768486+0000", 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1910000000000003e-06 2024-03-26T03:16:45.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768526+0000", 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0738000000000002e-05 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.768545+0000", 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8530999999999999e-05 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:16:45.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.770229+0000", 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001684175 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.770256+0000", 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6809999999999999e-05 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.777845+0000", 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0075888379999999997 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:16:45.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:41.808940+0000", 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.031095194999999999 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:16:45.066 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:16:45.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:45.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:45.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:45.239 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:16:45.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:45.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:45.428 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.457 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:45.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: 1.17 scrub starts 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: 1.17 scrub ok 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: 5.4 scrub starts 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: 5.4 scrub ok 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: pgmap v834: 162 pgs: 1 peering, 161 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 789 B/s rd, 0 op/s 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: 1.13 scrub starts 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: 1.13 scrub ok 2024-03-26T03:16:45.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2635803414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:16:45.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: 1.17 scrub starts 2024-03-26T03:16:45.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: 1.17 scrub ok 2024-03-26T03:16:45.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: 5.4 scrub starts 2024-03-26T03:16:45.497 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: 5.4 scrub ok 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: pgmap v834: 162 pgs: 1 peering, 161 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 789 B/s rd, 0 op/s 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: 1.13 scrub starts 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: 1.13 scrub ok 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2635803414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: 1.17 scrub starts 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: 1.17 scrub ok 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: 5.4 scrub starts 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: 5.4 scrub ok 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: pgmap v834: 162 pgs: 1 peering, 161 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 789 B/s rd, 0 op/s 2024-03-26T03:16:45.498 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: 1.13 scrub starts 2024-03-26T03:16:45.499 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: 1.13 scrub ok 2024-03-26T03:16:45.499 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2635803414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:16:45.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.587 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:45.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:45.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:45.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:45.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:45.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:45.947 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:45.974 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:45.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:46.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:46.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:46.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:46.233 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:46.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:46.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 5.13 scrub starts 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 5.13 scrub ok 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 1.14 scrub starts 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 1.14 scrub ok 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2635803414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: osdmap e399: 8 total, 8 up, 6 in 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:16:46.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 5.13 scrub starts 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 5.13 scrub ok 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 1.14 scrub starts 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 1.14 scrub ok 2024-03-26T03:16:46.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2635803414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:16:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: osdmap e399: 8 total, 8 up, 6 in 2024-03-26T03:16:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:16:46.361 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:46 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:16:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:46.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:46.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 5.13 scrub starts 2024-03-26T03:16:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 5.13 scrub ok 2024-03-26T03:16:46.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 1.14 scrub starts 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 1.14 scrub ok 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2635803414' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: osdmap e399: 8 total, 8 up, 6 in 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:16:46.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:46 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:16:46.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:46.591 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.620 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:46.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:46.721 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.750 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:46.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:46.850 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:46.879 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:46.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:46.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:47.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:47.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:47.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:47.240 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:47.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:47.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:47.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:47.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:47.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:47.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:16:47.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:16:47.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[29666]: pgmap v836: 162 pgs: 162 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T03:16:47.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:16:47.536 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:16:47.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:16:47.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:16:47.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[37920]: pgmap v836: 162 pgs: 162 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T03:16:47.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:16:47.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:47 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:16:47.629 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:47.657 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:47 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:16:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:47 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:16:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:47 smithi192 ceph-mon[33110]: pgmap v836: 162 pgs: 162 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T03:16:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:47 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:16:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:47 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:16:47.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:47.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:47.888 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:47.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:47.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:48.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:48.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:48.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:48.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:48.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:48.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:48 smithi066 ceph-mon[29666]: 1.18 deep-scrub starts 2024-03-26T03:16:48.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:48 smithi066 ceph-mon[29666]: 1.18 deep-scrub ok 2024-03-26T03:16:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:48 smithi066 ceph-mon[37920]: 1.18 deep-scrub starts 2024-03-26T03:16:48.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:48 smithi066 ceph-mon[37920]: 1.18 deep-scrub ok 2024-03-26T03:16:48.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:48.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:48.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:48.709 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:16:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:48 smithi192 ceph-mon[33110]: 1.18 deep-scrub starts 2024-03-26T03:16:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:48 smithi192 ceph-mon[33110]: 1.18 deep-scrub ok 2024-03-26T03:16:48.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:48.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:48.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:48.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:48.979 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:48.993 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:49.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:49.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:49.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:49.203 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.2 primary-affinity to 0.0051669 (83382) 2024-03-26T03:16:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:49.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:49.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.343 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:49.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:49.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[29666]: pgmap v837: 162 pgs: 162 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 713 B/s rd, 0 op/s 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/721054793' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0051668981141794745}]: dispatch 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/721054793' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0051668981141794745}]': finished 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[29666]: osdmap e400: 8 total, 8 up, 6 in 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[37920]: pgmap v837: 162 pgs: 162 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 713 B/s rd, 0 op/s 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/721054793' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0051668981141794745}]: dispatch 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/721054793' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0051668981141794745}]': finished 2024-03-26T03:16:49.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:49 smithi066 ceph-mon[37920]: osdmap e400: 8 total, 8 up, 6 in 2024-03-26T03:16:49.444 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.474 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:49.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:49.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.604 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:49.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:49.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:49.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:16:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:16:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-mon[33110]: pgmap v837: 162 pgs: 162 active+clean; 582 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 713 B/s rd, 0 op/s 2024-03-26T03:16:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/721054793' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0051668981141794745}]: dispatch 2024-03-26T03:16:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/721054793' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0051668981141794745}]': finished 2024-03-26T03:16:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-mon[33110]: osdmap e400: 8 total, 8 up, 6 in 2024-03-26T03:16:49.836 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:49.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.866 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:49.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:49.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:49.994 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:49.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:50.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:50.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:50.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:16:50.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:50.246 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:16:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:16:50.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:50.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:50.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:50.486 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:50.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:50.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: 1.1b scrub starts 2024-03-26T03:16:50.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: 1.1b scrub ok 2024-03-26T03:16:50.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: 4.1d deep-scrub starts 2024-03-26T03:16:50.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: 4.1d deep-scrub ok 2024-03-26T03:16:50.565 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:16:50.565 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:16:50.565 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[29666]: osdmap e401: 8 total, 8 up, 6 in 2024-03-26T03:16:50.565 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: 1.1b scrub starts 2024-03-26T03:16:50.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: 1.1b scrub ok 2024-03-26T03:16:50.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: 4.1d deep-scrub starts 2024-03-26T03:16:50.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: 4.1d deep-scrub ok 2024-03-26T03:16:50.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:16:50.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:16:50.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:50 smithi066 ceph-mon[37920]: osdmap e401: 8 total, 8 up, 6 in 2024-03-26T03:16:50.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:50.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: 1.1b scrub starts 2024-03-26T03:16:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: 1.1b scrub ok 2024-03-26T03:16:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: 4.1d deep-scrub starts 2024-03-26T03:16:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: 4.1d deep-scrub ok 2024-03-26T03:16:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:16:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:16:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:50 smithi192 ceph-mon[33110]: osdmap e401: 8 total, 8 up, 6 in 2024-03-26T03:16:50.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:50.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:50.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:50.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:50.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:51.006 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:51.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:51.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.165 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:51.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:51.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:51.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:51.396 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.425 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:51.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:51.526 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.554 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:51.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: 4.14 deep-scrub starts 2024-03-26T03:16:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: 4.14 deep-scrub ok 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: pgmap v840: 162 pgs: 7 activating, 9 peering, 146 active+clean; 635 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 4.8 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: 4.14 deep-scrub starts 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: 4.14 deep-scrub ok 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: pgmap v840: 162 pgs: 7 activating, 9 peering, 146 active+clean; 635 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 4.8 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:16:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:16:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:16:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:16:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:51 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:16:51.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:51.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: 4.14 deep-scrub starts 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: 4.14 deep-scrub ok 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: pgmap v840: 162 pgs: 7 activating, 9 peering, 146 active+clean; 635 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 4.8 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:16:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:51 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:16:51.786 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:51.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:51.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:51.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:51.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:52.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:52.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:52.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:52.195 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:52.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.325 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:52.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:52.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: 5.1a deep-scrub starts 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: 5.1a deep-scrub ok 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 12 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:16:52.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: 5.1a deep-scrub starts 2024-03-26T03:16:52.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: 5.1a deep-scrub ok 2024-03-26T03:16:52.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 12 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T03:16:52.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:16:52.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:16:52.360 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:52 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:16:52.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.456 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:52.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:52.556 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.584 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:52.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:52.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:52.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:16:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:16:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: 5.1a deep-scrub starts 2024-03-26T03:16:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: 5.1a deep-scrub ok 2024-03-26T03:16:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 12 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T03:16:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:16:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:16:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:52 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:16:52.813 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:52.841 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:52.943 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:52.971 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:52.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:53.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:53.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:53.201 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.230 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:53.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:53.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.359 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:53.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:53.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:53.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: 5.b deep-scrub starts 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: 5.b deep-scrub ok 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: pgmap v841: 162 pgs: 7 activating, 9 peering, 146 active+clean; 635 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 3.4 KiB/s rd, 31 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:16:53.568 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: 5.b deep-scrub starts 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: 5.b deep-scrub ok 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: pgmap v841: 162 pgs: 7 activating, 9 peering, 146 active+clean; 635 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 3.4 KiB/s rd, 31 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:16:53.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:16:53.570 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:16:53.570 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:53.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:53.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:53.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:16:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:16:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: 5.b deep-scrub starts 2024-03-26T03:16:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: 5.b deep-scrub ok 2024-03-26T03:16:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: pgmap v841: 162 pgs: 7 activating, 9 peering, 146 active+clean; 635 KiB data, 4.7 GiB used, 532 GiB / 536 GiB avail; 3.4 KiB/s rd, 31 KiB/s wr, 4 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-26T03:16:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:16:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:16:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:16:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:16:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:53.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:53.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:53.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:53.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:53.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:54.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:54.016 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:16:54.016 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:16:54.016 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 0 2024-03-26T03:16:54.016 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:16:54.096 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:16:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:16:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:16:54.111 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.142 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:54.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:54.243 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.273 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:54.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:54.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[29666]: 5.6 deep-scrub starts 2024-03-26T03:16:54.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[29666]: 5.6 deep-scrub ok 2024-03-26T03:16:54.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:16:54.352 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:16:54.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[29666]: osdmap e402: 8 total, 8 up, 6 in 2024-03-26T03:16:54.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[37920]: 5.6 deep-scrub starts 2024-03-26T03:16:54.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[37920]: 5.6 deep-scrub ok 2024-03-26T03:16:54.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:16:54.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:16:54.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:54 smithi066 ceph-mon[37920]: osdmap e402: 8 total, 8 up, 6 in 2024-03-26T03:16:54.375 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.403 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:54.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:54.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:54.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:54.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.664 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:54.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:54 smithi192 ceph-mon[33110]: 5.6 deep-scrub starts 2024-03-26T03:16:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:54 smithi192 ceph-mon[33110]: 5.6 deep-scrub ok 2024-03-26T03:16:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:54 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:16:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:54 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:16:54.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:54 smithi192 ceph-mon[33110]: osdmap e402: 8 total, 8 up, 6 in 2024-03-26T03:16:54.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:54.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:54.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:54.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:54.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:54.956 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:16:54.977 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:16:54.996 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:16:55.016 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:16:55.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:55.036 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:16:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:55.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:55.056 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:16:55.077 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:16:55.100 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:16:55.101 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:16:55.157 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.190 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:55.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:55.291 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.319 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:55.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:55.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:55.423 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:55.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:55.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: pgmap v843: 162 pgs: 7 activating, 4 peering, 151 active+clean; 635 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 4.0 KiB/s rd, 36 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3104573051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3341835190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T03:16:55.548 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[29666]: from='client.35810 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: pgmap v843: 162 pgs: 7 activating, 4 peering, 151 active+clean; 635 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 4.0 KiB/s rd, 36 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3104573051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3341835190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T03:16:55.549 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:55 smithi066 ceph-mon[37920]: from='client.35810 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T03:16:55.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:55.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:55.682 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.712 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:55.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: pgmap v843: 162 pgs: 7 activating, 4 peering, 151 active+clean; 635 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 4.0 KiB/s rd, 36 KiB/s wr, 5 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3104573051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3341835190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T03:16:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:55 smithi192 ceph-mon[33110]: from='client.35810 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T03:16:55.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:55.840 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:16:55.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T03:16:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:55.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:55.900 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_4' created 2024-03-26T03:16:55.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:55.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:55.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:56.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:16:56.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:16:56.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:56.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:56.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:56.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:56.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:56.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:56.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:16:56.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[29666]: from='client.35810 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[29666]: osdmap e403: 8 total, 8 up, 6 in 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[37920]: from='client.35810 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T03:16:56.524 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[37920]: osdmap e403: 8 total, 8 up, 6 in 2024-03-26T03:16:56.525 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:16:56.525 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:56 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:16:56.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:56.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:56.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:56 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:16:56.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:56 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:16:56.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:56 smithi192 ceph-mon[33110]: from='client.35810 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T03:16:56.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:56 smithi192 ceph-mon[33110]: osdmap e403: 8 total, 8 up, 6 in 2024-03-26T03:16:56.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:56 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:16:56.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:56 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:16:56.742 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:56.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:56.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:56.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:56.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:56.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:57.048 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-26T03:16:57.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:57.131 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:57.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:57.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:57.361 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:57.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:57.492 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.520 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:57.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:57.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:16:57.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:16:57.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[29666]: pgmap v845: 178 pgs: 16 unknown, 162 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:57.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:16:57.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:16:57.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[29666]: osdmap e404: 8 total, 8 up, 6 in 2024-03-26T03:16:57.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:16:57.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:16:57.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[37920]: pgmap v845: 178 pgs: 16 unknown, 162 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:57.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:16:57.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:16:57.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:57 smithi066 ceph-mon[37920]: osdmap e404: 8 total, 8 up, 6 in 2024-03-26T03:16:57.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.651 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:57.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:57 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:16:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:57 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:16:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:57 smithi192 ceph-mon[33110]: pgmap v845: 178 pgs: 16 unknown, 162 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:57 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:16:57.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:57 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:16:57.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:57 smithi192 ceph-mon[33110]: osdmap e404: 8 total, 8 up, 6 in 2024-03-26T03:16:57.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:57.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:57.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:57.913 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:57.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:58.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:58.044 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:58.145 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:16:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.174 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:58.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:16:58.275 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:16:58.304 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:16:58.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:58.414 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:58.419 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:58.419 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:16:58.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:58.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:16:58.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:16:58.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: 4.11 deep-scrub starts 2024-03-26T03:16:58.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: 4.11 deep-scrub ok 2024-03-26T03:16:58.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 4 pgs peering) 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[29666]: osdmap e405: 8 total, 8 up, 6 in 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: 4.11 deep-scrub starts 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: 4.11 deep-scrub ok 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 4 pgs peering) 2024-03-26T03:16:58.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:58 smithi066 ceph-mon[37920]: osdmap e405: 8 total, 8 up, 6 in 2024-03-26T03:16:58.536 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:58.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:58.665 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.701 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:58.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:16:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:16:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: 4.11 deep-scrub starts 2024-03-26T03:16:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: 4.11 deep-scrub ok 2024-03-26T03:16:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:16:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 4 pgs peering) 2024-03-26T03:16:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:58 smithi192 ceph-mon[33110]: osdmap e405: 8 total, 8 up, 6 in 2024-03-26T03:16:58.802 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:58.830 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:58.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:58.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:58.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:59.062 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:59.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:59.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:16:59.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:16:59.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.351 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:59.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:59.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:16:59.473 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[37920]: pgmap v848: 178 pgs: 16 unknown, 162 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[37920]: osdmap e406: 8 total, 8 up, 6 in 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2938030025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[29666]: pgmap v848: 178 pgs: 16 unknown, 162 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[29666]: osdmap e406: 8 total, 8 up, 6 in 2024-03-26T03:16:59.474 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:16:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2938030025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:16:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:59.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:16:59.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:16:59.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:16:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:16:59.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:16:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:59 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:16:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:59 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:16:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:59 smithi192 ceph-mon[33110]: pgmap v848: 178 pgs: 16 unknown, 162 active+clean; 582 KiB data, 4.6 GiB used, 532 GiB / 536 GiB avail 2024-03-26T03:16:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:59 smithi192 ceph-mon[33110]: osdmap e406: 8 total, 8 up, 6 in 2024-03-26T03:16:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:16:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2938030025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:16:59.741 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:16:59.843 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:16:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:16:59.872 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:16:59.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:16:59.973 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.002 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:00.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:00.102 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.131 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:00.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:00.202 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:17:00.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:00.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:00.283 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:17:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:17:00.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:00.364 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:17:00.390 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:00.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:00.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:00.534 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:17:00.534 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:17:00.534 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2938030025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[37920]: osdmap e407: 8 total, 8 up, 6 in 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2938030025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:17:00.535 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:00 smithi066 ceph-mon[29666]: osdmap e407: 8 total, 8 up, 6 in 2024-03-26T03:17:00.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.651 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:00.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:00.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:00 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:17:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:00 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:17:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:00 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2938030025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:17:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:00 smithi192 ceph-mon[33110]: osdmap e407: 8 total, 8 up, 6 in 2024-03-26T03:17:00.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:00.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:00.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:00.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:00.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:01.011 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.041 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:01.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:01.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:01.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:01.273 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:01.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:01.406 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:01.435 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:01.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:01.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[29666]: 4.15 scrub starts 2024-03-26T03:17:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[29666]: 4.15 scrub ok 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[29666]: pgmap v851: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[37920]: 4.15 scrub starts 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[37920]: 4.15 scrub ok 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:17:01.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:01 smithi066 ceph-mon[37920]: pgmap v851: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:17:01.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:01.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:01 smithi192 ceph-mon[33110]: 4.15 scrub starts 2024-03-26T03:17:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:01 smithi192 ceph-mon[33110]: 4.15 scrub ok 2024-03-26T03:17:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:01.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:01 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:17:01.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:01 smithi192 ceph-mon[33110]: pgmap v851: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:17:01.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:01.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:01.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:01.952 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:01.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:02.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:02.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:02.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:02.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:02.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:02.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:02.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:02.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:02.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:02.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:02.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.733 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:02.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:02.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:02.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:02.966 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:02.993 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:02.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:03.095 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.123 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:03.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:03.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.252 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:03.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:03.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:03.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:03.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:03.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:03.614 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.643 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:03.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:03.695 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:03 smithi066 ceph-mon[29666]: pgmap v852: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 929 B/s rd, 0 op/s 2024-03-26T03:17:03.695 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:03 smithi066 ceph-mon[37920]: pgmap v852: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 929 B/s rd, 0 op/s 2024-03-26T03:17:03.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:03 smithi192 ceph-mon[33110]: pgmap v852: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 929 B/s rd, 0 op/s 2024-03-26T03:17:03.744 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.772 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:03.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:03.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:03.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:03.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:03.946 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:17:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:17:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:17:04.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.032 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:04.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:04.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.161 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:04.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:04.262 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.291 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:04.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:04.326 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:17:04.326 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:17:04.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:17:04.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:04.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:04.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:04.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: osdmap e408: 8 total, 8 up, 6 in 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1475911975' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[29666]: from='client.35828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:04.555 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:04.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: osdmap e408: 8 total, 8 up, 6 in 2024-03-26T03:17:04.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:04.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:04.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1475911975' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:17:04.556 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:04 smithi066 ceph-mon[37920]: from='client.35828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:17:04.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:04.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: osdmap e408: 8 total, 8 up, 6 in 2024-03-26T03:17:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1475911975' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:17:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:04 smithi192 ceph-mon[33110]: from='client.35828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:17:04.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:04.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:04.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:04.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:04.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:05.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:05.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:05.160 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.189 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:05.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:05.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:05.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:05.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:17:05.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:05.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:05.507 INFO:teuthology.orchestra.run.smithi066.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-26T03:17:05.544 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.572 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:05.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:05.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[37920]: pgmap v854: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:17:05.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[37920]: from='client.35828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[37920]: osdmap e409: 8 total, 8 up, 6 in 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[29666]: pgmap v854: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[29666]: from='client.35828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[29666]: osdmap e409: 8 total, 8 up, 6 in 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:05.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:05 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:05.673 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.701 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:05.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:05.732 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:05 smithi192 ceph-mon[33110]: pgmap v854: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:17:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:05 smithi192 ceph-mon[33110]: from='client.35828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:17:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:05 smithi192 ceph-mon[33110]: osdmap e409: 8 total, 8 up, 6 in 2024-03-26T03:17:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:05 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:05 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:05.803 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:05.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:05.933 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:05.963 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:05.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:06.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:06.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:06.195 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:06.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:06.326 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:06.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:06.458 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:06.504 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:06.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:06.649 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:06 smithi066 ceph-mon[29666]: osdmap e410: 8 total, 8 up, 6 in 2024-03-26T03:17:06.650 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:06 smithi066 ceph-mon[37920]: osdmap e410: 8 total, 8 up, 6 in 2024-03-26T03:17:06.679 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:06 smithi192 ceph-mon[33110]: osdmap e410: 8 total, 8 up, 6 in 2024-03-26T03:17:06.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:06.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:06.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:06.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:06.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:07.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:07.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:07.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:07.201 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:07.250 DEBUG:teuthology.orchestra.run.smithi066:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-26T03:17:07.251 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:07.253 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:17:07.253 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:17:07.294 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:07.303 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.332 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:07.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:07.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:07.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:07.489 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T03:17:07.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:hints = 1 2024-03-26T03:17:07.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:17:07.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object prefix: benchmark_data_smithi066_175179 2024-03-26T03:17:07.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:17:07.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:17:07.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:07.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:07.627 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:07 smithi066 ceph-mon[29666]: pgmap v857: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:17:07.627 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:07 smithi066 ceph-mon[29666]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:17:07.627 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3326463191' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:17:07.627 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:07 smithi066 ceph-mon[37920]: pgmap v857: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:17:07.628 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:07 smithi066 ceph-mon[37920]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:17:07.628 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3326463191' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:17:07.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:07 smithi192 ceph-mon[33110]: pgmap v857: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:17:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:07 smithi192 ceph-mon[33110]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:17:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3326463191' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:17:07.750 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:07.852 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:07.900 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:07.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:08.001 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:08.050 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:08.152 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:08.203 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:08.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:08.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:08.440 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:08.487 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:08.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 1 16 3290 3274 204.62 204.625 0.00686841 0.00484413 2024-03-26T03:17:08.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:08.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:08.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:08.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:08.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:08 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3326463191' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:17:08.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:08 smithi066 ceph-mon[29666]: osdmap e411: 8 total, 8 up, 6 in 2024-03-26T03:17:08.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:08 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3326463191' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:17:08.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:08 smithi066 ceph-mon[37920]: osdmap e411: 8 total, 8 up, 6 in 2024-03-26T03:17:08.859 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:08.906 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:08.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:08.969 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T03:17:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:08 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3326463191' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:17:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:08 smithi192 ceph-mon[33110]: osdmap e411: 8 total, 8 up, 6 in 2024-03-26T03:17:09.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:09.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:09.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:09.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:09.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.329 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:09.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:09.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:09.477 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:09.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 2 16 6354 6338 198.05 191.5 0.00341457 0.00503705 2024-03-26T03:17:09.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:09.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:09.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[29666]: pgmap v859: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[29666]: osdmap e412: 8 total, 8 up, 6 in 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:09.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[37920]: pgmap v859: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:17:09.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:09 smithi066 ceph-mon[37920]: osdmap e412: 8 total, 8 up, 6 in 2024-03-26T03:17:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:09.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:09.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:09.943 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:09.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:09 smithi192 ceph-mon[33110]: pgmap v859: 178 pgs: 178 active+clean; 582 KiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:17:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:09 smithi192 ceph-mon[33110]: osdmap e412: 8 total, 8 up, 6 in 2024-03-26T03:17:10.044 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:10.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:10.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:10.244 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:10.346 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:10.388 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:10.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:17:10 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:17:10.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:10.516 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 3 16 9296 9280 193.32 183.875 0.00629865 0.0051626 2024-03-26T03:17:10.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:10.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:10.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:10.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:10.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:10.895 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:10.943 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:10.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:11.018 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:17:11.018 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:11.019 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:11.019 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[29666]: pgmap v861: 178 pgs: 178 active+clean; 439 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T03:17:11.019 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:17:11.019 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:11.019 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:11.019 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:10 smithi066 ceph-mon[37920]: pgmap v861: 178 pgs: 178 active+clean; 439 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T03:17:11.046 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:11.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:11.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:11.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:17:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:10 smithi192 ceph-mon[33110]: pgmap v861: 178 pgs: 178 active+clean; 439 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T03:17:11.321 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:11.372 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:11.474 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:11.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:11.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 4 15 11891 11876 185.549 162.25 0.00537324 0.00538032 2024-03-26T03:17:11.618 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.673 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:11.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:11.775 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:11.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:11.935 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:11.955 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:17:11.955 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:17:11.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:11.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:12.090 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.138 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:12.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:12.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:12.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:12.325 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:17:12.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.419 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:12.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:12.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 5 15 14247 14232 177.882 147.25 0.00506389 0.0056132 2024-03-26T03:17:12.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:12.570 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3603187488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:12.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:12 smithi066 ceph-mon[29666]: from='client.25789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3603187488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:12 smithi066 ceph-mon[37920]: from='client.25789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:12.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:12.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3603187488' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:12 smithi192 ceph-mon[33110]: from='client.25789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:12.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:12.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:12.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:12.971 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:12.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:13.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:13.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:13.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:13.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:13.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:13.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:13.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[29666]: from='client.25789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:17:13.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[29666]: osdmap e413: 8 total, 8 up, 6 in 2024-03-26T03:17:13.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[29666]: pgmap v863: 178 pgs: 178 active+clean; 439 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T03:17:13.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:13.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[37920]: from='client.25789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:17:13.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[37920]: osdmap e413: 8 total, 8 up, 6 in 2024-03-26T03:17:13.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-mon[37920]: pgmap v863: 178 pgs: 178 active+clean; 439 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T03:17:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:13.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:13.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:13.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 6 16 16740 16724 174.191 155.75 0.00495801 0.0057329 2024-03-26T03:17:13.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:13.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:13.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:13 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:13 smithi192 ceph-mon[33110]: from='client.25789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:17:13.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:13 smithi192 ceph-mon[33110]: osdmap e413: 8 total, 8 up, 6 in 2024-03-26T03:17:13.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:13 smithi192 ceph-mon[33110]: pgmap v863: 178 pgs: 178 active+clean; 439 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T03:17:13.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:13.803 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:13.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:13.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:13.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:14.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:14.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:14.087 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:17:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:17:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:17:14.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:14.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:14.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:14.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:14.442 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:17:14.443 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:14.500 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:14.517 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 7 16 19005 18989 169.528 141.562 0.00576463 0.005891 2024-03-26T03:17:14.602 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:14.651 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:14.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:14.785 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:14.887 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:14.904 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:14.937 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:14.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:15.039 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:15.095 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:15.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:15.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:15.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:15.354 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:15 smithi066 ceph-mon[29666]: osdmap e414: 8 total, 8 up, 6 in 2024-03-26T03:17:15.355 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:15 smithi066 ceph-mon[29666]: pgmap v865: 178 pgs: 178 active+clean; 692 MiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T03:17:15.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:15 smithi066 ceph-mon[37920]: osdmap e414: 8 total, 8 up, 6 in 2024-03-26T03:17:15.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:15 smithi066 ceph-mon[37920]: pgmap v865: 178 pgs: 178 active+clean; 692 MiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T03:17:15.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:15.395 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:17:15.438 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:17:15.464 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:15.480 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:17:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:15 smithi192 ceph-mon[33110]: osdmap e414: 8 total, 8 up, 6 in 2024-03-26T03:17:15.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:15 smithi192 ceph-mon[33110]: pgmap v865: 178 pgs: 178 active+clean; 692 MiB data, 5.9 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T03:17:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.513 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:15.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:15.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 8 16 21125 21109 164.898 132.5 0.00643195 0.00605719 2024-03-26T03:17:15.537 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:17:15.559 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:17:15.584 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:17:15.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:15.630 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:17:15.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:15.671 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:17:15.716 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:17:15.724 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:17:15.724 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:17:15.725 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:17:15.725 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:17:15.725 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:17:15.725 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:17:15.725 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:17:15.725 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:17:15.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:15.800 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:15.903 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:15.951 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:15.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:16.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:16.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:16.187 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:16.235 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:16.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:16.370 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:16.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:16.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2971438683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:16.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:16.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 9 16 23295 23279 161.644 135.625 0.00447589 0.00617982 2024-03-26T03:17:16.541 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2971438683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:16.541 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2971438683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:16.610 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.661 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:16.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:16.762 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:16.808 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:16.910 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:16.960 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:16.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:17.062 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:17.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:17.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:17.213 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:17.258 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:17.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:17.360 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:17.435 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:17.442 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:17.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:17.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:17 smithi192 ceph-mon[33110]: pgmap v866: 178 pgs: 178 active+clean; 1.2 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-26T03:17:17.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 10 16 25407 25391 158.666 132 0.00931317 0.0062961 2024-03-26T03:17:17.520 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:17 smithi066 ceph-mon[37920]: pgmap v866: 178 pgs: 178 active+clean; 1.2 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-26T03:17:17.520 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:17 smithi066 ceph-mon[29666]: pgmap v866: 178 pgs: 178 active+clean; 1.2 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-26T03:17:17.543 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:17.598 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:17.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:17.699 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:17.760 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:17.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:17.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:17.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:17.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:17.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:18.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:18.130 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:18.168 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:18.270 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:18.300 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:18.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:18.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:18.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 11 16 27309 27293 155.048 118.875 0.0109615 0.00644313 2024-03-26T03:17:18.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:18.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:18.665 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.699 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:18.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:18.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:18.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:18.934 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:18.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:18.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:19.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:19.098 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:19.200 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:19.240 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:19.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:19.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:19.400 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:17:19.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.510 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:19.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:19.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 12 16 28875 28859 150.283 97.875 0.0108915 0.0066479 2024-03-26T03:17:19.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:19.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:19 smithi192 ceph-mon[33110]: pgmap v867: 178 pgs: 178 active+clean; 1.2 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-26T03:17:19.746 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:19.783 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:19.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:19 smithi066 ceph-mon[29666]: pgmap v867: 178 pgs: 178 active+clean; 1.2 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-26T03:17:19.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:19 smithi066 ceph-mon[37920]: pgmap v867: 178 pgs: 178 active+clean; 1.2 GiB data, 7.9 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 98 MiB/s wr, 1.56k op/s 2024-03-26T03:17:19.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:19.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:19.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:20.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:20.064 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:20.166 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:20.212 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:20.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:20.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:20.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:20.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:20.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:17:20 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:17:20.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 13 16 30639 30623 147.203 110.25 0.00593326 0.0067865 2024-03-26T03:17:20.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:20.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:20.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:20.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:20.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:20.878 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:20.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:20.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:21.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:21.115 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:21.160 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:21.227 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:21.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:21.324 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:21.426 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:21.464 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:21.474 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:21.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 14 16 32267 32251 143.956 101.75 0.00388107 0.0069402 2024-03-26T03:17:21.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:21.621 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:21.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:21 smithi192 ceph-mon[33110]: pgmap v868: 178 pgs: 178 active+clean; 1.8 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.76k op/s 2024-03-26T03:17:21.753 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:21.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:21.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:21 smithi066 ceph-mon[29666]: pgmap v868: 178 pgs: 178 active+clean; 1.8 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.76k op/s 2024-03-26T03:17:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:21 smithi066 ceph-mon[37920]: pgmap v868: 178 pgs: 178 active+clean; 1.8 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.76k op/s 2024-03-26T03:17:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:21.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:21.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:21.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:22.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:22.113 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.141 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:22.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:22.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:22.275 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:22.377 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.410 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:22.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:22.489 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T03:17:22.511 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:22.518 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 15 16 33788 33772 140.696 95.0625 0.00704826 0.00709991 2024-03-26T03:17:22.560 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:22.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:22.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:22 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3067442725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:17:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:22 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:22 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:22.809 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.847 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:22.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:22.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:22 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3067442725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:17:22.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:22 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:22.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:22 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:22 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3067442725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:17:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:22 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:22.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:22 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:22.948 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:22.996 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:22.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:23.098 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.128 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:23.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:23.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:23.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:23.406 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:23.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:23.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 16 16 35446 35430 138.379 103.625 0.00849327 0.00722023 2024-03-26T03:17:23.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:23.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:23.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:23 smithi192 ceph-mon[33110]: pgmap v869: 178 pgs: 178 active+clean; 1.8 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T03:17:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:23 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3067442725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:17:23.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:23 smithi192 ceph-mon[33110]: osdmap e415: 8 total, 8 up, 6 in 2024-03-26T03:17:23.779 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:23.822 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:23.830 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-mon[37920]: pgmap v869: 178 pgs: 178 active+clean; 1.8 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T03:17:23.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3067442725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:17:23.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-mon[37920]: osdmap e415: 8 total, 8 up, 6 in 2024-03-26T03:17:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-mon[29666]: pgmap v869: 178 pgs: 178 active+clean; 1.8 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-26T03:17:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3067442725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:17:23.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-mon[29666]: osdmap e415: 8 total, 8 up, 6 in 2024-03-26T03:17:23.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:23.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:23.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:24.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:24.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:24.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:17:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:17:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:17:24.191 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:24.233 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:24.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:24.381 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:24.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:24.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 17 16 37070 37054 136.209 101.5 0.0128083 0.00733568 2024-03-26T03:17:24.530 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:24.632 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:24.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:24.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:24.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:24.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:24.926 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:24.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:25.028 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.075 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:25.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:25.176 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:25.220 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:25.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:25.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:25.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:25.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:25 smithi192 ceph-mon[33110]: osdmap e416: 8 total, 8 up, 6 in 2024-03-26T03:17:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:25 smithi192 ceph-mon[33110]: pgmap v872: 178 pgs: 178 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-26T03:17:25.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:25.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 18 16 38325 38309 132.999 78.4375 0.00511421 0.00749337 2024-03-26T03:17:25.587 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:25 smithi066 ceph-mon[29666]: osdmap e416: 8 total, 8 up, 6 in 2024-03-26T03:17:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:25 smithi066 ceph-mon[29666]: pgmap v872: 178 pgs: 178 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-26T03:17:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:25 smithi066 ceph-mon[37920]: osdmap e416: 8 total, 8 up, 6 in 2024-03-26T03:17:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:25 smithi066 ceph-mon[37920]: pgmap v872: 178 pgs: 178 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-26T03:17:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:25.630 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:25.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:25.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:25.857 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:25.902 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:25.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:25.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:26.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:26.032 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:26.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:26.134 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.181 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:26.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:26.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:26.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:26.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:26.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:26.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:26.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 19 15 39882 39867 131.124 97.375 0.0135715 0.0076208 2024-03-26T03:17:26.549 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:26.596 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:26.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:26.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:26.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:26.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:26.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:26.965 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:27.016 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:27.119 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:27.166 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:27.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:27.302 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:27.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:27.404 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:27.449 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:27.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T03:17:27.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:17:27.519543+0000 min lat: 0.00143665 max lat: 0.149616 avg lat: 0.00768268 2024-03-26T03:17:27.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:17:27.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 20 16 41635 41619 130.042 109.5 0.00379411 0.00768268 2024-03-26T03:17:27.551 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.600 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:27.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:27.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:27.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:27 smithi192 ceph-mon[33110]: pgmap v873: 178 pgs: 178 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.23k op/s 2024-03-26T03:17:27.835 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:27.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:27 smithi066 ceph-mon[29666]: pgmap v873: 178 pgs: 178 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.23k op/s 2024-03-26T03:17:27.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:27 smithi066 ceph-mon[37920]: pgmap v873: 178 pgs: 178 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.23k op/s 2024-03-26T03:17:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:27.890 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:27.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:27.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:28.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:28.126 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.172 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:28.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:28.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:28.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:28.406 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:28.453 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:28.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 21 16 43322 43306 128.87 105.438 0.00877637 0.00775352 2024-03-26T03:17:28.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:28.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:28.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:28.819 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:28.870 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:28.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:28.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:29.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:29.105 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.156 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:29.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:29.246 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:29.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:29.303 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:29.405 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.456 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:29.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:29.519 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 22 16 44933 44917 127.589 100.688 0.0108549 0.00783161 2024-03-26T03:17:29.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:29.615 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:29.622 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:29 smithi066 ceph-mon[29666]: pgmap v874: 178 pgs: 178 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:17:29.623 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:29 smithi066 ceph-mon[37920]: pgmap v874: 178 pgs: 178 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:17:29.716 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:29 smithi192 ceph-mon[33110]: pgmap v874: 178 pgs: 178 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:17:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:29.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:29.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:29.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:29.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15978.0:45650 11.4 e416/403 11:2eebc7f7:::benchmark_data_smithi066_175179_object45649:head v 416'2868, mlcod=416'2864)", 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:29.986232+0000", 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.001482496, 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0015212509999999999, 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:29.989 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986232+0000", 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986232+0000", 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986234+0000", 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.426e-06 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986262+0000", 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.813e-05 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986275+0000", 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2864000000000001e-05 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986282+0000", 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5020000000000003e-06 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.987245+0000", 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00096307700000000001 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.987265+0000", 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0194999999999999e-05 2024-03-26T03:17:29.992 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.15978.0:45648 11.6 e416/403 11:698df864:::benchmark_data_smithi066_175179_object45647:head v 416'2874, mlcod=416'2871)", 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:29.986310+0000", 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 0.0014044050000000001, 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0015219509999999999, 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986310+0000", 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986310+0000", 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986311+0000", 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2200000000000003e-07 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986332+0000", 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1182000000000001e-05 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986338+0000", 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2600000000000002e-06 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986344+0000", 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.259e-06 2024-03-26T03:17:29.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986437+0000", 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3596000000000003e-05 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:29.986459+0000", 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1333999999999999e-05 2024-03-26T03:17:29.996 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:29.997 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:29.997 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:29.997 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:29.997 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:17:29.997 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 2 2024-03-26T03:17:29.997 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:17:29.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:30.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:30.132 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:30.180 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:30.282 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.334 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:30.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:30.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:30.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:30.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:17:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:17:30.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 23 16 46432 46416 126.114 93.6875 0.0100653 0.00792355 2024-03-26T03:17:30.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:30.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:30.701 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:30.744 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:30.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:30.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:30.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:30.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:31.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:31.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:31.164 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:31.266 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:31.267 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:31.279 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:17:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.342 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:31.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:31.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:31.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:31.495 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:17:31.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 24 15 47971 47956 124.867 96.25 0.0068163 0.00800205 2024-03-26T03:17:31.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:31.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:31.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:31.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:31.840 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:31.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[29666]: pgmap v875: 178 pgs: 178 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3881831358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[29666]: from='client.25807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[37920]: pgmap v875: 178 pgs: 178 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3881831358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:17:31.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:31 smithi066 ceph-mon[37920]: from='client.25807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:17:31.901 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:31.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:31 smithi192 ceph-mon[33110]: pgmap v875: 178 pgs: 178 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-03-26T03:17:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:31 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3881831358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:17:31.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:31 smithi192 ceph-mon[33110]: from='client.25807 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:17:32.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:32.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:32.137 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:32.183 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:32.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:32.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:32.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:32.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:32.461 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:32.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 25 16 49542 49526 123.797 98.125 0.0103035 0.00807207 2024-03-26T03:17:32.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:32.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:32.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:32.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:32.814 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:32.814 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[29666]: from='client.25807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:17:32.814 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[29666]: osdmap e417: 8 total, 8 up, 6 in 2024-03-26T03:17:32.814 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:17:32.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:17:32.815 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:32.815 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[37920]: from='client.25807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:17:32.815 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[37920]: osdmap e417: 8 total, 8 up, 6 in 2024-03-26T03:17:32.815 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:17:32.815 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:32 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:17:32.838 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:32.897 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:32.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:32 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:17:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:32 smithi192 ceph-mon[33110]: from='client.25807 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:17:32.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:32 smithi192 ceph-mon[33110]: osdmap e417: 8 total, 8 up, 6 in 2024-03-26T03:17:32.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:32 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:17:32.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:32 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:17:32.998 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:33.048 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:33.150 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:33.212 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:33.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.364 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:33.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:33.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:33.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:33.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 26 16 51187 51171 122.989 102.812 0.0101808 0.00812586 2024-03-26T03:17:33.603 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.652 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:33.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:33.753 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:33.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:33.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 2.19 scrub starts 2024-03-26T03:17:33.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 2.19 scrub ok 2024-03-26T03:17:33.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:17:33.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:17:33.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: pgmap v877: 178 pgs: 178 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-26T03:17:33.815 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 1.f scrub starts 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 1.f scrub ok 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 2.19 scrub starts 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 2.19 scrub ok 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: pgmap v877: 178 pgs: 178 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 1.f scrub starts 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 1.f scrub ok 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:17:33.816 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:17:33.916 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:33.971 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:33.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:33.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 2.19 scrub starts 2024-03-26T03:17:33.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 2.19 scrub ok 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: pgmap v877: 178 pgs: 178 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 1.f scrub starts 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 1.f scrub ok 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:17:33.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:33 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:17:34.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:34.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:34.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:17:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:17:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:17:34.207 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:34.261 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:34.363 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:34.409 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:34.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:34.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 27 15 52725 52710 121.996 96.1875 0.00844851 0.00819141 2024-03-26T03:17:34.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:34.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:34.651 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:34.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:34.779 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.834 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:34.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:34.871 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:17:34.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:34.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:34.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:35.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.115 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:35.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:35.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:35.270 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:35.372 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:35.424 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:35.472 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[29666]: osdmap e418: 8 total, 8 up, 6 in 2024-03-26T03:17:35.473 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[29666]: pgmap v879: 178 pgs: 178 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-26T03:17:35.473 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:17:35.473 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:17:35.473 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:17:35.473 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:17:35.473 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[37920]: osdmap e418: 8 total, 8 up, 6 in 2024-03-26T03:17:35.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[37920]: pgmap v879: 178 pgs: 178 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-26T03:17:35.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:17:35.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:17:35.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:17:35.474 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:35 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:17:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:35 smithi192 ceph-mon[33110]: osdmap e418: 8 total, 8 up, 6 in 2024-03-26T03:17:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:35 smithi192 ceph-mon[33110]: pgmap v879: 178 pgs: 178 active+clean; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-26T03:17:35.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:35 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:17:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:35 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:17:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:35 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:17:35.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:35 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:17:35.520 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 28 16 54147 54131 120.811 88.8125 0.00963801 0.00827218 2024-03-26T03:17:35.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:35.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:35.659 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.708 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:35.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:35.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:35.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:35.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:35.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:35.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:36.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:36.121 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:36.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:36.224 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:36.244 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T03:17:36.271 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:36.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:36.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:36 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:17:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:36 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:17:36.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:36 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3524853537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:17:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:36 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:17:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:36 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:17:36.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:36 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3524853537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:17:36.373 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:36.418 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:36.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:36.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:36.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 29 16 55488 55472 119.535 83.8125 0.016087 0.00835978 2024-03-26T03:17:36.572 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:36.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:36.675 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:36.722 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:36.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:36 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:17:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:36 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:17:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:36 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3524853537' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:17:36.824 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:36.868 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:36.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:36.969 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:37.015 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:37.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:37.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:37.253 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:37.298 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:37.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:37.426 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3524853537' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: osdmap e419: 8 total, 8 up, 6 in 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: pgmap v881: 178 pgs: 178 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3524853537' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:17:37.427 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: osdmap e419: 8 total, 8 up, 6 in 2024-03-26T03:17:37.428 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:17:37.428 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: pgmap v881: 178 pgs: 178 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-26T03:17:37.428 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:37 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:17:37.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:37.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 30 16 56963 56947 118.623 92.1875 0.0173045 0.00842445 2024-03-26T03:17:37.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:37.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:37.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:37.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:37.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:17:37.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:17:37.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3524853537' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:17:37.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: osdmap e419: 8 total, 8 up, 6 in 2024-03-26T03:17:37.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:17:37.745 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: pgmap v881: 178 pgs: 178 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-26T03:17:37.746 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:37 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:17:37.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:37.877 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:37.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:37.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:38.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:38.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.156 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:38.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:38.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:38.347 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:38.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:38.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:38.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 31 16 58442 58426 117.778 92.4375 0.0163219 0.00848552 2024-03-26T03:17:38.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:38.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:38 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:17:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:38 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:17:38.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/172303036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:38.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:38 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:17:38.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:38 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:17:38.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/172303036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:38.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:38.717 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:38 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:17:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:38 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:17:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/172303036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.763 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:38.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:38.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:38.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:38.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:38.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:17:39.000 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:17:39.001 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:17:39.001 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:17:39.001 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:17:39.001 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:17:39.001 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:17:39.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:39.134 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:39.181 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:39.283 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:39.330 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:39.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:39.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.475 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:39.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:39.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 32 16 59961 59945 117.064 94.9375 0.0081775 0.00853677 2024-03-26T03:17:39.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[29666]: pgmap v882: 178 pgs: 178 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 738 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-26T03:17:39.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[29666]: osdmap e420: 8 total, 8 up, 6 in 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.12", "id": [5, 4]}]: dispatch 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[37920]: pgmap v882: 178 pgs: 178 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 738 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[37920]: osdmap e420: 8 total, 8 up, 6 in 2024-03-26T03:17:39.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:39 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.12", "id": [5, 4]}]: dispatch 2024-03-26T03:17:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:39.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:39.714 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:39 smithi192 ceph-mon[33110]: pgmap v882: 178 pgs: 178 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 738 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-26T03:17:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:39 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:17:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:39 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:17:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:39 smithi192 ceph-mon[33110]: osdmap e420: 8 total, 8 up, 6 in 2024-03-26T03:17:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:39 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.12", "id": [5, 4]}]: dispatch 2024-03-26T03:17:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.761 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:39.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:39.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:39.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:39.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:40.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:40.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:40.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:40.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:40.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:40.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:40.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:40.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:40.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:17:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:17:40.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:40 smithi192 ceph-mon[33110]: 1.b scrub starts 2024-03-26T03:17:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:40 smithi192 ceph-mon[33110]: 1.b scrub ok 2024-03-26T03:17:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.12", "id": [5, 4]}]': finished 2024-03-26T03:17:40.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:40 smithi192 ceph-mon[33110]: osdmap e421: 8 total, 8 up, 6 in 2024-03-26T03:17:40.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 33 16 61538 61522 116.503 98.5625 0.00681141 0.00857833 2024-03-26T03:17:40.580 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:40.601 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[29666]: 1.b scrub starts 2024-03-26T03:17:40.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[29666]: 1.b scrub ok 2024-03-26T03:17:40.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:40.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.12", "id": [5, 4]}]': finished 2024-03-26T03:17:40.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[29666]: osdmap e421: 8 total, 8 up, 6 in 2024-03-26T03:17:40.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[37920]: 1.b scrub starts 2024-03-26T03:17:40.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[37920]: 1.b scrub ok 2024-03-26T03:17:40.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:40.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.12", "id": [5, 4]}]': finished 2024-03-26T03:17:40.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:40 smithi066 ceph-mon[37920]: osdmap e421: 8 total, 8 up, 6 in 2024-03-26T03:17:40.646 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:40.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:40.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:40.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:40.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:40.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:40.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:41.016 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:41.062 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:41.164 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:41.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:41.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:41.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:41.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:41.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:41.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 34 16 63077 63061 115.906 96.1875 0.00339663 0.00862316 2024-03-26T03:17:41.550 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: 6.8 scrub starts 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: 6.8 scrub ok 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: pgmap v885: 178 pgs: 178 active+clean; 3.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[29666]: osdmap e422: 8 total, 8 up, 6 in 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:41.551 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:41.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: 6.8 scrub starts 2024-03-26T03:17:41.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: 6.8 scrub ok 2024-03-26T03:17:41.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: pgmap v885: 178 pgs: 178 active+clean; 3.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-26T03:17:41.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:17:41.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:17:41.552 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:41 smithi066 ceph-mon[37920]: osdmap e422: 8 total, 8 up, 6 in 2024-03-26T03:17:41.578 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:41.624 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:41.659 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:41.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: 6.8 scrub starts 2024-03-26T03:17:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: 6.8 scrub ok 2024-03-26T03:17:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: pgmap v885: 178 pgs: 178 active+clean; 3.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 132 MiB/s wr, 2.11k op/s 2024-03-26T03:17:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:17:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:17:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:41 smithi192 ceph-mon[33110]: osdmap e422: 8 total, 8 up, 6 in 2024-03-26T03:17:41.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:41.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:41.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:41.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:42.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.036 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:42.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:42.061 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:17:42.062 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:17:42.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:42.182 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:42.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:42.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:42.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:42.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:42.521 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 35 16 64488 64472 115.114 88.1875 0.00643529 0.00868168 2024-03-26T03:17:42.547 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.581 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:42.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:17:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:17:42.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:17:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:17:42.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[29666]: osdmap e423: 8 total, 8 up, 6 in 2024-03-26T03:17:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:17:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:17:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:17:42.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:17:42.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:42 smithi066 ceph-mon[37920]: osdmap e423: 8 total, 8 up, 6 in 2024-03-26T03:17:42.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:42.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:42 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:17:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:42 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:17:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:42 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:17:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:42 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:17:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:42 smithi192 ceph-mon[33110]: osdmap e423: 8 total, 8 up, 6 in 2024-03-26T03:17:42.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T03:17:42.815 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:42.893 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:42.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:42.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:43.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:43.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:43.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:43.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:43.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:43.329 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:17:43.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:43.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:43.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 36 16 65907 65891 114.377 88.6875 0.0227119 0.00873738 2024-03-26T03:17:43.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:43.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:43.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: 4.e scrub starts 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: 4.e scrub ok 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: pgmap v888: 178 pgs: 178 active+clean; 3.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[29666]: osdmap e424: 8 total, 8 up, 6 in 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: 4.e scrub starts 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: 4.e scrub ok 2024-03-26T03:17:43.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: pgmap v888: 178 pgs: 178 active+clean; 3.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-26T03:17:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:17:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:17:43.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-mon[37920]: osdmap e424: 8 total, 8 up, 6 in 2024-03-26T03:17:43.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:43.716 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:17:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:17:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: 4.e scrub starts 2024-03-26T03:17:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: 4.e scrub ok 2024-03-26T03:17:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: pgmap v888: 178 pgs: 178 active+clean; 3.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-26T03:17:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:17:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:17:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:43 smithi192 ceph-mon[33110]: osdmap e424: 8 total, 8 up, 6 in 2024-03-26T03:17:43.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:43.913 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:17:43.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:43.982 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:43.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:44.039 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:44.069 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:44.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:44.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:17:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:17:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:17:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:44.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:44.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:44.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:44.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:44.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:44.476 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.523 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:44.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:44.523 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 37 15 67161 67146 113.405 78.4375 0.0128681 0.00881306 2024-03-26T03:17:44.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 5.a scrub starts 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 5.a scrub ok 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[29666]: osdmap e425: 8 total, 8 up, 6 in 2024-03-26T03:17:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 5.a scrub starts 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 5.a scrub ok 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:17:44.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:44 smithi066 ceph-mon[37920]: osdmap e425: 8 total, 8 up, 6 in 2024-03-26T03:17:44.624 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.657 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:44.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 5.a scrub starts 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 5.a scrub ok 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:17:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:44 smithi192 ceph-mon[33110]: osdmap e425: 8 total, 8 up, 6 in 2024-03-26T03:17:44.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:44.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:44.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:44.935 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:44.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:45.039 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:45.083 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:45.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:45.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:45.314 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:45.372 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:45.474 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:45.517 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:45.522 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:45 smithi066 ceph-mon[37920]: 6.c scrub starts 2024-03-26T03:17:45.522 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:45 smithi066 ceph-mon[37920]: 6.c scrub ok 2024-03-26T03:17:45.522 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:45 smithi066 ceph-mon[37920]: pgmap v891: 178 pgs: 178 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 35 MiB/s wr, 552 op/s 2024-03-26T03:17:45.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:45 smithi066 ceph-mon[29666]: 6.c scrub starts 2024-03-26T03:17:45.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:45 smithi066 ceph-mon[29666]: 6.c scrub ok 2024-03-26T03:17:45.523 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:45 smithi066 ceph-mon[29666]: pgmap v891: 178 pgs: 178 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 35 MiB/s wr, 552 op/s 2024-03-26T03:17:45.523 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 38 16 68601 68585 112.788 89.9375 0.0160319 0.00886052 2024-03-26T03:17:45.619 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:45.663 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:45 smithi192 ceph-mon[33110]: 6.c scrub starts 2024-03-26T03:17:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:45 smithi192 ceph-mon[33110]: 6.c scrub ok 2024-03-26T03:17:45.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:45 smithi192 ceph-mon[33110]: pgmap v891: 178 pgs: 178 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 35 MiB/s wr, 552 op/s 2024-03-26T03:17:45.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:45.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:45.895 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:45.948 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:45.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:46.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:46.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:46.181 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:46.232 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:46.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:46.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:46.438 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[29666]: 4.1b scrub starts 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[29666]: 4.1b scrub ok 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:17:46.439 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[37920]: 4.1b scrub starts 2024-03-26T03:17:46.440 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:46 smithi066 ceph-mon[37920]: 4.1b scrub ok 2024-03-26T03:17:46.467 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:46.511 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:46.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 39 16 70091 70075 112.283 93.125 0.0075657 0.00890129 2024-03-26T03:17:46.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:46.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:46 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:17:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:46 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:17:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:46 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:17:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:46 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:17:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:46 smithi192 ceph-mon[33110]: 4.1b scrub starts 2024-03-26T03:17:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:46 smithi192 ceph-mon[33110]: 4.1b scrub ok 2024-03-26T03:17:46.746 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.792 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:46.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:46.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:46.947 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:46.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:47.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.096 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:47.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:47.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:47.220 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:17:47.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:47.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:47.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:47.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:47.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:47.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:17:47.522602+0000 min lat: 0.00143665 max lat: 0.149616 avg lat: 0.00893227 2024-03-26T03:17:47.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:17:47.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 40 16 71635 71619 111.889 96.5 0.0120499 0.00893227 2024-03-26T03:17:47.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:47.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:47 smithi066 ceph-mon[29666]: pgmap v892: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 3.7 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 15 KiB/s, 0 objects/s recovering 2024-03-26T03:17:47.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:47 smithi066 ceph-mon[37920]: pgmap v892: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 3.7 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 15 KiB/s, 0 objects/s recovering 2024-03-26T03:17:47.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:47.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:47 smithi192 ceph-mon[33110]: pgmap v892: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 3.7 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 15 KiB/s, 0 objects/s recovering 2024-03-26T03:17:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:47.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:47.867 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:47.912 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:47.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:48.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:48.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:48.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:48.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:48.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:48.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:48.421 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:48.469 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:48.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 41 16 73069 73053 111.345 89.625 0.0100697 0.00897643 2024-03-26T03:17:48.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:48.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:48 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:17:48.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:48 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:17:48.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:48 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:17:48.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:48 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:17:48.706 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:48 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:17:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:48 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:17:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.754 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:48.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:48.855 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:48.897 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:48.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:48.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:49.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:49.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:49.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:49.264 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:49.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:49.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:49.454 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:49.522 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 42 16 74538 74522 110.88 91.8125 0.0135294 0.00901378 2024-03-26T03:17:49.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:49.602 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:49.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:49.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:17:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:17:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:17:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:17:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: pgmap v893: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 3.1 KiB/s rd, 73 MiB/s wr, 1.16k op/s; 13 KiB/s, 0 objects/s recovering 2024-03-26T03:17:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:17:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:17:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:49 smithi192 ceph-mon[33110]: osdmap e426: 8 total, 8 up, 6 in 2024-03-26T03:17:49.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: pgmap v893: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 3.1 KiB/s rd, 73 MiB/s wr, 1.16k op/s; 13 KiB/s, 0 objects/s recovering 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[29666]: osdmap e426: 8 total, 8 up, 6 in 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:17:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:17:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:17:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:17:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: pgmap v893: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 3.1 KiB/s rd, 73 MiB/s wr, 1.16k op/s; 13 KiB/s, 0 objects/s recovering 2024-03-26T03:17:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:17:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:17:49.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:49 smithi066 ceph-mon[37920]: osdmap e426: 8 total, 8 up, 6 in 2024-03-26T03:17:49.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:49.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:49.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:49.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:50.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:50.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:50.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:50.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:50.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:50.373 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:17:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:17:50.411 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:50.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:50.523 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 43 15 75957 75942 110.362 88.75 0.00933283 0.00905626 2024-03-26T03:17:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:50.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:50.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:50.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:50 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:17:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:50 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:17:50.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:50.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:50.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:50 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:17:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:50 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:17:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:50 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:17:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:50 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:17:50.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:50.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:50.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:51.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:51.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:51.196 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.250 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:51.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:51.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:51.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:51.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:51.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:51.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 44 16 77505 77489 110.051 96.6875 0.0102833 0.00908196 2024-03-26T03:17:51.618 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.659 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:51.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:51 smithi192 ceph-mon[33110]: 5.14 scrub starts 2024-03-26T03:17:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:51 smithi192 ceph-mon[33110]: 5.14 scrub ok 2024-03-26T03:17:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:51 smithi192 ceph-mon[33110]: pgmap v895: 178 pgs: 178 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-26T03:17:51.760 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.799 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:51.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:51.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:51.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[29666]: 5.14 scrub starts 2024-03-26T03:17:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[29666]: 5.14 scrub ok 2024-03-26T03:17:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[29666]: pgmap v895: 178 pgs: 178 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-26T03:17:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:17:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[37920]: 5.14 scrub starts 2024-03-26T03:17:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[37920]: 5.14 scrub ok 2024-03-26T03:17:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:51 smithi066 ceph-mon[37920]: pgmap v895: 178 pgs: 178 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-26T03:17:51.901 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:51.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:51.948 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:51.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:52.049 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:52.099 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:52.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:52.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:52.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:52.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:52.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:52.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:52.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 45 16 78934 78918 109.59 89.3125 0.011199 0.00912014 2024-03-26T03:17:52.598 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.648 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:52.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:52.685 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:52 smithi066 ceph-mon[29666]: 1.1d scrub starts 2024-03-26T03:17:52.685 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:52 smithi066 ceph-mon[29666]: 1.1d scrub ok 2024-03-26T03:17:52.686 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:52 smithi066 ceph-mon[37920]: 1.1d scrub starts 2024-03-26T03:17:52.686 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:52 smithi066 ceph-mon[37920]: 1.1d scrub ok 2024-03-26T03:17:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:52 smithi192 ceph-mon[33110]: 1.1d scrub starts 2024-03-26T03:17:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:52 smithi192 ceph-mon[33110]: 1.1d scrub ok 2024-03-26T03:17:52.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:52.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:52.883 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:52.929 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:52.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:53.030 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:53.077 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:53.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:53.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:53.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:53.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:53.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:53.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:53.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 46 16 80432 80416 109.243 93.625 0.0135428 0.00914875 2024-03-26T03:17:53.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:53.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:53.722 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:53 smithi192 ceph-mon[33110]: 6.2 scrub starts 2024-03-26T03:17:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:53 smithi192 ceph-mon[33110]: 6.2 scrub ok 2024-03-26T03:17:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:53 smithi192 ceph-mon[33110]: pgmap v896: 178 pgs: 178 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T03:17:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:53 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:17:53.776 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:53.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[29666]: 6.2 scrub starts 2024-03-26T03:17:53.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[29666]: 6.2 scrub ok 2024-03-26T03:17:53.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[29666]: pgmap v896: 178 pgs: 178 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T03:17:53.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:17:53.792 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[37920]: 6.2 scrub starts 2024-03-26T03:17:53.793 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[37920]: 6.2 scrub ok 2024-03-26T03:17:53.793 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[37920]: pgmap v896: 178 pgs: 178 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T03:17:53.793 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:17:53.878 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:53.934 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:54.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:54.044 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:17:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:17:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:17:54.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:54.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:54.233 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:54.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:54.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:54.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:54.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:54.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 47 15 81693 81678 108.597 78.875 0.012055 0.00920345 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:17:54.600 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:54 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:17:54.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:54.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:54.732 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:54 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:17:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:54 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:17:54.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:54 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:17:54.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:54 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:17:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.779 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:54.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:54.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:54.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:54.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:55.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.063 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:55.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:55.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:55.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:55.299 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:55.346 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:55.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:55.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:55.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 48 16 83006 82990 108.043 82 0.00579723 0.00925071 2024-03-26T03:17:55.564 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T03:17:55.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:55.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:55.715 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: 1.5 scrub starts 2024-03-26T03:17:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: 1.5 scrub ok 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: pgmap v897: 178 pgs: 178 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1893549992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1930852657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:17:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:55 smithi192 ceph-mon[33110]: from='client.25828 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:17:55.743 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:55.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: 1.5 scrub starts 2024-03-26T03:17:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: 1.5 scrub ok 2024-03-26T03:17:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: pgmap v897: 178 pgs: 178 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T03:17:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:17:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:17:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:17:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:17:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:17:55.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1893549992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:55.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:17:55.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:17:55.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:55.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1930852657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:17:55.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[29666]: from='client.25828 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:17:55.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: 1.5 scrub starts 2024-03-26T03:17:55.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: 1.5 scrub ok 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: pgmap v897: 178 pgs: 178 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1893549992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:17:55.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1930852657' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:17:55.864 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:55 smithi066 ceph-mon[37920]: from='client.25828 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:17:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:55.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:55.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:55.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.000 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:56.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:56.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:56.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:56.227 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:56.256 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:56.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:56.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:17:56.481 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:17:56.482 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:17:56.482 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:17:56.482 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:17:56.482 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:17:56.483 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:17:56.483 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:17:56.483 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:17:56.483 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:17:56.483 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:17:56.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.513 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:56.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:56.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 49 16 83155 83139 106.028 9.3125 0.00295972 0.00925261 2024-03-26T03:17:56.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:17:56.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:17:56.740 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:56.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:56.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: 5.0 scrub starts 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: 5.0 scrub ok 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: osdmap e427: 8 total, 8 up, 6 in 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: 1.17 scrub starts 2024-03-26T03:17:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[29666]: 1.17 scrub ok 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: 5.0 scrub starts 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: 5.0 scrub ok 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: osdmap e427: 8 total, 8 up, 6 in 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: 1.17 scrub starts 2024-03-26T03:17:56.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:56 smithi066 ceph-mon[37920]: 1.17 scrub ok 2024-03-26T03:17:56.870 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:56.899 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:56.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:17:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:17:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: 5.0 scrub starts 2024-03-26T03:17:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: 5.0 scrub ok 2024-03-26T03:17:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: from='client.25828 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:17:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: osdmap e427: 8 total, 8 up, 6 in 2024-03-26T03:17:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: 1.17 scrub starts 2024-03-26T03:17:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:56 smithi192 ceph-mon[33110]: 1.17 scrub ok 2024-03-26T03:17:56.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:57.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:57.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:57.156 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:57.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:57.287 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:57.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:57.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:57.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:57.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 50 16 83155 83139 103.908 0 - 0.00925261 2024-03-26T03:17:57.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:57.568 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:17:57.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:57.678 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:57.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:57.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:57.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[29666]: pgmap v899: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-26T03:17:57.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[29666]: osdmap e428: 8 total, 8 up, 6 in 2024-03-26T03:17:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3271817546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[29666]: from='client.35888 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[37920]: pgmap v899: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-26T03:17:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[37920]: osdmap e428: 8 total, 8 up, 6 in 2024-03-26T03:17:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3271817546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:57 smithi066 ceph-mon[37920]: from='client.35888 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:57.915 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:57.959 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:57.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:57 smithi192 ceph-mon[33110]: pgmap v899: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-26T03:17:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:57 smithi192 ceph-mon[33110]: osdmap e428: 8 total, 8 up, 6 in 2024-03-26T03:17:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3271817546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:57 smithi192 ceph-mon[33110]: from='client.35888 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:17:58.061 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:17:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.109 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:58.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:17:58.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.261 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:58.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:58.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:58.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:58.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:58.524 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 51 16 84312 84296 103.288 36.1562 0.0138471 0.0096763 2024-03-26T03:17:58.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:58.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:58.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:58.766 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:17:58.810 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 4.1a scrub starts 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 4.1a scrub ok 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 2.e scrub starts 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 2.e scrub ok 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: from='client.35888 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: osdmap e429: 8 total, 8 up, 6 in 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 11.c starting backfill to osd.4 from (0'0,0'0] MAX to 426'5085 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 11.e starting backfill to osd.4 from (0'0,0'0] MAX to 426'5130 2024-03-26T03:17:58.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 426'5192 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 4.1a scrub starts 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 4.1a scrub ok 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 2.e scrub starts 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 2.e scrub ok 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: from='client.35888 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: osdmap e429: 8 total, 8 up, 6 in 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 11.c starting backfill to osd.4 from (0'0,0'0] MAX to 426'5085 2024-03-26T03:17:58.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 11.e starting backfill to osd.4 from (0'0,0'0] MAX to 426'5130 2024-03-26T03:17:58.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 426'5192 2024-03-26T03:17:58.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:17:58.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:58 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:17:58.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:58.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:58.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 4.1a scrub starts 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 4.1a scrub ok 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 2.e scrub starts 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 2.e scrub ok 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: from='client.35888 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: osdmap e429: 8 total, 8 up, 6 in 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 11.c starting backfill to osd.4 from (0'0,0'0] MAX to 426'5085 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 11.e starting backfill to osd.4 from (0'0,0'0] MAX to 426'5130 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 11.b starting backfill to osd.4 from (0'0,0'0] MAX to 426'5192 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:17:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:58 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:17:59.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:17:59.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:17:59.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:59.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:59.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:17:59.372 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:17:59.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137118 10.5 10:afbc2354:::benchmark_data_smithi066_148944_object12471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.188180+0000", 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.290197869, 2024-03-26T03:17:59.486 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.265863966, 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137118 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188180+0000", 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.487 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188180+0000", 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188181+0000", 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2100000000000001e-06 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188185+0000", 2024-03-26T03:17:59.488 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4789999999999998e-06 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188198+0000", 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2984e-05 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188206+0000", 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9969999999999998e-06 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.489 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.203599+0000", 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.015392407 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.203733+0000", 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000134074 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,7", 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.203871+0000", 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000138541 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.490 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.213765+0000", 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0098930890000000007 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.213865+0000", 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100188 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.213922+0000", 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7179999999999998e-05 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.491 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.248852+0000", 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.034929819000000001 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.248864+0000", 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2397e-05 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.453940+0000", 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2050763520000001 2024-03-26T03:17:59.492 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.454044+0000", 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000103249 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137142 10.5 10:a16c8a26:::benchmark_data_smithi066_148944_object12495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.215641+0000", 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.262736528, 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4465264339999999, 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.493 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137142 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215641+0000", 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.494 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215641+0000", 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215646+0000", 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3460000000000001e-06 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215652+0000", 2024-03-26T03:17:59.495 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.891e-06 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215663+0000", 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0343000000000001e-05 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215672+0000", 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1600000000000004e-06 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.249222+0000", 2024-03-26T03:17:59.496 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033549764000000003 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.249230+0000", 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6799999999999999e-06 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.455981+0000", 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2067507580000001 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.456156+0000", 2024-03-26T03:17:59.497 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000175243 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.456608+0000", 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00045140399999999998 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465016+0000", 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0084080200000000004 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.498 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465111+0000", 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5457000000000004e-05 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465149+0000", 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8049000000000001e-05 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465360+0000", 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021100700000000001 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.499 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.660380+0000", 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19501934300000001 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.660419+0000", 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9712999999999997e-05 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.662168+0000", 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017482559999999999 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.500 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137155 10.8 10:1e4e6b57:::benchmark_data_smithi066_148944_object12508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.256026+0000", 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.222351546, 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3845877340000001, 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137155 2024-03-26T03:17:59.501 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256026+0000", 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256026+0000", 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.502 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256028+0000", 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0250000000000001e-06 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256035+0000", 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.477e-06 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256047+0000", 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.236e-05 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.503 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256057+0000", 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.825e-06 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256130+0000", 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2993999999999999e-05 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256148+0000", 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.859e-05 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.504 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230452+0000", 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97430318100000002 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230465+0000", 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3777999999999999e-05 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.321474+0000", 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0910083530000001 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.505 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.321615+0000", 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000141803 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.322034+0000", 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000418694 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331259+0000", 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0092250690000000007 2024-03-26T03:17:59.506 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.337787+0000", 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0065275790000000004 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339170+0000", 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001382912 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640470+0000", 2024-03-26T03:17:59.507 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.30130003300000002 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640511+0000", 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1180999999999998e-05 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640614+0000", 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010288 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.508 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137157 10.5 10:ac4af61a:::benchmark_data_smithi066_148944_object12510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.260244+0000", 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.21813343700001, 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.403910816, 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137157 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.509 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260244+0000", 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260244+0000", 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260246+0000", 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9039999999999999e-06 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.510 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260253+0000", 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7429999999999996e-06 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260266+0000", 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2982999999999999e-05 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260274+0000", 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.106e-06 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.511 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260343+0000", 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8911999999999993e-05 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260353+0000", 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6690000000000001e-06 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.462763+0000", 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.202410202 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.512 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.513 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.462909+0000", 2024-03-26T03:17:59.513 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000146649 2024-03-26T03:17:59.513 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.513 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.513 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.463028+0000", 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118212 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465296+0000", 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002267958 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465316+0000", 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0967000000000001e-05 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.514 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465412+0000", 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5011000000000002e-05 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470453+0000", 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0050418879999999996 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.662290+0000", 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19183647000000001 2024-03-26T03:17:59.515 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.662359+0000", 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9095000000000002e-05 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.664155+0000", 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017960470000000001 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.516 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137160 10.8 10:1c0d5c53:::benchmark_data_smithi066_148944_object12513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.263557+0000", 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.214820441, 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.377314406, 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137160 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.517 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263557+0000", 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263557+0000", 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263559+0000", 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.711e-06 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.518 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263564+0000", 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4380000000000003e-06 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263573+0000", 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6300000000000004e-06 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263579+0000", 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3550000000000001e-06 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.519 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263624+0000", 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4768000000000002e-05 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263632+0000", 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0369999999999995e-06 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230498+0000", 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96686540200000004 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.520 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230506+0000", 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9129999999999992e-06 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.327883+0000", 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.097376098 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.327983+0000", 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100819 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.521 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.328068+0000", 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4331000000000003e-05 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332345+0000", 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042777809999999996 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.337829+0000", 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.005483936 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.522 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339214+0000", 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001384461 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640670+0000", 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.30145598099999998 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640734+0000", 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3977000000000001e-05 2024-03-26T03:17:59.523 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640872+0000", 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000137768 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137161 10.5 10:a7b10033:::benchmark_data_smithi066_148944_object12514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.265129+0000", 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.21324821300001, 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4080903339999999, 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.524 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137161 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265129+0000", 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.525 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265129+0000", 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265131+0000", 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.883e-06 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.526 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265138+0000", 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5910000000000001e-06 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265155+0000", 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7037999999999998e-05 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265164+0000", 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0140000000000004e-06 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.527 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265216+0000", 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2194000000000002e-05 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265227+0000", 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1316e-05 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465450+0000", 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.200222592 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.528 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465617+0000", 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016646799999999999 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465755+0000", 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013894299999999999 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470500+0000", 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047449329999999998 2024-03-26T03:17:59.529 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470541+0000", 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0467999999999999e-05 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475179+0000", 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046377140000000002 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.530 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475254+0000", 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4977999999999999e-05 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670899+0000", 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.195645029 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670944+0000", 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5716000000000001e-05 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.531 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.673220+0000", 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022754569999999998 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137169 10.8 10:102fb2c0:::benchmark_data_smithi066_148944_object12522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.277033+0000", 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.201344664, 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3641067549999999, 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.532 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137169 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277033+0000", 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277033+0000", 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.533 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277036+0000", 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.9730000000000002e-06 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277046+0000", 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8179999999999999e-06 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277069+0000", 2024-03-26T03:17:59.534 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2932e-05 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277080+0000", 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1245e-05 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277187+0000", 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000107459 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277206+0000", 2024-03-26T03:17:59.535 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9026999999999999e-05 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230531+0000", 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95332462299999998 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230539+0000", 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2120000000000003e-06 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.329290+0000", 2024-03-26T03:17:59.536 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.098750739 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.329363+0000", 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2788999999999997e-05 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.329439+0000", 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6247000000000004e-05 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332355+0000", 2024-03-26T03:17:59.537 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002916189 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339249+0000", 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0068942120000000003 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.343995+0000", 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047458589999999998 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.538 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640965+0000", 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29696974700000001 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641026+0000", 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1428999999999997e-05 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641140+0000", 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113255 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.539 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137172 10.5 10:ace37db4:::benchmark_data_smithi066_148944_object12525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.277612+0000", 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.20076606999999, 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.398316208, 2024-03-26T03:17:59.540 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137172 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277612+0000", 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.541 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277612+0000", 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277613+0000", 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.517e-06 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.542 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277619+0000", 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4239999999999998e-06 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277633+0000", 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3534e-05 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277643+0000", 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9280000000000005e-06 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.543 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277748+0000", 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000105078 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277759+0000", 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0529e-05 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.467847+0000", 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.190088931 2024-03-26T03:17:59.544 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.467995+0000", 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014706800000000001 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.468143+0000", 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000148209 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470590+0000", 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002447187 2024-03-26T03:17:59.545 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475240+0000", 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046503409999999997 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475258+0000", 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8178000000000001e-05 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475378+0000", 2024-03-26T03:17:59.546 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011933400000000001 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.673285+0000", 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19790735300000001 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.673325+0000", 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0071999999999999e-05 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.547 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.675928+0000", 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026025250000000001 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137173 10.5 10:a3a4d821:::benchmark_data_smithi066_148944_object12526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.277668+0000", 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.20070953600001, 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4018042610000001, 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.548 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137173 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277668+0000", 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277668+0000", 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.549 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277669+0000", 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8299999999999996e-07 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277673+0000", 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3100000000000002e-06 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.550 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277681+0000", 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9289999999999998e-06 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277688+0000", 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6250000000000001e-06 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277784+0000", 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6658999999999997e-05 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.551 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277791+0000", 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7959999999999998e-06 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470626+0000", 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1928344019999999 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470770+0000", 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014413499999999999 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.552 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470931+0000", 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000160917 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475263+0000", 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043327280000000001 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475317+0000", 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3677999999999997e-05 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.553 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.554 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475478+0000", 2024-03-26T03:17:59.554 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000161201 2024-03-26T03:17:59.554 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.554 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.554 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.554 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477904+0000", 2024-03-26T03:17:59.554 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 52 16 85612 85596 102.864 81.25 0.0101071 0.00971615 2024-03-26T03:17:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.555 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:59.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024256159999999998 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.676369+0000", 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.198464741 2024-03-26T03:17:59.556 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.676435+0000", 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.601e-05 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.679472+0000", 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030377310000000001 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.557 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137178 10.5 10:a07e3bbe:::benchmark_data_smithi066_148944_object12531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.284294+0000", 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.19408348299999, 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.39794836, 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137178 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.558 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284294+0000", 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284294+0000", 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284296+0000", 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.798e-06 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.559 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284301+0000", 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.797e-06 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284309+0000", 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7299999999999994e-06 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284317+0000", 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4800000000000004e-06 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.560 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.303364+0000", 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019046674999999999 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.303373+0000", 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5100000000000004e-06 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.472923+0000", 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1695500619999999 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.561 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.473065+0000", 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014189500000000001 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.473206+0000", 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014065700000000001 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475390+0000", 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002184189 2024-03-26T03:17:59.562 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475435+0000", 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5034999999999998e-05 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477283+0000", 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018481120000000001 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477960+0000", 2024-03-26T03:17:59.563 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00067646199999999996 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.679536+0000", 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20157619900000001 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.679603+0000", 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7126999999999996e-05 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.682243+0000", 2024-03-26T03:17:59.564 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026396319999999998 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137190 10.8 10:195bf631:::benchmark_data_smithi066_148944_object12543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.297120+0000", 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.18125775199999, 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3442652210000001, 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.565 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137190 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297120+0000", 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297120+0000", 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.566 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297121+0000", 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4589999999999999e-06 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297127+0000", 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0009999999999998e-06 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297138+0000", 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1158999999999999e-05 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.567 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297148+0000", 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5889999999999992e-06 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297206+0000", 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8326999999999999e-05 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297213+0000", 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8050000000000001e-06 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.568 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230563+0000", 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93334995799999998 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230571+0000", 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1419999999999993e-06 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.330312+0000", 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.099740924 2024-03-26T03:17:59.569 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.330386+0000", 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3553999999999994e-05 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.330464+0000", 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7879000000000004e-05 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334292+0000", 2024-03-26T03:17:59.570 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038279049999999999 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339277+0000", 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049859120000000003 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.344065+0000", 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047876869999999997 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641197+0000", 2024-03-26T03:17:59.571 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29713157400000001 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641241+0000", 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4073999999999998e-05 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641385+0000", 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000144272 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.572 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137198 10.5 10:a2491f0f:::benchmark_data_smithi066_148944_object12551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.315521+0000", 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 203.16285686500001, 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.36946773, 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137198 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.573 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315521+0000", 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315521+0000", 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315523+0000", 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.829e-06 2024-03-26T03:17:59.574 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315530+0000", 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1579999999999999e-06 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315545+0000", 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5211999999999999e-05 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315554+0000", 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3479999999999993e-06 2024-03-26T03:17:59.575 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315608+0000", 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3573999999999999e-05 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315615+0000", 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6889999999999998e-06 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475505+0000", 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1598899730000001 2024-03-26T03:17:59.576 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475593+0000", 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8819000000000005e-05 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475684+0000", 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0387e-05 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477331+0000", 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001646822 2024-03-26T03:17:59.577 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477355+0000", 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4907000000000001e-05 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477692+0000", 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00033635099999999998 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.478062+0000", 2024-03-26T03:17:59.578 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00036998899999999998 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.682301+0000", 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.204239217 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.682371+0000", 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0308000000000005e-05 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.684988+0000", 2024-03-26T03:17:59.579 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026171469999999998 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137230 10.8 10:19c7df3e:::benchmark_data_smithi066_148944_object12583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.670559+0000", 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.807818865, 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97105015299999997, 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.580 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137230 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670559+0000", 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670559+0000", 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.581 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670560+0000", 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6700000000000001e-06 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670566+0000", 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.327e-06 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.582 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670576+0000", 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0475999999999999e-05 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670583+0000", 2024-03-26T03:17:59.583 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6000000000000003e-06 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670649+0000", 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5718999999999999e-05 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670663+0000", 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3971999999999999e-05 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.584 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331415+0000", 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.660752322 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331490+0000", 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4967000000000002e-05 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331579+0000", 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9543000000000006e-05 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.585 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336904+0000", 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053242899999999998 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339335+0000", 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024316820000000001 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.344103+0000", 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047680020000000004 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.586 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641431+0000", 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29732734 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641483+0000", 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1867999999999999e-05 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641609+0000", 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000126375 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.587 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137239 10.8 10:1ce3fc92:::benchmark_data_smithi066_148944_object12592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.683080+0000", 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.795297686, 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95876216599999997, 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137239 2024-03-26T03:17:59.588 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683080+0000", 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683080+0000", 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.589 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683082+0000", 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0600000000000002e-06 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683088+0000", 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3890000000000002e-06 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683104+0000", 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5648999999999999e-05 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.590 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683113+0000", 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5790000000000004e-06 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683196+0000", 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3761e-05 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683208+0000", 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1533e-05 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.591 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332389+0000", 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.64918131899999998 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332464+0000", 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4837000000000002e-05 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332546+0000", 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1971999999999994e-05 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.592 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336911+0000", 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043649839999999997 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339365+0000", 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024541720000000001 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.346280+0000", 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069149889999999999 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.593 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641659+0000", 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29537875899999999 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641713+0000", 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3622000000000003e-05 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641842+0000", 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000129541 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.594 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137244 10.8 10:16d9ddfd:::benchmark_data_smithi066_148944_object12597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.79261208400001, 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95632148400000005, 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.595 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137244 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.596 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3300000000000001e-07 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685769+0000", 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.066e-06 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685775+0000", 2024-03-26T03:17:59.597 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3040000000000004e-06 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685780+0000", 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1059999999999999e-06 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685841+0000", 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1367999999999999e-05 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.598 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685853+0000", 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1452e-05 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.333336+0000", 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.64748331100000001 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.333409+0000", 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2689999999999997e-05 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.599 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.333481+0000", 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2821e-05 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336919+0000", 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0034379860000000001 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339397+0000", 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002477726 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.600 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.346327+0000", 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069297710000000004 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641902+0000", 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29557549100000002 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641961+0000", 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8820000000000003e-05 2024-03-26T03:17:59.601 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642087+0000", 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012583900000000001 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137255 10.8 10:1f88f654:::benchmark_data_smithi066_148944_object12608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.708111+0000", 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.77026625600001, 2024-03-26T03:17:59.602 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93419730499999998, 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137255 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708111+0000", 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.603 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708111+0000", 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708113+0000", 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.339e-06 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708118+0000", 2024-03-26T03:17:59.604 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6910000000000002e-06 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708132+0000", 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3526000000000001e-05 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708141+0000", 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0040000000000005e-06 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708195+0000", 2024-03-26T03:17:59.605 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4114999999999999e-05 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708205+0000", 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7079999999999992e-06 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334330+0000", 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.62612499600000004 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.606 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334401+0000", 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1440999999999996e-05 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334487+0000", 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5426999999999994e-05 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336925+0000", 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024385259999999999 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.607 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339541+0000", 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026162009999999999 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.346355+0000", 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0068138799999999996 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642132+0000", 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29577706300000001 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.608 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642184+0000", 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1867000000000003e-05 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642309+0000", 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000124521 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137258 10.8 10:1812d821:::benchmark_data_smithi066_148944_object12611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:17:59.609 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.730073+0000", 2024-03-26T03:17:59.610 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 201.74830422100001, 2024-03-26T03:17:59.610 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0739304270000001, 2024-03-26T03:17:59.610 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137258 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730073+0000", 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.611 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730073+0000", 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730075+0000", 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9089999999999998e-06 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730081+0000", 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.366e-06 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.612 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730092+0000", 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1022000000000001e-05 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730098+0000", 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.584e-06 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730152+0000", 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3817999999999999e-05 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.613 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730162+0000", 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5100000000000004e-06 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.335310+0000", 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.60514800899999999 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.335439+0000", 2024-03-26T03:17:59.614 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012949700000000001 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.335583+0000", 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000143836 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.337013+0000", 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014299860000000001 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.343941+0000", 2024-03-26T03:17:59.615 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069278960000000002 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.347470+0000", 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0035294720000000001 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.803783+0000", 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.45631306399999999 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.803853+0000", 2024-03-26T03:17:59.616 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9276999999999994e-05 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.804004+0000", 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015118099999999999 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137296 10.8 10:12c9bc13:::benchmark_data_smithi066_148944_object12649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:38.810787+0000", 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 200.667590437, 2024-03-26T03:17:59.617 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.41837110700000002, 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137296 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810787+0000", 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.618 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810787+0000", 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810789+0000", 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7260000000000001e-06 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810796+0000", 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6390000000000003e-06 2024-03-26T03:17:59.619 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810808+0000", 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.259e-05 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810816+0000", 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7870000000000002e-06 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810861+0000", 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5003000000000003e-05 2024-03-26T03:17:59.620 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.811020+0000", 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015900199999999999 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.811155+0000", 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013474900000000001 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.813175+0000", 2024-03-26T03:17:59.621 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020204369999999999 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.813385+0000", 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020939400000000001 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.814390+0000", 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00100573 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:39.228979+0000", 2024-03-26T03:17:59.622 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.41458832400000001 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:39.229026+0000", 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7212000000000001e-05 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:39.229158+0000", 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000132514 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.623 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:138292 10.5 10:a14f9faa:::benchmark_data_smithi066_148944_object13645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:40.164725+0000", 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 199.31365232499999, 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.47151003000000002, 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138292 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.624 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164725+0000", 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164725+0000", 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164726+0000", 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1480000000000001e-06 2024-03-26T03:17:59.625 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164731+0000", 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6210000000000003e-06 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164739+0000", 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0409999999999998e-06 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164747+0000", 2024-03-26T03:17:59.626 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9889999999999992e-06 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.166114+0000", 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001366985 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.166122+0000", 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7570000000000004e-06 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.187915+0000", 2024-03-26T03:17:59.627 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.021792709 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.188075+0000", 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000160008 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.188239+0000", 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016421 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198753+0000", 2024-03-26T03:17:59.628 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010514122000000001 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198809+0000", 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6205999999999997e-05 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198847+0000", 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7579000000000001e-05 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.629 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198856+0000", 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3530000000000001e-06 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.633328+0000", 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.434472203 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.633393+0000", 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4839999999999996e-05 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.630 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.636235+0000", 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028422590000000002 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:138344 10.5 10:a8c08bd4:::benchmark_data_smithi066_148944_object13697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:40.234748+0000", 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 199.243630053, 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.40442437799999997, 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:17:59.631 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138344 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234748+0000", 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234748+0000", 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:17:59.632 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234750+0000", 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2029999999999999e-06 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234757+0000", 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.742e-06 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234774+0000", 2024-03-26T03:17:59.633 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7654000000000001e-05 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234783+0000", 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1200000000000008e-06 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234845+0000", 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1529e-05 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:17:59.634 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.235008+0000", 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016331399999999999 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.235176+0000", 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016819599999999999 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.238849+0000", 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036731490000000001 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:17:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.239003+0000", 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000153931 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.239070+0000", 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6387e-05 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.239322+0000", 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025251000000000001 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.636 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.636301+0000", 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.39697872000000001 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.636371+0000", 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9941e-05 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.639172+0000", 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002800982 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.637 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.638 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:17:59.638 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:17:59.638 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:17:59.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:17:59.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:17:59.792 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:17:59.841 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:17:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:17:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[29666]: pgmap v902: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:17:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[29666]: osdmap e430: 8 total, 8 up, 6 in 2024-03-26T03:17:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:17:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:17:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[37920]: pgmap v902: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:17:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[37920]: osdmap e430: 8 total, 8 up, 6 in 2024-03-26T03:17:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:17:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:17:59 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:17:59.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:17:59.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T03:17:59.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:17:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:17:59.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:17:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:59 smithi192 ceph-mon[33110]: pgmap v902: 178 pgs: 178 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-26T03:17:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:59 smithi192 ceph-mon[33110]: osdmap e430: 8 total, 8 up, 6 in 2024-03-26T03:17:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:59 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:17:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:17:59 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:18:00.076 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.129 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:00.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:00.230 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:00.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:00.306 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:18:00.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:00.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:00.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:18:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:18:00.516 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:00.525 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 53 16 86837 86821 102.366 76.5625 0.00805828 0.00976361 2024-03-26T03:18:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.562 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:00.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:00.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:00.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:00.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:00.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:00 smithi066 ceph-mon[29666]: osdmap e431: 8 total, 8 up, 6 in 2024-03-26T03:18:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:00 smithi066 ceph-mon[29666]: pgmap v905: 178 pgs: 1 active+remapped+backfilling, 2 peering, 175 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.5 KiB/s rd, 43 MiB/s wr, 683 op/s; 5069/255075 objects misplaced (1.987%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T03:18:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:00 smithi066 ceph-mon[37920]: osdmap e431: 8 total, 8 up, 6 in 2024-03-26T03:18:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:00 smithi066 ceph-mon[37920]: pgmap v905: 178 pgs: 1 active+remapped+backfilling, 2 peering, 175 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.5 KiB/s rd, 43 MiB/s wr, 683 op/s; 5069/255075 objects misplaced (1.987%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T03:18:00.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:00.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:00.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:00 smithi192 ceph-mon[33110]: osdmap e431: 8 total, 8 up, 6 in 2024-03-26T03:18:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:00 smithi192 ceph-mon[33110]: pgmap v905: 178 pgs: 1 active+remapped+backfilling, 2 peering, 175 active+clean; 5.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.5 KiB/s rd, 43 MiB/s wr, 683 op/s; 5069/255075 objects misplaced (1.987%); 4.2 MiB/s, 67 objects/s recovering 2024-03-26T03:18:01.058 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:01.102 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:01.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:01.131 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:18:01.204 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:01.256 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:01.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:01.357 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:01.411 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:01.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:01.513 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:01.525 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 54 16 87974 87958 101.786 71.0625 0.0335042 0.00981888 2024-03-26T03:18:01.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:18:01.589 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:01.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:01.640 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:01.666 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:01.691 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:01.763 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:01.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:01.790 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:01 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:18:01.791 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:01 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:18:01.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:01 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:18:01.791 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:01 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:18:01.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:01.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:01.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:01.954 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:01 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:18:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:01 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:18:02.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:02.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:02.148 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:02.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:02.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:02.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:02.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:02.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:02.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 55 16 89289 89273 101.429 82.1875 0.00460406 0.00985399 2024-03-26T03:18:02.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.624 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:02.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:02.725 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:02 smithi192 ceph-mon[33110]: osdmap e432: 8 total, 8 up, 6 in 2024-03-26T03:18:02.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:02 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:18:02.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:02 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:18:02.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:02 smithi192 ceph-mon[33110]: pgmap v907: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 173 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 2.3 KiB/s rd, 50 MiB/s wr, 807 op/s; 15186/258639 objects misplaced (5.872%); 10 MiB/s, 164 objects/s recovering 2024-03-26T03:18:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.776 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:02.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:02.877 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:02.912 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:02.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:03.014 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:03.060 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:03.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[29666]: osdmap e432: 8 total, 8 up, 6 in 2024-03-26T03:18:03.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:18:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:18:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[29666]: pgmap v907: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 173 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 2.3 KiB/s rd, 50 MiB/s wr, 807 op/s; 15186/258639 objects misplaced (5.872%); 10 MiB/s, 164 objects/s recovering 2024-03-26T03:18:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[37920]: osdmap e432: 8 total, 8 up, 6 in 2024-03-26T03:18:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:18:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:18:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:02 smithi066 ceph-mon[37920]: pgmap v907: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 173 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 2.3 KiB/s rd, 50 MiB/s wr, 807 op/s; 15186/258639 objects misplaced (5.872%); 10 MiB/s, 164 objects/s recovering 2024-03-26T03:18:03.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:03.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:03.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:03.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:03.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:03.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:03.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 56 16 90561 90545 101.038 79.5 0.00676917 0.00989227 2024-03-26T03:18:03.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:03.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:03.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:03.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:03.777 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:03.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-mon[29666]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:18:03.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:18:03.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:18:03.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-mon[37920]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:18:03.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:18:03.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:18:03.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:03.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:03.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:03.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:03.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:03 smithi192 ceph-mon[33110]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:18:03.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:03 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:18:03.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:03 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:18:04.008 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:04.053 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:18:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:18:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:18:04.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:04.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:04.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:04.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:04.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:04.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:04.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 57 16 91811 91795 100.635 78.125 0.0172818 0.00993136 2024-03-26T03:18:04.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:04.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:04.686 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:04.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:04.837 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:04.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:04.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:04.994 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:05.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:05.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:05.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:05.282 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:05.328 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:05.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:05.476 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:05.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 58 15 93105 93090 100.296 80.9375 0.0102074 0.00996541 2024-03-26T03:18:05.553 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:05 smithi192 ceph-mon[33110]: pgmap v908: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 173 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.8 KiB/s rd, 41 MiB/s wr, 654 op/s; 15186/258639 objects misplaced (5.872%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T03:18:05.554 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:05 smithi066 ceph-mon[29666]: pgmap v908: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 173 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.8 KiB/s rd, 41 MiB/s wr, 654 op/s; 15186/258639 objects misplaced (5.872%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T03:18:05.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:05 smithi066 ceph-mon[37920]: pgmap v908: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 peering, 173 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.8 KiB/s rd, 41 MiB/s wr, 654 op/s; 15186/258639 objects misplaced (5.872%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T03:18:05.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.629 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:05.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:05.657 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:05.657 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(11.b 432/428 [PushOp(11:d36a7485:::benchmark_data_smithi066_175179_object61124:head, version: 421'3859, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:d36a7485:::benchmark_data_smithi066_175179_object61124:head@421'3859, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:05.655267+0000", 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.000135671, 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019082999999999999, 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655267+0000", 2024-03-26T03:18:05.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655267+0000", 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655268+0000", 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.344e-06 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655332+0000", 2024-03-26T03:18:05.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3728999999999999e-05 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655352+0000", 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0367e-05 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655369+0000", 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6237999999999999e-05 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:05.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "pg_backfill(progress 11.b e 432/432 lb 11:d3693fa9:::benchmark_data_smithi066_175179_object31861:head)", 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:05.655399+0000", 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 3.304e-06, 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012484, 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655399+0000", 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655399+0000", 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655400+0000", 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2099999999999996e-07 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655407+0000", 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3830000000000003e-06 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655418+0000", 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0249e-05 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:05.655424+0000", 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8360000000000003e-06 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-03-26T03:18:05.663 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:05.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:05.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:05.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:05.909 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:05.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:06.011 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:06.056 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:06.158 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:06.194 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:06.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:06.348 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:06.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:06 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:18:06.397 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:06 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:18:06.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:18:06.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.479 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:06.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:06.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:06 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:18:06.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:06 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:18:06.493 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:06 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:18:06.493 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:06 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:18:06.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 59 16 94253 94237 99.8107 71.6875 0.0054682 0.010008 2024-03-26T03:18:06.580 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:06.620 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:06.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:06.749 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:06.768 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:06.872 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:06.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:06.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:07.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:07.063 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:07.165 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:07.212 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:07.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:07.276 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T03:18:07.297 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:07 smithi066 ceph-mon[29666]: pgmap v909: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 2.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 14976/277302 objects misplaced (5.401%); 16 MiB/s, 263 objects/s recovering 2024-03-26T03:18:07.297 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1923832735' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:18:07.297 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:07 smithi066 ceph-mon[29666]: from='client.35894 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:18:07.297 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:07 smithi066 ceph-mon[37920]: pgmap v909: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 2.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 14976/277302 objects misplaced (5.401%); 16 MiB/s, 263 objects/s recovering 2024-03-26T03:18:07.297 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1923832735' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:18:07.297 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:07 smithi066 ceph-mon[37920]: from='client.35894 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:18:07.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:07.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:07.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:07 smithi192 ceph-mon[33110]: pgmap v909: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 2.0 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 14976/277302 objects misplaced (5.401%); 16 MiB/s, 263 objects/s recovering 2024-03-26T03:18:07.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1923832735' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:18:07.439 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:07 smithi192 ceph-mon[33110]: from='client.35894 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:18:07.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:07.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:07.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:18:07.526785+0000 min lat: 0.00143665 max lat: 2.14475 avg lat: 0.0100463 2024-03-26T03:18:07.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:18:07.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 60 16 95540 95524 99.4877 80.4375 0.0142132 0.0100463 2024-03-26T03:18:07.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:07.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:07.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:07.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:07.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:07.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:07.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:07.988 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:08.034 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:08.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.171 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:08.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:08.272 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:08.319 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:08.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:08 smithi192 ceph-mon[33110]: from='client.35894 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:18:08.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:08 smithi192 ceph-mon[33110]: osdmap e433: 8 total, 8 up, 6 in 2024-03-26T03:18:08.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:08 smithi192 ceph-mon[33110]: 6.a deep-scrub starts 2024-03-26T03:18:08.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:08 smithi192 ceph-mon[33110]: 6.a deep-scrub ok 2024-03-26T03:18:08.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:08.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:08.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 61 16 96505 96489 98.8454 60.3125 0.0219278 0.0101118 2024-03-26T03:18:08.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[29666]: from='client.35894 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[29666]: osdmap e433: 8 total, 8 up, 6 in 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[29666]: 6.a deep-scrub starts 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[29666]: 6.a deep-scrub ok 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[37920]: from='client.35894 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[37920]: osdmap e433: 8 total, 8 up, 6 in 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[37920]: 6.a deep-scrub starts 2024-03-26T03:18:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:08 smithi066 ceph-mon[37920]: 6.a deep-scrub ok 2024-03-26T03:18:08.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:08.741 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:08.783 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:08.886 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:08.923 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:08.923 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:18:08.923 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:18:08.923 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:18:08.923 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:08.926 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:08.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:09.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:09.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:09.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:09.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:09.303 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.335 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:09.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:09.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:09.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:09.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 62 16 97577 97561 98.3318 67 0.0102197 0.0101646 2024-03-26T03:18:09.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:09.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:09 smithi066 ceph-mon[29666]: pgmap v911: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 927 op/s; 14976/277302 objects misplaced (5.401%); 14 MiB/s, 224 objects/s recovering 2024-03-26T03:18:09.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:09 smithi066 ceph-mon[37920]: pgmap v911: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 927 op/s; 14976/277302 objects misplaced (5.401%); 14 MiB/s, 224 objects/s recovering 2024-03-26T03:18:09.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:09.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:09.735 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:09 smithi192 ceph-mon[33110]: pgmap v911: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 927 op/s; 14976/277302 objects misplaced (5.401%); 14 MiB/s, 224 objects/s recovering 2024-03-26T03:18:09.828 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:09.871 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:09.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:09.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:10.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:10.106 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:10.152 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:10.254 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:10.290 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:10.391 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:10.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:10.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:18:10 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:18:10.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 63 16 98814 98798 97.998 77.3125 0.0115147 0.0101991 2024-03-26T03:18:10.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:10.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:10.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:10.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:10.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:10.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:10.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:18:10.942 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:10.988 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:10.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:11.089 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.131 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:11.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:11.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:11.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:11 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:18:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:11 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:18:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:11 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:11.238 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:11 smithi192 ceph-mon[33110]: pgmap v912: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 580 B/s rd, 72 MiB/s wr, 1.16k op/s; 14594/289230 objects misplaced (5.046%); 14 MiB/s, 218 objects/s recovering 2024-03-26T03:18:11.249 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:11.272 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:11.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[29666]: pgmap v912: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 580 B/s rd, 72 MiB/s wr, 1.16k op/s; 14594/289230 objects misplaced (5.046%); 14 MiB/s, 218 objects/s recovering 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:11.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:11 smithi066 ceph-mon[37920]: pgmap v912: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 580 B/s rd, 72 MiB/s wr, 1.16k op/s; 14594/289230 objects misplaced (5.046%); 14 MiB/s, 218 objects/s recovering 2024-03-26T03:18:11.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:11.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:11.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:11.526 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 64 16 100001 99985 97.6259 74.1875 0.0329929 0.0102362 2024-03-26T03:18:11.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:11.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:11.685 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:11.786 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:11.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:11.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:11.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:11.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:12.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:12.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:12.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:12.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:12.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:12.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:12.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:12.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:12.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 65 16 101315 101299 97.3874 82.125 0.0140545 0.0102636 2024-03-26T03:18:12.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:12.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:12.782 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:12.828 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:12.931 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:12.977 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:12.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:13.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:13.120 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:13.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:13.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137145 10.1 10:88db8228:::benchmark_data_smithi066_148944_object12498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.237799+0000", 2024-03-26T03:18:13.243 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 216.99952498299999, 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1472831160000001, 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137145 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237799+0000", 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237799+0000", 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237801+0000", 2024-03-26T03:18:13.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.804e-06 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237808+0000", 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7880000000000001e-06 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237819+0000", 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1447000000000001e-05 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237828+0000", 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.422e-06 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.246680+0000", 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0088519389999999996 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.246688+0000", 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8470000000000002e-06 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.348802+0000", 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.102113731 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.348940+0000", 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013755199999999999 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,2,7", 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.349324+0000", 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038390800000000002 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353114+0000", 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037899159999999999 2024-03-26T03:18:13.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353135+0000", 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1654999999999998e-05 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353587+0000", 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000452163 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353676+0000", 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8873000000000001e-05 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.384809+0000", 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031132667999999999 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.384928+0000", 2024-03-26T03:18:13.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001192 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.385082+0000", 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000154203 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137152 10.c 10:39382015:::benchmark_data_smithi066_148944_object12505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:18:13.252 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.235713+0000", 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 216.00161096599999, 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1013530659999999, 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137152 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235713+0000", 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235713+0000", 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235715+0000", 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.663e-06 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235720+0000", 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.775e-06 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235730+0000", 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0030000000000001e-05 2024-03-26T03:18:13.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235740+0000", 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0193999999999999e-05 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235903+0000", 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000163346 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235917+0000", 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3515e-05 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.316994+0000", 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.081077627 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for missing object", 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317110+0000", 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011518000000000001 2024-03-26T03:18:13.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.336923+0000", 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019813408000000001 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337066+0000", 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014332799999999999 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.258 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137181 10.c 10:30f0c64e:::benchmark_data_smithi066_148944_object12534:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.235757+0000", 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 216.001566895, 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.251399749, 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.259 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137181 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235757+0000", 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235757+0000", 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235758+0000", 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2500000000000005e-07 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235761+0000", 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7429999999999998e-06 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235767+0000", 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3230000000000003e-06 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235775+0000", 2024-03-26T03:18:13.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5569999999999998e-06 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235956+0000", 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018114699999999999 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235980+0000", 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4681999999999998e-05 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317365+0000", 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0813844930000001 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317499+0000", 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000133934 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-26T03:18:13.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317899+0000", 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040043700000000002 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.328624+0000", 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010724559 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.328669+0000", 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5330999999999997e-05 2024-03-26T03:18:13.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334991+0000", 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063218470000000002 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339323+0000", 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043321030000000003 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.486979+0000", 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14765573800000001 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487052+0000", 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3473000000000006e-05 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487157+0000", 2024-03-26T03:18:13.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104757 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137196 10.c 10:35e1a6a7:::benchmark_data_smithi066_148944_object12549:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.235782+0000", 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 216.00154217100001, 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.252017725, 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.268 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137196 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235782+0000", 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235782+0000", 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235783+0000", 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9299999999999998e-07 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235785+0000", 2024-03-26T03:18:13.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.633e-06 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235791+0000", 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235797+0000", 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8470000000000001e-06 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.236010+0000", 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000213396 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.236020+0000", 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8079999999999999e-06 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326110+0000", 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.090089571 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326258+0000", 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014866799999999999 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326415+0000", 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015718700000000001 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.328712+0000", 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022967980000000001 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.333008+0000", 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042961630000000004 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334998+0000", 2024-03-26T03:18:13.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019894999999999999 2024-03-26T03:18:13.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339362+0000", 2024-03-26T03:18:13.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043638030000000003 2024-03-26T03:18:13.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487563+0000", 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14820169799999999 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487620+0000", 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6239000000000001e-05 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487800+0000", 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018014699999999999 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.277 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158897 10.6 10:65a0f9c2:::benchmark_data_smithi066_148944_object34250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.525934+0000", 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.711390585, 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.018215927, 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.278 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158897 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525934+0000", 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525934+0000", 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525935+0000", 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1170000000000001e-06 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525939+0000", 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4909999999999999e-06 2024-03-26T03:18:13.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525947+0000", 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0709999999999995e-06 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525953+0000", 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.429e-06 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528310+0000", 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023570219999999998 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528318+0000", 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5359999999999992e-06 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.532896+0000", 2024-03-26T03:18:13.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0045775850000001 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.533016+0000", 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120381 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.533101+0000", 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5241000000000006e-05 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537203+0000", 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041013949999999999 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540057+0000", 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002853642 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544078+0000", 2024-03-26T03:18:13.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040211420000000001 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544112+0000", 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3914000000000001e-05 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544150+0000", 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7960999999999998e-05 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158898 10.6 10:6ab00462:::benchmark_data_smithi066_148944_object34251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.525959+0000", 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.71136484799999, 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0183360560000001, 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158898 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525959+0000", 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525959+0000", 2024-03-26T03:18:13.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525960+0000", 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3599999999999999e-07 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525970+0000", 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0013000000000001e-05 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525975+0000", 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4229999999999996e-06 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525980+0000", 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2109999999999997e-06 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528341+0000", 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002360441 2024-03-26T03:18:13.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528347+0000", 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6760000000000004e-06 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.535316+0000", 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0069691119999999 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.535414+0000", 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8239000000000006e-05 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.535495+0000", 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1291000000000005e-05 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540086+0000", 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045907839999999997 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541353+0000", 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012669770000000001 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544219+0000", 2024-03-26T03:18:13.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002865912 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544250+0000", 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1639e-05 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544295+0000", 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4901999999999998e-05 2024-03-26T03:18:13.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158904 10.6 10:686bb8c5:::benchmark_data_smithi066_148944_object34257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.532299+0000", 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.705025624, 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0121124100000001, 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.294 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158904 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532299+0000", 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532299+0000", 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532300+0000", 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.705e-06 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532306+0000", 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8189999999999997e-06 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532315+0000", 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9619999999999999e-06 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532323+0000", 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6629999999999993e-06 2024-03-26T03:18:13.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.533108+0000", 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00078506300000000002 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.533116+0000", 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0120000000000005e-06 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537242+0000", 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0041266049999999 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537334+0000", 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1242000000000005e-05 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537403+0000", 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9321999999999998e-05 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540173+0000", 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002770242 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541359+0000", 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011858389999999999 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544354+0000", 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029949830000000001 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544386+0000", 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2308000000000003e-05 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544411+0000", 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4644999999999999e-05 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.302 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158914 10.6 10:63d66798:::benchmark_data_smithi066_148944_object34267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.538143+0000", 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.699180963, 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0063660169999999, 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158914 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.303 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538143+0000", 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538143+0000", 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538144+0000", 2024-03-26T03:18:13.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538151+0000", 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8299999999999998e-06 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538159+0000", 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6079999999999998e-06 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538165+0000", 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6309999999999997e-06 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.539129+0000", 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096368300000000001 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.539137+0000", 2024-03-26T03:18:13.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0129999999999993e-06 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.538712+0000", 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99957449899999995 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.538824+0000", 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112478 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.538942+0000", 2024-03-26T03:18:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:13.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:13.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118229 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.543952+0000", 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050095180000000001 2024-03-26T03:18:13.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.543994+0000", 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2228000000000003e-05 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544446+0000", 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045226100000000003 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544475+0000", 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8370000000000001e-05 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544509+0000", 2024-03-26T03:18:13.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4514000000000002e-05 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158922 10.6 10:69344914:::benchmark_data_smithi066_148944_object34275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.546660+0000", 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.69066416199999, 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.998002905, 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.313 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158922 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546660+0000", 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546660+0000", 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546662+0000", 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.545e-06 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546667+0000", 2024-03-26T03:18:13.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2669999999999997e-06 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546677+0000", 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0567999999999999e-05 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546685+0000", 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9170000000000006e-06 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.547356+0000", 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067084800000000004 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.547364+0000", 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1990000000000006e-06 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540199+0000", 2024-03-26T03:18:13.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99283470699999998 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540267+0000", 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7447999999999995e-05 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540332+0000", 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5779999999999997e-05 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544028+0000", 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036954750000000001 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544036+0000", 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8019999999999992e-06 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544600+0000", 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056420099999999996 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544633+0000", 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2948999999999999e-05 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544663+0000", 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0199e-05 2024-03-26T03:18:13.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158945 10.6 10:6a979a68:::benchmark_data_smithi066_148944_object34298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.566506+0000", 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.67081789400001, 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97834264000000004, 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.321 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158945 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566506+0000", 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566506+0000", 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566508+0000", 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.635e-06 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566513+0000", 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6829999999999999e-06 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566524+0000", 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0987e-05 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566529+0000", 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5589999999999996e-06 2024-03-26T03:18:13.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566576+0000", 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6502999999999998e-05 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566582+0000", 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3330000000000002e-06 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541387+0000", 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.974805221 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541452+0000", 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4992000000000004e-05 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541573+0000", 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120808 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544041+0000", 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002468231 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544566+0000", 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052513699999999996 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544750+0000", 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018393199999999999 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544815+0000", 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4380000000000004e-05 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544849+0000", 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4239000000000002e-05 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158946 10.6 10:63718db7:::benchmark_data_smithi066_148944_object34299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.566744+0000", 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.670579839, 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.979667344, 2024-03-26T03:18:13.329 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158946 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566744+0000", 2024-03-26T03:18:13.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566744+0000", 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566745+0000", 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6599999999999996e-07 2024-03-26T03:18:13.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566748+0000", 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0089999999999999e-06 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566753+0000", 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5730000000000001e-06 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566757+0000", 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9280000000000003e-06 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566786+0000", 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9879e-05 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566791+0000", 2024-03-26T03:18:13.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9520000000000002e-06 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.542682+0000", 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97589034500000005 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.542781+0000", 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9197000000000006e-05 2024-03-26T03:18:13.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.542869+0000", 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8534000000000003e-05 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544527+0000", 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016575520000000001 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544701+0000", 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174272 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546312+0000", 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016107610000000001 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546371+0000", 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8791e-05 2024-03-26T03:18:13.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546412+0000", 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0884999999999999e-05 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158956 10.6 10:6f00b1fc:::benchmark_data_smithi066_148944_object34309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.573836+0000", 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.66348806299999, 2024-03-26T03:18:13.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.977362116, 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158956 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573836+0000", 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573836+0000", 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573838+0000", 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.719e-06 2024-03-26T03:18:13.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573842+0000", 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2799999999999997e-06 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573850+0000", 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0239999999999997e-06 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573857+0000", 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8349999999999998e-06 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573910+0000", 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2846000000000002e-05 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573918+0000", 2024-03-26T03:18:13.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4120000000000001e-06 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544882+0000", 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97096349500000001 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544970+0000", 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8603999999999994e-05 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.545040+0000", 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9832999999999994e-05 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550896+0000", 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058562409999999999 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550935+0000", 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8210000000000002e-05 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.551073+0000", 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013858800000000001 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.551156+0000", 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2978000000000004e-05 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.551198+0000", 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2051000000000001e-05 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158959 10.6 10:6f864bb5:::benchmark_data_smithi066_148944_object34312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.577141+0000", 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.66018271900001, 2024-03-26T03:18:13.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97805649400000005, 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158959 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577141+0000", 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577141+0000", 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577143+0000", 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1930000000000001e-06 2024-03-26T03:18:13.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577147+0000", 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4830000000000001e-06 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577155+0000", 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3389999999999993e-06 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577163+0000", 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1210000000000001e-06 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577233+0000", 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0706999999999999e-05 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577242+0000", 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4109999999999996e-06 2024-03-26T03:18:13.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546459+0000", 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96921719200000001 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546574+0000", 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114706 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:18:13.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546686+0000", 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112698 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550947+0000", 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042605810000000003 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550984+0000", 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6906999999999997e-05 2024-03-26T03:18:13.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.555122+0000", 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041379909999999997 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.555154+0000", 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1736999999999998e-05 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.555198+0000", 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4428000000000002e-05 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:199030 10.4 10:291ad30d:::benchmark_data_smithi066_148944_object74383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:44.926131+0000", 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 148.31119342700001, 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97667217900000003, 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.353 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199030 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926131+0000", 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926131+0000", 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926132+0000", 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8400000000000003e-07 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926135+0000", 2024-03-26T03:18:13.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5470000000000001e-06 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926145+0000", 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6199999999999994e-06 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926151+0000", 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.981e-06 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926486+0000", 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000334965 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926494+0000", 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5869999999999993e-06 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.895660+0000", 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96916596799999999 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.895793+0000", 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013325199999999999 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.895932+0000", 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013806499999999999 2024-03-26T03:18:13.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900704+0000", 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047728720000000001 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902598+0000", 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018933330000000001 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902726+0000", 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012818199999999999 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902762+0000", 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6371999999999997e-05 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902803+0000", 2024-03-26T03:18:13.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0651000000000001e-05 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:199032 10.4 10:255721a4:::benchmark_data_smithi066_148944_object74385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:44.930307+0000", 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 148.30701728, 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97661484300000001, 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.361 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199032 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930307+0000", 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930307+0000", 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930308+0000", 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3200000000000004e-07 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930311+0000", 2024-03-26T03:18:13.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4879999999999999e-06 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930318+0000", 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8600000000000004e-06 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930324+0000", 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6840000000000001e-06 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930409+0000", 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4892e-05 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930417+0000", 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8540000000000003e-06 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.898572+0000", 2024-03-26T03:18:13.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96815444900000003 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.898646+0000", 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4466000000000002e-05 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.898721+0000", 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4299000000000001e-05 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.904286+0000", 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00556581 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906712+0000", 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024256460000000001 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906821+0000", 2024-03-26T03:18:13.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108491 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906868+0000", 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589000000000001e-05 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906922+0000", 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3582999999999998e-05 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.368 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:199042 10.4 10:2c3d90fc:::benchmark_data_smithi066_148944_object74395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:44.940858+0000", 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 148.29646609100001, 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96623120299999998, 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:13.369 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199042 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940858+0000", 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940858+0000", 2024-03-26T03:18:13.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940859+0000", 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3799999999999999e-06 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940865+0000", 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0810000000000001e-06 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940874+0000", 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7820000000000005e-06 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940882+0000", 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6059999999999997e-06 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940941+0000", 2024-03-26T03:18:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9290999999999999e-05 2024-03-26T03:18:13.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940949+0000", 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9799999999999998e-06 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900739+0000", 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95978980899999999 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900823+0000", 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3926999999999998e-05 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900898+0000", 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5498000000000001e-05 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.904293+0000", 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033949520000000001 2024-03-26T03:18:13.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906762+0000", 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024681970000000001 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906989+0000", 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022715999999999999 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.907045+0000", 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6215999999999998e-05 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.907089+0000", 2024-03-26T03:18:13.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4323999999999998e-05 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:83024 11.e 11.d2d4311e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.550240+0000", 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 17.687084122000002, 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0111298360000001, 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.378 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83024 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550240+0000", 2024-03-26T03:18:13.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550240+0000", 2024-03-26T03:18:13.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550241+0000", 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.384e-06 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550288+0000", 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616999999999998e-05 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550299+0000", 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0472000000000001e-05 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550306+0000", 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3560000000000004e-06 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.561812+0000", 2024-03-26T03:18:13.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011506003000000001 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.561820+0000", 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0730000000000005e-06 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561358+0000", 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99953816900000003 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561370+0000", 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1762e-05 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.34346.0:4038 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94613743831168 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.551335+0000", 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 17.685989659000001, 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.023797048, 2024-03-26T03:18:13.385 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.34346", 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/2004587607", 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4038 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551335+0000", 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551335+0000", 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551336+0000", 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2619999999999999e-06 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551340+0000", 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9380000000000002e-06 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551347+0000", 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.661e-06 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551353+0000", 2024-03-26T03:18:13.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8479999999999997e-06 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.557762+0000", 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0064087220000000004 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.557770+0000", 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9540000000000004e-06 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.566358+0000", 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.00858767 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for missing object", 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.566514+0000", 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000156711 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.570613+0000", 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040989889999999999 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for rw locks", 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.570718+0000", 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104448 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.571395+0000", 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067711099999999999 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.571649+0000", 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025428899999999999 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.575132+0000", 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034824449999999998 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:83048 11.e 11.35baa8e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.564394+0000", 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 17.672930334, 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99701060100000005, 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:18:13.393 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83048 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564394+0000", 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564394+0000", 2024-03-26T03:18:13.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564396+0000", 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564438+0000", 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1622000000000002e-05 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564449+0000", 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1365e-05 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564457+0000", 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564513+0000", 2024-03-26T03:18:13.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5358000000000001e-05 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564521+0000", 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0360000000000006e-06 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561397+0000", 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99687634400000003 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561404+0000", 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3119999999999996e-06 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:83051 11.e 11.d761e45e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.569213+0000", 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 17.668111539999998, 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99222091400000001, 2024-03-26T03:18:13.398 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83051 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569213+0000", 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569213+0000", 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569214+0000", 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.813e-06 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:13.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569274+0000", 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9602000000000003e-05 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569284+0000", 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0396e-05 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569290+0000", 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569333+0000", 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2273e-05 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569338+0000", 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4e-06 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:13.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561426+0000", 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99208782399999995 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561434+0000", 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5660000000000001e-06 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:13.403 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:13.410 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:13.455 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:13.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:13.514 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:18:13.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 66 16 102459 102443 96.9951 71.5 0.0081299 0.0103047 2024-03-26T03:18:13.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:13.593 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:13.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:13.696 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:13.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:13.739 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:13.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:13.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:13 smithi192 ceph-mon[33110]: pgmap v913: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s; 14675/294711 objects misplaced (4.979%); 15 MiB/s, 247 objects/s recovering 2024-03-26T03:18:13.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:13 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1967339609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:18:13.740 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:13 smithi192 ceph-mon[33110]: from='client.35900 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:18:13.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-mon[29666]: pgmap v913: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s; 14675/294711 objects misplaced (4.979%); 15 MiB/s, 247 objects/s recovering 2024-03-26T03:18:13.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1967339609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:18:13.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-mon[29666]: from='client.35900 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:18:13.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-mon[37920]: pgmap v913: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s; 14675/294711 objects misplaced (4.979%); 15 MiB/s, 247 objects/s recovering 2024-03-26T03:18:13.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1967339609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:18:13.833 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-mon[37920]: from='client.35900 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:18:13.840 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:13.884 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:13.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:13.986 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.032 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:14.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:14.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T03:18:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:18:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:18:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:18:14.134 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.207 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:14.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:14.309 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:14.366 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:14.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:18:14.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:14.518 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:14.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 67 16 103744 103728 96.746 80.3125 0.00672122 0.0103316 2024-03-26T03:18:14.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:14.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:14 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:14 smithi192 ceph-mon[33110]: from='client.35900 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:18:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:14 smithi192 ceph-mon[33110]: osdmap e434: 8 total, 8 up, 6 in 2024-03-26T03:18:14.774 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:14.793 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:14.821 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:14.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:14 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:14 smithi066 ceph-mon[29666]: from='client.35900 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:18:14.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:14 smithi066 ceph-mon[29666]: osdmap e434: 8 total, 8 up, 6 in 2024-03-26T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:14 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:14 smithi066 ceph-mon[37920]: from='client.35900 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:18:14.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:14 smithi066 ceph-mon[37920]: osdmap e434: 8 total, 8 up, 6 in 2024-03-26T03:18:14.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:14.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:14.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:15.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:15.132 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:15.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:15.234 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:15.281 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:15.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:15.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:15.416 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:15.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:15.518 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:15.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 68 15 104956 104941 96.438 75.8125 0.0116514 0.0103648 2024-03-26T03:18:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:15.566 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:15.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:15.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:15.716 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:15.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:15 smithi192 ceph-mon[33110]: pgmap v915: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s; 14675/294711 objects misplaced (4.979%); 9.4 MiB/s, 150 objects/s recovering 2024-03-26T03:18:15.817 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:15.840 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:15 smithi066 ceph-mon[37920]: pgmap v915: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s; 14675/294711 objects misplaced (4.979%); 9.4 MiB/s, 150 objects/s recovering 2024-03-26T03:18:15.842 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:15 smithi066 ceph-mon[29666]: pgmap v915: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.0 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s; 14675/294711 objects misplaced (4.979%); 9.4 MiB/s, 150 objects/s recovering 2024-03-26T03:18:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:15.866 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:15.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:15.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:16.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:16.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:16.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:16.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:16.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:16.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:16.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:16.519 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:16.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 69 15 106253 106238 96.2151 81.0625 0.00822553 0.0103885 2024-03-26T03:18:16.565 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:16.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:16 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:18:16.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:16 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:18:16.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:16 smithi192 ceph-mon[33110]: osdmap e435: 8 total, 8 up, 6 in 2024-03-26T03:18:16.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:16.714 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:16.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:16.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:16 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:18:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:16 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:18:16.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:16 smithi066 ceph-mon[29666]: osdmap e435: 8 total, 8 up, 6 in 2024-03-26T03:18:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:16 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:18:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:16 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:18:16.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:16 smithi066 ceph-mon[37920]: osdmap e435: 8 total, 8 up, 6 in 2024-03-26T03:18:16.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:16.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:16.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15978.0:106881 11.c e435/428 11:368e895d:::benchmark_data_smithi066_175179_object106880:head v 435'6620, mlcod=435'6618)", 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:17.015821+0000", 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0045630089999999998, 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045871549999999999, 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:17.022 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.015821+0000", 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.015821+0000", 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.015823+0000", 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9640000000000002e-06 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.015862+0000", 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9400000000000002e-05 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.015870+0000", 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4300000000000006e-06 2024-03-26T03:18:17.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.015878+0000", 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5089999999999996e-06 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016191+0000", 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031311099999999998 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016216+0000", 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5425000000000001e-05 2024-03-26T03:18:17.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15978.0:106882 11.c e435/428 11:3028f440:::benchmark_data_smithi066_175179_object106881:head v 435'6621, mlcod=435'6618)", 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:17.016554+0000", 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0038299919999999999, 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039388349999999999, 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:18:17.026 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016554+0000", 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016554+0000", 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016555+0000", 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1489999999999999e-06 2024-03-26T03:18:17.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016599+0000", 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4267999999999997e-05 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016613+0000", 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3767999999999999e-05 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:17.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.016623+0000", 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0599e-05 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.018478+0000", 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018543889999999999 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.018504+0000", 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6475999999999999e-05 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:17.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:106890 11.7 11:e8ceb016:::benchmark_data_smithi066_175179_object106889:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:17.019863+0000", 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00052103700000000002, 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00070921999999999999, 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:18:17.030 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 106890 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.019863+0000", 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.019863+0000", 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:17.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.019864+0000", 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7260000000000001e-06 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.019935+0000", 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0556999999999995e-05 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.019949+0000", 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.381e-05 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.019957+0000", 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4999999999999999e-06 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.020029+0000", 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.216e-05 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.020191+0000", 2024-03-26T03:18:17.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016138700000000001 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:17.020349+0000", 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015844399999999999 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 3 2024-03-26T03:18:17.034 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:17.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:17.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:17.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:17.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:17.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:17.413 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:17.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:17.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 70 16 107552 107536 95.9995 81.125 0.00818909 0.0104123 2024-03-26T03:18:17.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:17.652 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:17.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:17 smithi192 ceph-mon[33110]: pgmap v917: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s; 14344/313167 objects misplaced (4.580%); 19 MiB/s, 305 objects/s recovering 2024-03-26T03:18:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.699 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:17.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:17.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:18:17.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:17.841 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:17.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:17 smithi066 ceph-mon[29666]: pgmap v917: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s; 14344/313167 objects misplaced (4.580%); 19 MiB/s, 305 objects/s recovering 2024-03-26T03:18:17.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:17 smithi066 ceph-mon[37920]: pgmap v917: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s; 14344/313167 objects misplaced (4.580%); 19 MiB/s, 305 objects/s recovering 2024-03-26T03:18:17.943 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:17.990 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:17.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:18.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:18.115 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:18.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:18.179 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:18.229 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.257 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:18.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:18.358 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:18.407 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:18.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:18.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 71 16 108791 108775 95.7379 77.4375 0.0105122 0.0104405 2024-03-26T03:18:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:18.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:18.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:18.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:18.794 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:18.841 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:18.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:18.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:18.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:19.074 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:19.119 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:19.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:19.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:19.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:19.394 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:19.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:19.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:19.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 72 16 110046 110030 95.4976 78.4375 0.0104134 0.0104666 2024-03-26T03:18:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:19 smithi066 ceph-mon[29666]: pgmap v918: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 998 op/s; 14344/313167 objects misplaced (4.580%); 14 MiB/s, 223 objects/s recovering 2024-03-26T03:18:19.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:19 smithi066 ceph-mon[37920]: pgmap v918: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 998 op/s; 14344/313167 objects misplaced (4.580%); 14 MiB/s, 223 objects/s recovering 2024-03-26T03:18:19.628 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:19.673 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:19.728 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:19 smithi192 ceph-mon[33110]: pgmap v918: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 998 op/s; 14344/313167 objects misplaced (4.580%); 14 MiB/s, 223 objects/s recovering 2024-03-26T03:18:19.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:19.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:19.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:19.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:19.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:20.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:20.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:20.155 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:18:20.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:20.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:20.294 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:18:20 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:18:20.317 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:20.368 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:20.369 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:18:20.369 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:18:20.369 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:18:20.369 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:20.381 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:20.483 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:20.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:20.527 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 73 16 111112 111096 95.1019 66.625 0.0423899 0.0105096 2024-03-26T03:18:20.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:20.664 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:20.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:20.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:20.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:20.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:20.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:21.039 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:21.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:18:21.085 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:21.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:21.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:21.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.376 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:21.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:21.426 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:21.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:21.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:21.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 74 16 112406 112390 94.9096 80.875 0.00580977 0.010532 2024-03-26T03:18:21.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:21.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:21 smithi066 ceph-mon[37920]: pgmap v919: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s; 13936/326211 objects misplaced (4.272%); 15 MiB/s, 235 objects/s recovering 2024-03-26T03:18:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:21.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:21 smithi066 ceph-mon[29666]: pgmap v919: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s; 13936/326211 objects misplaced (4.272%); 15 MiB/s, 235 objects/s recovering 2024-03-26T03:18:21.609 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:21.638 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:21.739 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.786 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:21.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:21.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:21.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:21.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:21.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:21 smithi192 ceph-mon[33110]: pgmap v919: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.6 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.31k op/s; 13936/326211 objects misplaced (4.272%); 15 MiB/s, 235 objects/s recovering 2024-03-26T03:18:22.022 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:22.061 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:22.163 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.193 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:22.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:22.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:22.340 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:22.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:22.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:22.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 75 16 113537 113521 94.5866 70.6875 0.0100227 0.0105677 2024-03-26T03:18:22.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.613 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:22.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:22.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:22.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:22.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:22.878 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:22.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:22.980 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:23.026 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:23.128 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:23.177 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:23.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:23.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:23.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:23.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:23.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 76 16 114557 114541 94.1808 63.75 0.0115012 0.0106132 2024-03-26T03:18:23.552 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:23.602 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:23.704 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:23.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:23 smithi192 ceph-mon[33110]: pgmap v920: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 14036/332073 objects misplaced (4.227%); 17 MiB/s, 269 objects/s recovering 2024-03-26T03:18:23.734 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:23.832 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:23 smithi066 ceph-mon[29666]: pgmap v920: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 14036/332073 objects misplaced (4.227%); 17 MiB/s, 269 objects/s recovering 2024-03-26T03:18:23.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:23 smithi066 ceph-mon[37920]: pgmap v920: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s; 14036/332073 objects misplaced (4.227%); 17 MiB/s, 269 objects/s recovering 2024-03-26T03:18:23.836 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:23.879 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:23.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:23.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:24.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:24.105 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:18:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:18:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:18:24.121 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:24.154 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:24.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.291 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:24.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:24.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:24.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:24.453 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:18:24.453 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:18:24.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 77 16 115621 115605 93.8212 66.5 0.00674444 0.0106536 2024-03-26T03:18:24.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:24.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:24.615 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:24 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4254853764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:24.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:24.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:24.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:24.854 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:24.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:24 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4254853764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:24.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:24 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4254853764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:24.956 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:25.000 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:25.102 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.151 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:25.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:25.252 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:25.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:18:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.331 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:25.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:25.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:18:25.455 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:18:25.456 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:18:25.479 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:25.528 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 78 16 116964 116948 93.6944 83.9375 0.00674303 0.0106687 2024-03-26T03:18:25.561 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:25 smithi066 ceph-mon[29666]: pgmap v921: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s; 14036/332073 objects misplaced (4.227%); 15 MiB/s, 240 objects/s recovering 2024-03-26T03:18:25.561 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:25.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:25.562 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:25 smithi066 ceph-mon[37920]: pgmap v921: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s; 14036/332073 objects misplaced (4.227%); 15 MiB/s, 240 objects/s recovering 2024-03-26T03:18:25.562 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:25.562 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:25.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:25.622 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:25.639 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:25.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:25.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:25.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:25 smithi192 ceph-mon[33110]: pgmap v921: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 6.7 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s; 14036/332073 objects misplaced (4.227%); 15 MiB/s, 240 objects/s recovering 2024-03-26T03:18:25.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:25.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:25.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:25.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:25.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:25.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:26.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:26.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.176 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:26.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:26.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:26.327 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:26.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:26.476 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:26.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 79 16 118335 118319 93.592 85.6875 0.00944105 0.0106794 2024-03-26T03:18:26.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:26.599 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:26.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:26.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.765 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:26.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:26.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:26.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:26.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:27.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.044 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:27.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:27.145 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:27.200 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:27.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:27.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:27.437 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:27.486 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:27.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:2024-03-26T03:18:27.529492+0000 min lat: 0.00143665 max lat: 2.14475 avg lat: 0.0107023 2024-03-26T03:18:27.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:18:27.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 80 16 119582 119566 93.3962 77.9375 0.0109129 0.0107023 2024-03-26T03:18:27.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:27.634 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:27.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:27.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:27.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[29666]: pgmap v922: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.1 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1004 B/s rd, 73 MiB/s wr, 1.18k op/s; 13381/349086 objects misplaced (3.833%); 15 MiB/s, 242 objects/s recovering 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[29666]: osdmap e436: 8 total, 8 up, 6 in 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[37920]: pgmap v922: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.1 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1004 B/s rd, 73 MiB/s wr, 1.18k op/s; 13381/349086 objects misplaced (3.833%); 15 MiB/s, 242 objects/s recovering 2024-03-26T03:18:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:27 smithi066 ceph-mon[37920]: osdmap e436: 8 total, 8 up, 6 in 2024-03-26T03:18:27.876 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:27 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:18:27.876 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:27 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:18:27.876 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:27 smithi192 ceph-mon[33110]: pgmap v922: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.1 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1004 B/s rd, 73 MiB/s wr, 1.18k op/s; 13381/349086 objects misplaced (3.833%); 15 MiB/s, 242 objects/s recovering 2024-03-26T03:18:27.876 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:27 smithi192 ceph-mon[33110]: osdmap e436: 8 total, 8 up, 6 in 2024-03-26T03:18:27.883 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:27.934 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:27.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:28.035 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:28.076 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:28.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:28.222 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:28.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:28.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:28.386 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:18:28.386 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:18:28.462 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:28.514 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:28.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 81 16 120858 120842 93.2276 79.75 0.0150432 0.0107215 2024-03-26T03:18:28.545 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T03:18:28.584 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:18:28.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:28.676 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:28.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:28.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:28.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1705862915' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:18:28.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:28 smithi066 ceph-mon[29666]: from='client.35912 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:18:28.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1705862915' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:18:28.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:28 smithi066 ceph-mon[37920]: from='client.35912 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:18:28.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:28.967 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:28.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:28.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1705862915' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:18:28.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:28 smithi192 ceph-mon[33110]: from='client.35912 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:18:29.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:18:29.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:29.127 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:29.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:29.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:29.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:29.411 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:29.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:29.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 82 16 122075 122059 93.0182 76.0625 0.0234047 0.0107463 2024-03-26T03:18:29.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:29.660 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.707 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:29.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:29.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:29.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:29 smithi192 ceph-mon[33110]: pgmap v924: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.1 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s; 13381/349086 objects misplaced (3.833%); 15 MiB/s, 247 objects/s recovering 2024-03-26T03:18:29.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:29 smithi192 ceph-mon[33110]: from='client.35912 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:18:29.836 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:29 smithi192 ceph-mon[33110]: osdmap e437: 8 total, 8 up, 6 in 2024-03-26T03:18:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:29.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:29.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:29 smithi066 ceph-mon[37920]: pgmap v924: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.1 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s; 13381/349086 objects misplaced (3.833%); 15 MiB/s, 247 objects/s recovering 2024-03-26T03:18:29.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:29 smithi066 ceph-mon[37920]: from='client.35912 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:18:29.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:29 smithi066 ceph-mon[37920]: osdmap e437: 8 total, 8 up, 6 in 2024-03-26T03:18:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:29 smithi066 ceph-mon[29666]: pgmap v924: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.1 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s; 13381/349086 objects misplaced (3.833%); 15 MiB/s, 247 objects/s recovering 2024-03-26T03:18:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:29 smithi066 ceph-mon[29666]: from='client.35912 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:18:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:29 smithi066 ceph-mon[29666]: osdmap e437: 8 total, 8 up, 6 in 2024-03-26T03:18:29.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:29.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:29.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:30.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:30.134 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:30.236 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:30.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:30.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:30.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:30.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:18:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:18:30.522 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:30.529 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 83 15 123310 123295 92.8282 77.25 0.0137834 0.0107676 2024-03-26T03:18:30.573 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:30.675 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:30.704 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:30.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:30.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:30.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:30.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:30.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:31.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:31.103 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T03:18:31.103 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:18:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:31.135 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:31.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:31.237 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:31.272 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:18:31.273 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:18:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:31.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:31.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:31.385 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:31.437 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:31.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:31.530 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 84 16 124425 124409 92.5512 69.625 0.0113326 0.0108001 2024-03-26T03:18:31.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:31.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:31.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:31.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:31 smithi066 ceph-mon[29666]: pgmap v926: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s; 12785/362403 objects misplaced (3.528%); 16 MiB/s, 251 objects/s recovering 2024-03-26T03:18:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:31 smithi066 ceph-mon[37920]: pgmap v926: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s; 12785/362403 objects misplaced (3.528%); 16 MiB/s, 251 objects/s recovering 2024-03-26T03:18:31.678 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:31.685 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:31.752 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:31.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:31.769 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:31.854 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:31.905 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:31.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:31 smithi192 ceph-mon[33110]: pgmap v926: 178 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 175 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s; 12785/362403 objects misplaced (3.528%); 16 MiB/s, 251 objects/s recovering 2024-03-26T03:18:32.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:32.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:32.142 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:32.197 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:32.300 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.332 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:32.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:32.433 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:32.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:32.530 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 85 16 125552 125536 92.291 70.4375 0.0111716 0.0108299 2024-03-26T03:18:32.582 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:32.629 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:32.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:32.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:32.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[29666]: osdmap e438: 8 total, 8 up, 6 in 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[37920]: osdmap e438: 8 total, 8 up, 6 in 2024-03-26T03:18:32.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:18:32.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:32 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:18:32.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:32.895 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:32.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:32.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:32 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:18:32.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:32 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:18:32.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:32 smithi192 ceph-mon[33110]: osdmap e438: 8 total, 8 up, 6 in 2024-03-26T03:18:32.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:32 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:18:32.926 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:32 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:18:32.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:18:32.997 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:33.043 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:33.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:33.189 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:33.291 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:33.335 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:33.382 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:33.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:33.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:33.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 86 16 126819 126803 92.1386 79.1875 0.0128479 0.0108485 2024-03-26T03:18:33.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:33.599 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:18:33.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:33.708 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:33.723 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[29666]: pgmap v928: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 12695/368232 objects misplaced (3.448%); 14 MiB/s, 223 objects/s recovering 2024-03-26T03:18:33.723 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[29666]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:18:33.723 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/84467696' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:18:33.723 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[29666]: osdmap e439: 8 total, 8 up, 6 in 2024-03-26T03:18:33.723 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[29666]: from='client.25858 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:18:33.723 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[37920]: pgmap v928: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 12695/368232 objects misplaced (3.448%); 14 MiB/s, 223 objects/s recovering 2024-03-26T03:18:33.724 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[37920]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:18:33.724 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/84467696' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:18:33.724 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[37920]: osdmap e439: 8 total, 8 up, 6 in 2024-03-26T03:18:33.724 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-mon[37920]: from='client.25858 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:18:33.765 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:33 smithi192 ceph-mon[33110]: pgmap v928: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.06k op/s; 12695/368232 objects misplaced (3.448%); 14 MiB/s, 223 objects/s recovering 2024-03-26T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:33 smithi192 ceph-mon[33110]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:33 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/84467696' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:33 smithi192 ceph-mon[33110]: osdmap e439: 8 total, 8 up, 6 in 2024-03-26T03:18:33.779 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:33 smithi192 ceph-mon[33110]: from='client.25858 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:18:33.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:33.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:33.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:34.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:34.056 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:34.076 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:18:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:18:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:18:34.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:34.189 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:34.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:34.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:34.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:34.431 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:34.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:34.530 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 87 16 128152 128136 92.037 83.3125 0.00469526 0.0108607 2024-03-26T03:18:34.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:34.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:34.714 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:34 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:34.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:34 smithi066 ceph-mon[29666]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:18:34.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:34 smithi066 ceph-mon[29666]: osdmap e440: 8 total, 8 up, 6 in 2024-03-26T03:18:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:34 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:34 smithi066 ceph-mon[37920]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:18:34.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:34 smithi066 ceph-mon[37920]: osdmap e440: 8 total, 8 up, 6 in 2024-03-26T03:18:34.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:34.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:34.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:34 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:34.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:34 smithi192 ceph-mon[33110]: from='client.25858 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:18:34.850 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:34 smithi192 ceph-mon[33110]: osdmap e440: 8 total, 8 up, 6 in 2024-03-26T03:18:34.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:34.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:34.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:34.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:35.013 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:35.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:35.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:35.212 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:35.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:35.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:35.421 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-03-26T03:18:35.421 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:18:35.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:35.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:35.530 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 88 16 129469 129453 91.9265 82.3125 0.0137645 0.010874 2024-03-26T03:18:35.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:35.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:35.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:35 smithi192 ceph-mon[33110]: pgmap v931: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 866 B/s rd, 68 MiB/s wr, 1.08k op/s; 12695/368232 objects misplaced (3.448%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:18:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:35.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:35.777 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:35 smithi066 ceph-mon[29666]: pgmap v931: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 866 B/s rd, 68 MiB/s wr, 1.08k op/s; 12695/368232 objects misplaced (3.448%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:18:35.777 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:35 smithi066 ceph-mon[37920]: pgmap v931: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 866 B/s rd, 68 MiB/s wr, 1.08k op/s; 12695/368232 objects misplaced (3.448%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:18:35.867 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:35.913 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:35.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:36.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:36.032 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:18:36.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:36.085 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:18:36.124 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:18:36.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:36.170 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:18:36.202 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:36.214 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:18:36.254 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:18:36.299 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:18:36.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:36.342 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:18:36.346 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:18:36.351 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:36.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:18:36.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:36.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:36.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 89 16 130706 130690 91.7622 77.3125 0.0134271 0.0108929 2024-03-26T03:18:36.592 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.647 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:36.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:36.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:36 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2081311600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:36.748 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:36.784 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:36.790 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:36.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:36 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2081311600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:36.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:36 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2081311600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:36.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:36.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:36.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:37.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:37.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:37.176 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:37.229 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:37.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.384 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:37.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:37.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:37.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:18:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:37.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:37.530 INFO:tasks.radosbench.radosbench.0.smithi066.stdout: 90 1 131895 131894 91.5787 75.25 0.0284074 0.0109147 2024-03-26T03:18:37.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:37.626 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:37 smithi066 ceph-mon[29666]: pgmap v932: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 11445/386742 objects misplaced (2.959%); 26 MiB/s, 410 objects/s recovering 2024-03-26T03:18:37.626 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:37 smithi066 ceph-mon[37920]: pgmap v932: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 11445/386742 objects misplaced (2.959%); 26 MiB/s, 410 objects/s recovering 2024-03-26T03:18:37.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:37.749 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:37.789 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:37.879 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:37.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:37.933 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:37.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:37 smithi192 ceph-mon[33110]: pgmap v932: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 11445/386742 objects misplaced (2.959%); 26 MiB/s, 410 objects/s recovering 2024-03-26T03:18:38.011 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:18:38.015 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:18:38.033 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.070 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:38.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:38.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:38.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:38.302 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:38.336 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:38.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:38.472 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total time run: 90.0143 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Total writes made: 131895 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Write size: 65536 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Object size: 65536 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Bandwidth (MB/sec): 91.5792 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Bandwidth: 30.5633 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max bandwidth (MB/sec): 204.625 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min bandwidth (MB/sec): 0 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average IOPS: 1465 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev IOPS: 489.022 2024-03-26T03:18:38.531 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max IOPS: 3274 2024-03-26T03:18:38.532 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min IOPS: 0 2024-03-26T03:18:38.532 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Average Latency(s): 0.0109148 2024-03-26T03:18:38.532 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Stddev Latency(s): 0.0242861 2024-03-26T03:18:38.532 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Max latency(s): 2.14475 2024-03-26T03:18:38.532 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Min latency(s): 0.00143665 2024-03-26T03:18:38.535 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:18:38.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:38.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:38.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:38.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:38 smithi066 ceph-mon[29666]: from='client.25870 v1:172.21.15.66:0/3315830933' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:18:38.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:38 smithi066 ceph-mon[29666]: pgmap v933: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 745 B/s rd, 56 MiB/s wr, 899 op/s; 11445/386742 objects misplaced (2.959%); 17 MiB/s, 266 objects/s recovering 2024-03-26T03:18:38.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:38 smithi066 ceph-mon[37920]: from='client.25870 v1:172.21.15.66:0/3315830933' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:18:38.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:38 smithi066 ceph-mon[37920]: pgmap v933: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 745 B/s rd, 56 MiB/s wr, 899 op/s; 11445/386742 objects misplaced (2.959%); 17 MiB/s, 266 objects/s recovering 2024-03-26T03:18:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:38.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:38.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:38.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:38.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:38.914 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:38 smithi192 ceph-mon[33110]: from='client.25870 v1:172.21.15.66:0/3315830933' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:18:38.914 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:38 smithi192 ceph-mon[33110]: pgmap v933: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 745 B/s rd, 56 MiB/s wr, 899 op/s; 11445/386742 objects misplaced (2.959%); 17 MiB/s, 266 objects/s recovering 2024-03-26T03:18:38.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:38.993 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":932,"stamp":"2024-03-26T03:18:36.431856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8434044433,"num_objects":128914,"num_object_clones":0,"num_object_copies":386742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11445,"num_objects_unfound":0,"num_objects_dirty":128914,"num_whiteouts":0,"num_read":2587,"num_read_kb":2310,"num_write":129602,"num_write_kb":8235876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10389,"num_bytes_recovered":665826529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57828,"ondisk_log_size":57828,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":537,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":29981300,"kb_used_data":25322908,"kb_used_omap":12,"kb_used_meta":4656563,"kb_avail":532489612,"statfs":{"total":575970213888,"available":545269362688,"internally_reserved":0,"allocated":25930657792,"data_stored":25926437986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12419,"internal_metadata":4768321405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531693568,"num_objects":8113,"num_object_clones":0,"num_object_copies":24339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1340,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8113,"num_write_kb":519232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2465,"num_bytes_recovered":161546240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002311"},"pg_stats":[{"pgid":"11.7","version":"440'7986","reported_seq":8070,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.767131+0000","last_change":"2024-03-26T03:16:56.903159+0000","last_active":"2024-03-26T03:18:34.767131+0000","last_peered":"2024-03-26T03:18:34.767131+0000","last_clean":"2024-03-26T03:18:34.767131+0000","last_became_active":"2024-03-26T03:16:56.900984+0000","last_became_peered":"2024-03-26T03:16:56.900984+0000","last_unstale":"2024-03-26T03:18:34.767131+0000","last_undegraded":"2024-03-26T03:18:34.767131+0000","last_fullsized":"2024-03-26T03:18:34.767131+0000","mapping_epoch":403,"log_start":"426'4700","ondisk_log_start":"426'4700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3286,"log_dups_size":928,"ondisk_log_size":3286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.825071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523370496,"num_objects":7986,"num_object_clones":0,"num_object_copies":23958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":511104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":735,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598591+0000","last_change":"2024-03-26T03:16:31.121425+0000","last_active":"2024-03-26T03:18:33.598591+0000","last_peered":"2024-03-26T03:18:33.598591+0000","last_clean":"2024-03-26T03:18:33.598591+0000","last_became_active":"2024-03-26T03:14:40.174462+0000","last_became_peered":"2024-03-26T03:14:40.174462+0000","last_unstale":"2024-03-26T03:18:33.598591+0000","last_undegraded":"2024-03-26T03:18:33.598591+0000","last_fullsized":"2024-03-26T03:18:33.598591+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.121331+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.121331+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"342'10","reported_seq":423,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589266+0000","last_change":"2024-03-26T03:17:48.646060+0000","last_active":"2024-03-26T03:18:33.589266+0000","last_peered":"2024-03-26T03:18:33.589266+0000","last_clean":"2024-03-26T03:18:33.589266+0000","last_became_active":"2024-03-26T03:14:37.273000+0000","last_became_peered":"2024-03-26T03:14:37.273000+0000","last_unstale":"2024-03-26T03:18:33.589266+0000","last_undegraded":"2024-03-26T03:18:33.589266+0000","last_fullsized":"2024-03-26T03:18:33.589266+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'10","last_scrub_stamp":"2024-03-26T03:17:48.645977+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.645977+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:53.530710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":940,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.582855+0000","last_change":"2024-03-26T03:16:51.697617+0000","last_active":"2024-03-26T03:18:33.582855+0000","last_peered":"2024-03-26T03:18:33.582855+0000","last_clean":"2024-03-26T03:18:33.582855+0000","last_became_active":"2024-03-26T03:14:37.267205+0000","last_became_peered":"2024-03-26T03:14:37.267205+0000","last_unstale":"2024-03-26T03:18:33.582855+0000","last_undegraded":"2024-03-26T03:18:33.582855+0000","last_fullsized":"2024-03-26T03:18:33.582855+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:16:51.697464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.697464+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.825124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":245,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595769+0000","last_change":"2024-03-26T03:17:57.225859+0000","last_active":"2024-03-26T03:18:33.595769+0000","last_peered":"2024-03-26T03:18:33.595769+0000","last_clean":"2024-03-26T03:18:33.595769+0000","last_became_active":"2024-03-26T03:14:37.264771+0000","last_became_peered":"2024-03-26T03:14:37.264771+0000","last_unstale":"2024-03-26T03:18:33.595769+0000","last_undegraded":"2024-03-26T03:18:33.595769+0000","last_fullsized":"2024-03-26T03:18:33.595769+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:57.708114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":650,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.588794+0000","last_change":"2024-03-26T03:18:07.660598+0000","last_active":"2024-03-26T03:18:33.588794+0000","last_peered":"2024-03-26T03:18:33.588794+0000","last_clean":"2024-03-26T03:18:33.588794+0000","last_became_active":"2024-03-26T03:13:35.269469+0000","last_became_peered":"2024-03-26T03:13:35.269469+0000","last_unstale":"2024-03-26T03:18:33.588794+0000","last_undegraded":"2024-03-26T03:18:33.588794+0000","last_fullsized":"2024-03-26T03:18:33.588794+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_clean_scrub_stamp":"2024-03-26T03:18:07.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.629198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"440'7921","reported_seq":8005,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.729443+0000","last_change":"2024-03-26T03:16:56.903838+0000","last_active":"2024-03-26T03:18:34.729443+0000","last_peered":"2024-03-26T03:18:34.729443+0000","last_clean":"2024-03-26T03:18:34.729443+0000","last_became_active":"2024-03-26T03:16:56.903390+0000","last_became_peered":"2024-03-26T03:16:56.903390+0000","last_unstale":"2024-03-26T03:18:34.729443+0000","last_undegraded":"2024-03-26T03:18:34.729443+0000","last_fullsized":"2024-03-26T03:18:34.729443+0000","mapping_epoch":403,"log_start":"425'4400","ondisk_log_start":"425'4400","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519110656,"num_objects":7921,"num_object_clones":0,"num_object_copies":23763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7921,"num_write_kb":506944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":332,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599299+0000","last_change":"2024-03-26T03:17:59.592471+0000","last_active":"2024-03-26T03:18:33.599299+0000","last_peered":"2024-03-26T03:18:33.599299+0000","last_clean":"2024-03-26T03:18:33.599299+0000","last_became_active":"2024-03-26T03:17:59.592113+0000","last_became_peered":"2024-03-26T03:17:59.592113+0000","last_unstale":"2024-03-26T03:18:33.599299+0000","last_undegraded":"2024-03-26T03:18:33.599299+0000","last_fullsized":"2024-03-26T03:18:33.599299+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.778219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.778219+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:37.330671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":715,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600878+0000","last_change":"2024-03-26T03:17:38.464630+0000","last_active":"2024-03-26T03:18:33.600878+0000","last_peered":"2024-03-26T03:18:33.600878+0000","last_clean":"2024-03-26T03:18:33.600878+0000","last_became_active":"2024-03-26T03:14:22.579044+0000","last_became_peered":"2024-03-26T03:14:22.579044+0000","last_unstale":"2024-03-26T03:18:33.600878+0000","last_undegraded":"2024-03-26T03:18:33.600878+0000","last_fullsized":"2024-03-26T03:18:33.600878+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:17:38.464358+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:17:38.464358+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:59.515865+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":824,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601572+0000","last_change":"2024-03-26T03:17:56.571530+0000","last_active":"2024-03-26T03:18:33.601572+0000","last_peered":"2024-03-26T03:18:33.601572+0000","last_clean":"2024-03-26T03:18:33.601572+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:18:33.601572+0000","last_undegraded":"2024-03-26T03:18:33.601572+0000","last_fullsized":"2024-03-26T03:18:33.601572+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:16.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:15:16.933002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597643+0000","last_change":"2024-03-26T03:17:56.568649+0000","last_active":"2024-03-26T03:18:33.597643+0000","last_peered":"2024-03-26T03:18:33.597643+0000","last_clean":"2024-03-26T03:18:33.597643+0000","last_became_active":"2024-03-26T03:17:56.568322+0000","last_became_peered":"2024-03-26T03:17:56.568322+0000","last_unstale":"2024-03-26T03:18:33.597643+0000","last_undegraded":"2024-03-26T03:18:33.597643+0000","last_fullsized":"2024-03-26T03:18:33.597643+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.222175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.222175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:56.779239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":138,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595880+0000","last_change":"2024-03-26T03:16:50.206535+0000","last_active":"2024-03-26T03:18:33.595880+0000","last_peered":"2024-03-26T03:18:33.595880+0000","last_clean":"2024-03-26T03:18:33.595880+0000","last_became_active":"2024-03-26T03:16:50.206224+0000","last_became_peered":"2024-03-26T03:16:50.206224+0000","last_unstale":"2024-03-26T03:18:33.595880+0000","last_undegraded":"2024-03-26T03:18:33.595880+0000","last_fullsized":"2024-03-26T03:18:33.595880+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:26.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:13:26.722546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"440'8054","reported_seq":8138,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.787047+0000","last_change":"2024-03-26T03:16:56.908753+0000","last_active":"2024-03-26T03:18:34.787047+0000","last_peered":"2024-03-26T03:18:34.787047+0000","last_clean":"2024-03-26T03:18:34.787047+0000","last_became_active":"2024-03-26T03:16:56.908452+0000","last_became_peered":"2024-03-26T03:16:56.908452+0000","last_unstale":"2024-03-26T03:18:34.787047+0000","last_undegraded":"2024-03-26T03:18:34.787047+0000","last_fullsized":"2024-03-26T03:18:34.787047+0000","mapping_epoch":403,"log_start":"426'4800","ondisk_log_start":"426'4800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3254,"log_dups_size":986,"ondisk_log_size":3254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.825192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527826944,"num_objects":8054,"num_object_clones":0,"num_object_copies":24162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8054,"num_write_kb":515456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":911,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589961+0000","last_change":"2024-03-26T03:17:32.537859+0000","last_active":"2024-03-26T03:18:33.589961+0000","last_peered":"2024-03-26T03:18:33.589961+0000","last_clean":"2024-03-26T03:18:33.589961+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:18:33.589961+0000","last_undegraded":"2024-03-26T03:18:33.589961+0000","last_fullsized":"2024-03-26T03:18:33.589961+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:32.537761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.537761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:34.406361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":955,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589241+0000","last_change":"2024-03-26T03:18:02.671958+0000","last_active":"2024-03-26T03:18:33.589241+0000","last_peered":"2024-03-26T03:18:33.589241+0000","last_clean":"2024-03-26T03:18:33.589241+0000","last_became_active":"2024-03-26T03:14:37.266473+0000","last_became_peered":"2024-03-26T03:14:37.266473+0000","last_unstale":"2024-03-26T03:18:33.589241+0000","last_undegraded":"2024-03-26T03:18:33.589241+0000","last_fullsized":"2024-03-26T03:18:33.589241+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:18:02.671821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:02.671821+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:20.660317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":684,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.603978+0000","last_change":"2024-03-26T03:17:54.517355+0000","last_active":"2024-03-26T03:18:33.603978+0000","last_peered":"2024-03-26T03:18:33.603978+0000","last_clean":"2024-03-26T03:18:33.603978+0000","last_became_active":"2024-03-26T03:16:50.208923+0000","last_became_peered":"2024-03-26T03:16:50.208923+0000","last_unstale":"2024-03-26T03:18:33.603978+0000","last_undegraded":"2024-03-26T03:18:33.603978+0000","last_fullsized":"2024-03-26T03:18:33.603978+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:54.517230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.517230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:09.950125+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":855,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589874+0000","last_change":"2024-03-26T03:18:05.489195+0000","last_active":"2024-03-26T03:18:33.589874+0000","last_peered":"2024-03-26T03:18:33.589874+0000","last_clean":"2024-03-26T03:18:33.589874+0000","last_became_active":"2024-03-26T03:14:37.242217+0000","last_became_peered":"2024-03-26T03:14:37.242217+0000","last_unstale":"2024-03-26T03:18:33.589874+0000","last_undegraded":"2024-03-26T03:18:33.589874+0000","last_fullsized":"2024-03-26T03:18:33.589874+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:05.489069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.227664+0000","last_clean_scrub_stamp":"2024-03-26T03:18:05.489069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:21.866500+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":775,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593406+0000","last_change":"2024-03-26T03:17:40.190354+0000","last_active":"2024-03-26T03:18:33.593406+0000","last_peered":"2024-03-26T03:18:33.593406+0000","last_clean":"2024-03-26T03:18:33.593406+0000","last_became_active":"2024-03-26T03:14:37.270129+0000","last_became_peered":"2024-03-26T03:14:37.270129+0000","last_unstale":"2024-03-26T03:18:33.593406+0000","last_undegraded":"2024-03-26T03:18:33.593406+0000","last_fullsized":"2024-03-26T03:18:33.593406+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.189873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.189873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:47.578903+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"440'8122","reported_seq":8206,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.471454+0000","last_change":"2024-03-26T03:16:56.905137+0000","last_active":"2024-03-26T03:18:34.471454+0000","last_peered":"2024-03-26T03:18:34.471454+0000","last_clean":"2024-03-26T03:18:34.471454+0000","last_became_active":"2024-03-26T03:16:56.904798+0000","last_became_peered":"2024-03-26T03:16:56.904798+0000","last_unstale":"2024-03-26T03:18:34.471454+0000","last_undegraded":"2024-03-26T03:18:34.471454+0000","last_fullsized":"2024-03-26T03:18:34.471454+0000","mapping_epoch":403,"log_start":"426'4900","ondisk_log_start":"426'4900","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3222,"log_dups_size":2487,"ondisk_log_size":3222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532283392,"num_objects":8122,"num_object_clones":0,"num_object_copies":24366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":519808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":53,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598494+0000","last_change":"2024-03-26T03:17:56.585807+0000","last_active":"2024-03-26T03:18:33.598494+0000","last_peered":"2024-03-26T03:18:33.598494+0000","last_clean":"2024-03-26T03:18:33.598494+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:18:33.598494+0000","last_undegraded":"2024-03-26T03:18:33.598494+0000","last_fullsized":"2024-03-26T03:18:33.598494+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:17:36.443102+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:17:36.443102+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.831709+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":35,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598418+0000","last_change":"2024-03-26T03:17:56.568076+0000","last_active":"2024-03-26T03:18:33.598418+0000","last_peered":"2024-03-26T03:18:33.598418+0000","last_clean":"2024-03-26T03:18:33.598418+0000","last_became_active":"2024-03-26T03:17:56.567051+0000","last_became_peered":"2024-03-26T03:17:56.567051+0000","last_unstale":"2024-03-26T03:18:33.598418+0000","last_undegraded":"2024-03-26T03:18:33.598418+0000","last_fullsized":"2024-03-26T03:18:33.598418+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.464363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.464363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":825,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.604787+0000","last_change":"2024-03-26T03:16:46.947202+0000","last_active":"2024-03-26T03:18:33.604787+0000","last_peered":"2024-03-26T03:18:33.604787+0000","last_clean":"2024-03-26T03:18:33.604787+0000","last_became_active":"2024-03-26T03:13:35.277164+0000","last_became_peered":"2024-03-26T03:13:35.277164+0000","last_unstale":"2024-03-26T03:18:33.604787+0000","last_undegraded":"2024-03-26T03:18:33.604787+0000","last_fullsized":"2024-03-26T03:18:33.604787+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:46.947118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.947118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":231,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596695+0000","last_change":"2024-03-26T03:14:21.588254+0000","last_active":"2024-03-26T03:18:33.596695+0000","last_peered":"2024-03-26T03:18:33.596695+0000","last_clean":"2024-03-26T03:18:33.596695+0000","last_became_active":"2024-03-26T03:14:21.586859+0000","last_became_peered":"2024-03-26T03:14:21.586859+0000","last_unstale":"2024-03-26T03:18:33.596695+0000","last_undegraded":"2024-03-26T03:18:33.596695+0000","last_fullsized":"2024-03-26T03:18:33.596695+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":533,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593172+0000","last_change":"2024-03-26T03:16:28.866577+0000","last_active":"2024-03-26T03:18:33.593172+0000","last_peered":"2024-03-26T03:18:33.593172+0000","last_clean":"2024-03-26T03:18:33.593172+0000","last_became_active":"2024-03-26T03:14:21.590629+0000","last_became_peered":"2024-03-26T03:14:21.590629+0000","last_unstale":"2024-03-26T03:18:33.593172+0000","last_undegraded":"2024-03-26T03:18:33.593172+0000","last_fullsized":"2024-03-26T03:18:33.593172+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:28.866477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.866477+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"440'7968","reported_seq":8052,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.706059+0000","last_change":"2024-03-26T03:16:56.903756+0000","last_active":"2024-03-26T03:18:34.706059+0000","last_peered":"2024-03-26T03:18:34.706059+0000","last_clean":"2024-03-26T03:18:34.706059+0000","last_became_active":"2024-03-26T03:16:56.903414+0000","last_became_peered":"2024-03-26T03:16:56.903414+0000","last_unstale":"2024-03-26T03:18:34.706059+0000","last_undegraded":"2024-03-26T03:18:34.706059+0000","last_fullsized":"2024-03-26T03:18:34.706059+0000","mapping_epoch":403,"log_start":"425'4500","ondisk_log_start":"425'4500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3468,"log_dups_size":0,"ondisk_log_size":3468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522190848,"num_objects":7968,"num_object_clones":0,"num_object_copies":23904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7968,"num_write_kb":509952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":700,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593368+0000","last_change":"2024-03-26T03:16:30.821771+0000","last_active":"2024-03-26T03:18:33.593368+0000","last_peered":"2024-03-26T03:18:33.593368+0000","last_clean":"2024-03-26T03:18:33.593368+0000","last_became_active":"2024-03-26T03:14:40.169332+0000","last_became_peered":"2024-03-26T03:14:40.169332+0000","last_unstale":"2024-03-26T03:18:33.593368+0000","last_undegraded":"2024-03-26T03:18:33.593368+0000","last_fullsized":"2024-03-26T03:18:33.593368+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.821670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.821670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":849,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599026+0000","last_change":"2024-03-26T03:16:39.220608+0000","last_active":"2024-03-26T03:18:33.599026+0000","last_peered":"2024-03-26T03:18:33.599026+0000","last_clean":"2024-03-26T03:18:33.599026+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:18:33.599026+0000","last_undegraded":"2024-03-26T03:18:33.599026+0000","last_fullsized":"2024-03-26T03:18:33.599026+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'20","last_scrub_stamp":"2024-03-26T03:16:39.220501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.220501+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":811,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.602022+0000","last_change":"2024-03-26T03:15:09.812769+0000","last_active":"2024-03-26T03:18:33.602022+0000","last_peered":"2024-03-26T03:18:33.602022+0000","last_clean":"2024-03-26T03:18:33.602022+0000","last_became_active":"2024-03-26T03:14:37.251633+0000","last_became_peered":"2024-03-26T03:14:37.251633+0000","last_unstale":"2024-03-26T03:18:33.602022+0000","last_undegraded":"2024-03-26T03:18:33.602022+0000","last_fullsized":"2024-03-26T03:18:33.602022+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:09.812652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:15:09.812652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"401'11","reported_seq":934,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594059+0000","last_change":"2024-03-26T03:16:50.207177+0000","last_active":"2024-03-26T03:18:33.594059+0000","last_peered":"2024-03-26T03:18:33.594059+0000","last_clean":"2024-03-26T03:18:33.594059+0000","last_became_active":"2024-03-26T03:16:50.206797+0000","last_became_peered":"2024-03-26T03:16:50.206797+0000","last_unstale":"2024-03-26T03:18:33.594059+0000","last_undegraded":"2024-03-26T03:18:33.594059+0000","last_fullsized":"2024-03-26T03:18:33.594059+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:12:28.903038+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.903038+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599379+0000","last_change":"2024-03-26T03:17:56.574248+0000","last_active":"2024-03-26T03:18:33.599379+0000","last_peered":"2024-03-26T03:18:33.599379+0000","last_clean":"2024-03-26T03:18:33.599379+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:18:33.599379+0000","last_undegraded":"2024-03-26T03:18:33.599379+0000","last_fullsized":"2024-03-26T03:18:33.599379+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:32.853807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.853807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"440'7995","reported_seq":8079,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:35.510145+0000","last_change":"2024-03-26T03:16:56.909076+0000","last_active":"2024-03-26T03:18:35.510145+0000","last_peered":"2024-03-26T03:18:35.510145+0000","last_clean":"2024-03-26T03:18:35.510145+0000","last_became_active":"2024-03-26T03:16:56.908486+0000","last_became_peered":"2024-03-26T03:16:56.908486+0000","last_unstale":"2024-03-26T03:18:35.510145+0000","last_undegraded":"2024-03-26T03:18:35.510145+0000","last_fullsized":"2024-03-26T03:18:35.510145+0000","mapping_epoch":403,"log_start":"426'4700","ondisk_log_start":"426'4700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523960320,"num_objects":7995,"num_object_clones":0,"num_object_copies":23985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7995,"num_write_kb":511680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":390,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.588736+0000","last_change":"2024-03-26T03:16:50.672364+0000","last_active":"2024-03-26T03:18:33.588736+0000","last_peered":"2024-03-26T03:18:33.588736+0000","last_clean":"2024-03-26T03:18:33.588736+0000","last_became_active":"2024-03-26T03:16:50.205474+0000","last_became_peered":"2024-03-26T03:16:50.205474+0000","last_unstale":"2024-03-26T03:18:33.588736+0000","last_undegraded":"2024-03-26T03:18:33.588736+0000","last_fullsized":"2024-03-26T03:18:33.588736+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:50.672268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.672268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":874,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601867+0000","last_change":"2024-03-26T03:16:28.788850+0000","last_active":"2024-03-26T03:18:33.601867+0000","last_peered":"2024-03-26T03:18:33.601867+0000","last_clean":"2024-03-26T03:18:33.601867+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:18:33.601867+0000","last_undegraded":"2024-03-26T03:18:33.601867+0000","last_fullsized":"2024-03-26T03:18:33.601867+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'27","last_scrub_stamp":"2024-03-26T03:16:28.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.788749+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":811,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597963+0000","last_change":"2024-03-26T03:14:39.896972+0000","last_active":"2024-03-26T03:18:33.597963+0000","last_peered":"2024-03-26T03:18:33.597963+0000","last_clean":"2024-03-26T03:18:33.597963+0000","last_became_active":"2024-03-26T03:14:37.249790+0000","last_became_peered":"2024-03-26T03:14:37.249790+0000","last_unstale":"2024-03-26T03:18:33.597963+0000","last_undegraded":"2024-03-26T03:18:33.597963+0000","last_fullsized":"2024-03-26T03:18:33.597963+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:39.896823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:14:39.896823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"428'12","reported_seq":48,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599400+0000","last_change":"2024-03-26T03:17:56.575479+0000","last_active":"2024-03-26T03:18:33.599400+0000","last_peered":"2024-03-26T03:18:33.599400+0000","last_clean":"2024-03-26T03:18:33.599400+0000","last_became_active":"2024-03-26T03:17:56.566007+0000","last_became_peered":"2024-03-26T03:17:56.566007+0000","last_unstale":"2024-03-26T03:18:33.599400+0000","last_undegraded":"2024-03-26T03:18:33.599400+0000","last_fullsized":"2024-03-26T03:18:33.599400+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:13:30.083400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:30.083400+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":626,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601109+0000","last_change":"2024-03-26T03:13:35.276632+0000","last_active":"2024-03-26T03:18:33.601109+0000","last_peered":"2024-03-26T03:18:33.601109+0000","last_clean":"2024-03-26T03:18:33.601109+0000","last_became_active":"2024-03-26T03:13:35.275396+0000","last_became_peered":"2024-03-26T03:13:35.275396+0000","last_unstale":"2024-03-26T03:18:33.601109+0000","last_undegraded":"2024-03-26T03:18:33.601109+0000","last_fullsized":"2024-03-26T03:18:33.601109+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:18.063340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:13:18.063340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"440'8031","reported_seq":8115,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.724918+0000","last_change":"2024-03-26T03:16:56.905918+0000","last_active":"2024-03-26T03:18:34.724918+0000","last_peered":"2024-03-26T03:18:34.724918+0000","last_clean":"2024-03-26T03:18:34.724918+0000","last_became_active":"2024-03-26T03:16:56.905606+0000","last_became_peered":"2024-03-26T03:16:56.905606+0000","last_unstale":"2024-03-26T03:18:34.724918+0000","last_undegraded":"2024-03-26T03:18:34.724918+0000","last_fullsized":"2024-03-26T03:18:34.724918+0000","mapping_epoch":403,"log_start":"425'4500","ondisk_log_start":"425'4500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526319616,"num_objects":8031,"num_object_clones":0,"num_object_copies":24093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":951,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600469+0000","last_change":"2024-03-26T03:17:39.489433+0000","last_active":"2024-03-26T03:18:33.600469+0000","last_peered":"2024-03-26T03:18:33.600469+0000","last_clean":"2024-03-26T03:18:33.600469+0000","last_became_active":"2024-03-26T03:13:38.314531+0000","last_became_peered":"2024-03-26T03:13:38.314531+0000","last_unstale":"2024-03-26T03:18:33.600469+0000","last_undegraded":"2024-03-26T03:18:33.600469+0000","last_fullsized":"2024-03-26T03:18:33.600469+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:39.489210+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:17:39.489210+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.198140+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"291'31","reported_seq":878,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599811+0000","last_change":"2024-03-26T03:17:42.061990+0000","last_active":"2024-03-26T03:18:33.599811+0000","last_peered":"2024-03-26T03:18:33.599811+0000","last_clean":"2024-03-26T03:18:33.599811+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:18:33.599811+0000","last_undegraded":"2024-03-26T03:18:33.599811+0000","last_fullsized":"2024-03-26T03:18:33.599811+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:17:42.061780+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.061780+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.768905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":779,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598616+0000","last_change":"2024-03-26T03:14:37.253086+0000","last_active":"2024-03-26T03:18:33.598616+0000","last_peered":"2024-03-26T03:18:33.598616+0000","last_clean":"2024-03-26T03:18:33.598616+0000","last_became_active":"2024-03-26T03:14:37.252745+0000","last_became_peered":"2024-03-26T03:14:37.252745+0000","last_unstale":"2024-03-26T03:18:33.598616+0000","last_undegraded":"2024-03-26T03:18:33.598616+0000","last_fullsized":"2024-03-26T03:18:33.598616+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:31.980008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.980008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":631,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599123+0000","last_change":"2024-03-26T03:16:55.201337+0000","last_active":"2024-03-26T03:18:33.599123+0000","last_peered":"2024-03-26T03:18:33.599123+0000","last_clean":"2024-03-26T03:18:33.599123+0000","last_became_active":"2024-03-26T03:14:37.249741+0000","last_became_peered":"2024-03-26T03:14:37.249741+0000","last_unstale":"2024-03-26T03:18:33.599123+0000","last_undegraded":"2024-03-26T03:18:33.599123+0000","last_fullsized":"2024-03-26T03:18:33.599123+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.201190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.201190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":452,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594727+0000","last_change":"2024-03-26T03:17:44.249680+0000","last_active":"2024-03-26T03:18:33.594727+0000","last_peered":"2024-03-26T03:18:33.594727+0000","last_clean":"2024-03-26T03:18:33.594727+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:18:33.594727+0000","last_undegraded":"2024-03-26T03:18:33.594727+0000","last_fullsized":"2024-03-26T03:18:33.594727+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.249561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.249561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:03.829651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"440'7973","reported_seq":16077,"reported_epoch":440,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:18:35.557192+0000","last_change":"2024-03-26T03:17:57.690462+0000","last_active":"2024-03-26T03:18:35.557192+0000","last_peered":"2024-03-26T03:18:35.557192+0000","last_clean":"2024-03-26T03:17:55.558348+0000","last_became_active":"2024-03-26T03:17:57.680052+0000","last_became_peered":"2024-03-26T03:17:57.680052+0000","last_unstale":"2024-03-26T03:18:35.557192+0000","last_undegraded":"2024-03-26T03:18:35.557192+0000","last_fullsized":"2024-03-26T03:18:35.557192+0000","mapping_epoch":428,"log_start":"426'4700","ondisk_log_start":"426'4700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522518528,"num_objects":7973,"num_object_clones":0,"num_object_copies":23919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3046,"num_objects_unfound":0,"num_objects_dirty":7973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4008,"num_bytes_recovered":262668288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":7973}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":630,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593108+0000","last_change":"2024-03-26T03:17:55.204987+0000","last_active":"2024-03-26T03:18:33.593108+0000","last_peered":"2024-03-26T03:18:33.593108+0000","last_clean":"2024-03-26T03:18:33.593108+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:18:33.593108+0000","last_undegraded":"2024-03-26T03:18:33.593108+0000","last_fullsized":"2024-03-26T03:18:33.593108+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:17:55.204836+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.204836+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:56.241884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":884,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597636+0000","last_change":"2024-03-26T03:16:56.199180+0000","last_active":"2024-03-26T03:18:33.597636+0000","last_peered":"2024-03-26T03:18:33.597636+0000","last_clean":"2024-03-26T03:18:33.597636+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:18:33.597636+0000","last_undegraded":"2024-03-26T03:18:33.597636+0000","last_fullsized":"2024-03-26T03:18:33.597636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:16:56.199057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.199057+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":842,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590569+0000","last_change":"2024-03-26T03:17:41.533036+0000","last_active":"2024-03-26T03:18:33.590569+0000","last_peered":"2024-03-26T03:18:33.590569+0000","last_clean":"2024-03-26T03:18:33.590569+0000","last_became_active":"2024-03-26T03:14:37.276008+0000","last_became_peered":"2024-03-26T03:14:37.276008+0000","last_unstale":"2024-03-26T03:18:33.590569+0000","last_undegraded":"2024-03-26T03:18:33.590569+0000","last_fullsized":"2024-03-26T03:18:33.590569+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.532899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.532899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:45.840192+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":217,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590115+0000","last_change":"2024-03-26T03:16:40.872720+0000","last_active":"2024-03-26T03:18:33.590115+0000","last_peered":"2024-03-26T03:18:33.590115+0000","last_clean":"2024-03-26T03:18:33.590115+0000","last_became_active":"2024-03-26T03:14:37.248955+0000","last_became_peered":"2024-03-26T03:14:37.248955+0000","last_unstale":"2024-03-26T03:18:33.590115+0000","last_undegraded":"2024-03-26T03:18:33.590115+0000","last_fullsized":"2024-03-26T03:18:33.590115+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:40.872619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.872619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834711+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":623,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595238+0000","last_change":"2024-03-26T03:16:29.852173+0000","last_active":"2024-03-26T03:18:33.595238+0000","last_peered":"2024-03-26T03:18:33.595238+0000","last_clean":"2024-03-26T03:18:33.595238+0000","last_became_active":"2024-03-26T03:14:40.166583+0000","last_became_peered":"2024-03-26T03:14:40.166583+0000","last_unstale":"2024-03-26T03:18:33.595238+0000","last_undegraded":"2024-03-26T03:18:33.595238+0000","last_fullsized":"2024-03-26T03:18:33.595238+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.852073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.852073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"440'8241","reported_seq":8325,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:35.541616+0000","last_change":"2024-03-26T03:16:56.900055+0000","last_active":"2024-03-26T03:18:35.541616+0000","last_peered":"2024-03-26T03:18:35.541616+0000","last_clean":"2024-03-26T03:18:35.541616+0000","last_became_active":"2024-03-26T03:16:56.899679+0000","last_became_peered":"2024-03-26T03:16:56.899679+0000","last_unstale":"2024-03-26T03:18:35.541616+0000","last_undegraded":"2024-03-26T03:18:35.541616+0000","last_fullsized":"2024-03-26T03:18:35.541616+0000","mapping_epoch":403,"log_start":"426'5000","ondisk_log_start":"426'5000","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3241,"log_dups_size":0,"ondisk_log_size":3241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540082176,"num_objects":8241,"num_object_clones":0,"num_object_copies":24723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8241,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":801,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598209+0000","last_change":"2024-03-26T03:16:36.206735+0000","last_active":"2024-03-26T03:18:33.598209+0000","last_peered":"2024-03-26T03:18:33.598209+0000","last_clean":"2024-03-26T03:18:33.598209+0000","last_became_active":"2024-03-26T03:14:37.257004+0000","last_became_peered":"2024-03-26T03:14:37.257004+0000","last_unstale":"2024-03-26T03:18:33.598209+0000","last_undegraded":"2024-03-26T03:18:33.598209+0000","last_fullsized":"2024-03-26T03:18:33.598209+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:36.206634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.206634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":617,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593523+0000","last_change":"2024-03-26T03:16:38.931105+0000","last_active":"2024-03-26T03:18:33.593523+0000","last_peered":"2024-03-26T03:18:33.593523+0000","last_clean":"2024-03-26T03:18:33.593523+0000","last_became_active":"2024-03-26T03:14:37.278227+0000","last_became_peered":"2024-03-26T03:14:37.278227+0000","last_unstale":"2024-03-26T03:18:33.593523+0000","last_undegraded":"2024-03-26T03:18:33.593523+0000","last_fullsized":"2024-03-26T03:18:33.593523+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'54","last_scrub_stamp":"2024-03-26T03:16:38.931005+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.931005+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600346+0000","last_change":"2024-03-26T03:13:29.030426+0000","last_active":"2024-03-26T03:18:33.600346+0000","last_peered":"2024-03-26T03:18:33.600346+0000","last_clean":"2024-03-26T03:18:33.600346+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:18:33.600346+0000","last_undegraded":"2024-03-26T03:18:33.600346+0000","last_fullsized":"2024-03-26T03:18:33.600346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:13:29.030298+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.030298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597729+0000","last_change":"2024-03-26T03:17:56.567975+0000","last_active":"2024-03-26T03:18:33.597729+0000","last_peered":"2024-03-26T03:18:33.597729+0000","last_clean":"2024-03-26T03:18:33.597729+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:18:33.597729+0000","last_undegraded":"2024-03-26T03:18:33.597729+0000","last_fullsized":"2024-03-26T03:18:33.597729+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.179410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.179410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:09.819892+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":934,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598632+0000","last_change":"2024-03-26T03:17:54.095189+0000","last_active":"2024-03-26T03:18:33.598632+0000","last_peered":"2024-03-26T03:18:33.598632+0000","last_clean":"2024-03-26T03:18:33.598632+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:18:33.598632+0000","last_undegraded":"2024-03-26T03:18:33.598632+0000","last_fullsized":"2024-03-26T03:18:33.598632+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:54.095058+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.095058+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:01.417606+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"440'7970","reported_seq":9956,"reported_epoch":440,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:18:35.542895+0000","last_change":"2024-03-26T03:18:29.644500+0000","last_active":"2024-03-26T03:18:35.542895+0000","last_peered":"2024-03-26T03:18:35.542895+0000","last_clean":"2024-03-26T03:17:55.548043+0000","last_became_active":"2024-03-26T03:17:57.671666+0000","last_became_peered":"2024-03-26T03:17:57.671666+0000","last_unstale":"2024-03-26T03:18:35.542895+0000","last_undegraded":"2024-03-26T03:18:35.542895+0000","last_fullsized":"2024-03-26T03:18:35.542895+0000","mapping_epoch":428,"log_start":"426'4700","ondisk_log_start":"426'4700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3270,"log_dups_size":0,"ondisk_log_size":3270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522321920,"num_objects":7970,"num_object_clones":0,"num_object_copies":23910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6866,"num_objects_unfound":0,"num_objects_dirty":7970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7970,"num_write_kb":510080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":944,"num_bytes_recovered":61865984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":7970}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":834,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589548+0000","last_change":"2024-03-26T03:17:34.573811+0000","last_active":"2024-03-26T03:18:33.589548+0000","last_peered":"2024-03-26T03:18:33.589548+0000","last_clean":"2024-03-26T03:18:33.589548+0000","last_became_active":"2024-03-26T03:13:35.269721+0000","last_became_peered":"2024-03-26T03:13:35.269721+0000","last_unstale":"2024-03-26T03:18:33.589548+0000","last_undegraded":"2024-03-26T03:18:33.589548+0000","last_fullsized":"2024-03-26T03:18:33.589548+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.573719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.573719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:45.178496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":766,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.586883+0000","last_change":"2024-03-26T03:16:56.710503+0000","last_active":"2024-03-26T03:18:33.586883+0000","last_peered":"2024-03-26T03:18:33.586883+0000","last_clean":"2024-03-26T03:18:33.586883+0000","last_became_active":"2024-03-26T03:14:37.258677+0000","last_became_peered":"2024-03-26T03:14:37.258677+0000","last_unstale":"2024-03-26T03:18:33.586883+0000","last_undegraded":"2024-03-26T03:18:33.586883+0000","last_fullsized":"2024-03-26T03:18:33.586883+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:16:56.710408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.710408+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.825221+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":764,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598978+0000","last_change":"2024-03-26T03:14:37.257762+0000","last_active":"2024-03-26T03:18:33.598978+0000","last_peered":"2024-03-26T03:18:33.598978+0000","last_clean":"2024-03-26T03:18:33.598978+0000","last_became_active":"2024-03-26T03:14:37.257422+0000","last_became_peered":"2024-03-26T03:14:37.257422+0000","last_unstale":"2024-03-26T03:18:33.598978+0000","last_undegraded":"2024-03-26T03:18:33.598978+0000","last_fullsized":"2024-03-26T03:18:33.598978+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:35.954172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:35.954172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":819,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598211+0000","last_change":"2024-03-26T03:13:24.055056+0000","last_active":"2024-03-26T03:18:33.598211+0000","last_peered":"2024-03-26T03:18:33.598211+0000","last_clean":"2024-03-26T03:18:33.598211+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:18:33.598211+0000","last_undegraded":"2024-03-26T03:18:33.598211+0000","last_fullsized":"2024-03-26T03:18:33.598211+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:24.054721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:13:24.054721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":904,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590869+0000","last_change":"2024-03-26T03:16:27.907348+0000","last_active":"2024-03-26T03:18:33.590869+0000","last_peered":"2024-03-26T03:18:33.590869+0000","last_clean":"2024-03-26T03:18:33.590869+0000","last_became_active":"2024-03-26T03:14:22.569851+0000","last_became_peered":"2024-03-26T03:14:22.569851+0000","last_unstale":"2024-03-26T03:18:33.590869+0000","last_undegraded":"2024-03-26T03:18:33.590869+0000","last_fullsized":"2024-03-26T03:18:33.590869+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:27.907229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:27.907229+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"440'7973","reported_seq":8057,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.471967+0000","last_change":"2024-03-26T03:16:56.907700+0000","last_active":"2024-03-26T03:18:34.471967+0000","last_peered":"2024-03-26T03:18:34.471967+0000","last_clean":"2024-03-26T03:18:34.471967+0000","last_became_active":"2024-03-26T03:16:56.907183+0000","last_became_peered":"2024-03-26T03:16:56.907183+0000","last_unstale":"2024-03-26T03:18:34.471967+0000","last_undegraded":"2024-03-26T03:18:34.471967+0000","last_fullsized":"2024-03-26T03:18:34.471967+0000","mapping_epoch":403,"log_start":"426'4800","ondisk_log_start":"426'4800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":2387,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522518528,"num_objects":7973,"num_object_clones":0,"num_object_copies":23919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":849,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.584997+0000","last_change":"2024-03-26T03:17:42.743976+0000","last_active":"2024-03-26T03:18:33.584997+0000","last_peered":"2024-03-26T03:18:33.584997+0000","last_clean":"2024-03-26T03:18:33.584997+0000","last_became_active":"2024-03-26T03:13:35.271824+0000","last_became_peered":"2024-03-26T03:13:35.271824+0000","last_unstale":"2024-03-26T03:18:33.584997+0000","last_undegraded":"2024-03-26T03:18:33.584997+0000","last_fullsized":"2024-03-26T03:18:33.584997+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:42.743853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.743853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:57.009889+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":723,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600590+0000","last_change":"2024-03-26T03:14:37.277935+0000","last_active":"2024-03-26T03:18:33.600590+0000","last_peered":"2024-03-26T03:18:33.600590+0000","last_clean":"2024-03-26T03:18:33.600590+0000","last_became_active":"2024-03-26T03:14:37.246803+0000","last_became_peered":"2024-03-26T03:14:37.246803+0000","last_unstale":"2024-03-26T03:18:33.600590+0000","last_undegraded":"2024-03-26T03:18:33.600590+0000","last_fullsized":"2024-03-26T03:18:33.600590+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:13:15.467975+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.467975+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":753,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598706+0000","last_change":"2024-03-26T03:16:56.019977+0000","last_active":"2024-03-26T03:18:33.598706+0000","last_peered":"2024-03-26T03:18:33.598706+0000","last_clean":"2024-03-26T03:18:33.598706+0000","last_became_active":"2024-03-26T03:14:37.265520+0000","last_became_peered":"2024-03-26T03:14:37.265520+0000","last_unstale":"2024-03-26T03:18:33.598706+0000","last_undegraded":"2024-03-26T03:18:33.598706+0000","last_fullsized":"2024-03-26T03:18:33.598706+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:56.019845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.019845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597351+0000","last_change":"2024-03-26T03:17:56.568202+0000","last_active":"2024-03-26T03:18:33.597351+0000","last_peered":"2024-03-26T03:18:33.597351+0000","last_clean":"2024-03-26T03:18:33.597351+0000","last_became_active":"2024-03-26T03:17:56.567590+0000","last_became_peered":"2024-03-26T03:17:56.567590+0000","last_unstale":"2024-03-26T03:18:33.597351+0000","last_undegraded":"2024-03-26T03:18:33.597351+0000","last_fullsized":"2024-03-26T03:18:33.597351+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":801,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.606616+0000","last_change":"2024-03-26T03:15:17.827310+0000","last_active":"2024-03-26T03:18:33.606616+0000","last_peered":"2024-03-26T03:18:33.606616+0000","last_clean":"2024-03-26T03:18:33.606616+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:18:33.606616+0000","last_undegraded":"2024-03-26T03:18:33.606616+0000","last_fullsized":"2024-03-26T03:18:33.606616+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:17.827186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:15:17.827186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":940,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600578+0000","last_change":"2024-03-26T03:17:59.603321+0000","last_active":"2024-03-26T03:18:33.600578+0000","last_peered":"2024-03-26T03:18:33.600578+0000","last_clean":"2024-03-26T03:18:33.600578+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:18:33.600578+0000","last_undegraded":"2024-03-26T03:18:33.600578+0000","last_fullsized":"2024-03-26T03:18:33.600578+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:43.863351+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:43.863351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"440'7986","reported_seq":8069,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.488261+0000","last_change":"2024-03-26T03:16:56.907179+0000","last_active":"2024-03-26T03:18:34.488261+0000","last_peered":"2024-03-26T03:18:34.488261+0000","last_clean":"2024-03-26T03:18:34.488261+0000","last_became_active":"2024-03-26T03:16:56.906797+0000","last_became_peered":"2024-03-26T03:16:56.906797+0000","last_unstale":"2024-03-26T03:18:34.488261+0000","last_undegraded":"2024-03-26T03:18:34.488261+0000","last_fullsized":"2024-03-26T03:18:34.488261+0000","mapping_epoch":403,"log_start":"426'4800","ondisk_log_start":"426'4800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":2387,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523370496,"num_objects":7986,"num_object_clones":0,"num_object_copies":23958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":511104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":92,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.603880+0000","last_change":"2024-03-26T03:16:50.209616+0000","last_active":"2024-03-26T03:18:33.603880+0000","last_peered":"2024-03-26T03:18:33.603880+0000","last_clean":"2024-03-26T03:18:33.603880+0000","last_became_active":"2024-03-26T03:16:50.208883+0000","last_became_peered":"2024-03-26T03:16:50.208883+0000","last_unstale":"2024-03-26T03:18:33.603880+0000","last_undegraded":"2024-03-26T03:18:33.603880+0000","last_fullsized":"2024-03-26T03:18:33.603880+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.231346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.231346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"342'29","reported_seq":1132,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.591282+0000","last_change":"2024-03-26T03:16:54.697496+0000","last_active":"2024-03-26T03:18:33.591282+0000","last_peered":"2024-03-26T03:18:33.591282+0000","last_clean":"2024-03-26T03:18:33.591282+0000","last_became_active":"2024-03-26T03:14:37.265178+0000","last_became_peered":"2024-03-26T03:14:37.265178+0000","last_unstale":"2024-03-26T03:18:33.591282+0000","last_undegraded":"2024-03-26T03:18:33.591282+0000","last_fullsized":"2024-03-26T03:18:33.591282+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'29","last_scrub_stamp":"2024-03-26T03:16:54.697359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:54.697359+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":512,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.588847+0000","last_change":"2024-03-26T03:16:31.804230+0000","last_active":"2024-03-26T03:18:33.588847+0000","last_peered":"2024-03-26T03:18:33.588847+0000","last_clean":"2024-03-26T03:18:33.588847+0000","last_became_active":"2024-03-26T03:14:40.180729+0000","last_became_peered":"2024-03-26T03:14:40.180729+0000","last_unstale":"2024-03-26T03:18:33.588847+0000","last_undegraded":"2024-03-26T03:18:33.588847+0000","last_fullsized":"2024-03-26T03:18:33.588847+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.804125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.804125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":871,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.586781+0000","last_change":"2024-03-26T03:17:52.570959+0000","last_active":"2024-03-26T03:18:33.586781+0000","last_peered":"2024-03-26T03:18:33.586781+0000","last_clean":"2024-03-26T03:18:33.586781+0000","last_became_active":"2024-03-26T03:14:37.274568+0000","last_became_peered":"2024-03-26T03:14:37.274568+0000","last_unstale":"2024-03-26T03:18:33.586781+0000","last_undegraded":"2024-03-26T03:18:33.586781+0000","last_fullsized":"2024-03-26T03:18:33.586781+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.570835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.570835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:19.356483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":507,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598668+0000","last_change":"2024-03-26T03:14:21.657126+0000","last_active":"2024-03-26T03:18:33.598668+0000","last_peered":"2024-03-26T03:18:33.598668+0000","last_clean":"2024-03-26T03:18:33.598668+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:18:33.598668+0000","last_undegraded":"2024-03-26T03:18:33.598668+0000","last_fullsized":"2024-03-26T03:18:33.598668+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":808,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.585085+0000","last_change":"2024-03-26T03:17:41.822116+0000","last_active":"2024-03-26T03:18:33.585085+0000","last_peered":"2024-03-26T03:18:33.585085+0000","last_clean":"2024-03-26T03:18:33.585085+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:18:33.585085+0000","last_undegraded":"2024-03-26T03:18:33.585085+0000","last_fullsized":"2024-03-26T03:18:33.585085+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.822032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.822032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:06.248445+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"440'8080","reported_seq":8164,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.478481+0000","last_change":"2024-03-26T03:16:56.908716+0000","last_active":"2024-03-26T03:18:34.478481+0000","last_peered":"2024-03-26T03:18:34.478481+0000","last_clean":"2024-03-26T03:18:34.478481+0000","last_became_active":"2024-03-26T03:16:56.908005+0000","last_became_peered":"2024-03-26T03:16:56.908005+0000","last_unstale":"2024-03-26T03:18:34.478481+0000","last_undegraded":"2024-03-26T03:18:34.478481+0000","last_fullsized":"2024-03-26T03:18:34.478481+0000","mapping_epoch":403,"log_start":"426'4900","ondisk_log_start":"426'4900","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3180,"log_dups_size":2487,"ondisk_log_size":3180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529530880,"num_objects":8080,"num_object_clones":0,"num_object_copies":24240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8080,"num_write_kb":517120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":725,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.586029+0000","last_change":"2024-03-26T03:16:52.721328+0000","last_active":"2024-03-26T03:18:33.586029+0000","last_peered":"2024-03-26T03:18:33.586029+0000","last_clean":"2024-03-26T03:18:33.586029+0000","last_became_active":"2024-03-26T03:16:50.206619+0000","last_became_peered":"2024-03-26T03:16:50.206619+0000","last_unstale":"2024-03-26T03:18:33.586029+0000","last_undegraded":"2024-03-26T03:18:33.586029+0000","last_fullsized":"2024-03-26T03:18:33.586029+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.721247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":909,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.602050+0000","last_change":"2024-03-26T03:14:37.294831+0000","last_active":"2024-03-26T03:18:33.602050+0000","last_peered":"2024-03-26T03:18:33.602050+0000","last_clean":"2024-03-26T03:18:33.602050+0000","last_became_active":"2024-03-26T03:14:37.272426+0000","last_became_peered":"2024-03-26T03:14:37.272426+0000","last_unstale":"2024-03-26T03:18:33.602050+0000","last_undegraded":"2024-03-26T03:18:33.602050+0000","last_fullsized":"2024-03-26T03:18:33.602050+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:13:23.013434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:23.013434+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":674,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593558+0000","last_change":"2024-03-26T03:16:50.211076+0000","last_active":"2024-03-26T03:18:33.593558+0000","last_peered":"2024-03-26T03:18:33.593558+0000","last_clean":"2024-03-26T03:18:33.593558+0000","last_became_active":"2024-03-26T03:16:50.210676+0000","last_became_peered":"2024-03-26T03:16:50.210676+0000","last_unstale":"2024-03-26T03:18:33.593558+0000","last_undegraded":"2024-03-26T03:18:33.593558+0000","last_fullsized":"2024-03-26T03:18:33.593558+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:41.390700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:41.390700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2013,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.048967+0000","last_change":"2024-03-26T03:13:31.075605+0000","last_active":"2024-03-26T03:18:34.048967+0000","last_peered":"2024-03-26T03:18:34.048967+0000","last_clean":"2024-03-26T03:18:34.048967+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:18:34.048967+0000","last_undegraded":"2024-03-26T03:18:34.048967+0000","last_fullsized":"2024-03-26T03:18:34.048967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:13:31.074419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.074419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1167,"num_read_kb":1162,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":783,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.591068+0000","last_change":"2024-03-26T03:16:37.826030+0000","last_active":"2024-03-26T03:18:33.591068+0000","last_peered":"2024-03-26T03:18:33.591068+0000","last_clean":"2024-03-26T03:18:33.591068+0000","last_became_active":"2024-03-26T03:14:37.373684+0000","last_became_peered":"2024-03-26T03:14:37.373684+0000","last_unstale":"2024-03-26T03:18:33.591068+0000","last_undegraded":"2024-03-26T03:18:33.591068+0000","last_fullsized":"2024-03-26T03:18:33.591068+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.825906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.825906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834377+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":876,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597929+0000","last_change":"2024-03-26T03:17:59.594483+0000","last_active":"2024-03-26T03:18:33.597929+0000","last_peered":"2024-03-26T03:18:33.597929+0000","last_clean":"2024-03-26T03:18:33.597929+0000","last_became_active":"2024-03-26T03:17:59.594123+0000","last_became_peered":"2024-03-26T03:17:59.594123+0000","last_unstale":"2024-03-26T03:18:33.597929+0000","last_undegraded":"2024-03-26T03:18:33.597929+0000","last_fullsized":"2024-03-26T03:18:33.597929+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:50.701574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.701574+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.b","version":"440'7959","reported_seq":18435,"reported_epoch":440,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:18:34.487088+0000","last_change":"2024-03-26T03:17:57.696694+0000","last_active":"2024-03-26T03:18:34.487088+0000","last_peered":"2024-03-26T03:18:34.487088+0000","last_clean":"2024-03-26T03:17:55.541263+0000","last_became_active":"2024-03-26T03:17:57.679084+0000","last_became_peered":"2024-03-26T03:17:57.679084+0000","last_unstale":"2024-03-26T03:18:34.487088+0000","last_undegraded":"2024-03-26T03:18:34.487088+0000","last_fullsized":"2024-03-26T03:18:34.487088+0000","mapping_epoch":428,"log_start":"426'4800","ondisk_log_start":"426'4800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":2387,"ondisk_log_size":3159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521601024,"num_objects":7959,"num_object_clones":0,"num_object_copies":23877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1533,"num_objects_unfound":0,"num_objects_dirty":7959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7959,"num_write_kb":509376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5193,"num_bytes_recovered":340328448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":7959}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":278,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601341+0000","last_change":"2024-03-26T03:17:56.570342+0000","last_active":"2024-03-26T03:18:33.601341+0000","last_peered":"2024-03-26T03:18:33.601341+0000","last_clean":"2024-03-26T03:18:33.601341+0000","last_became_active":"2024-03-26T03:17:56.570006+0000","last_became_peered":"2024-03-26T03:17:56.570006+0000","last_unstale":"2024-03-26T03:18:33.601341+0000","last_undegraded":"2024-03-26T03:18:33.601341+0000","last_fullsized":"2024-03-26T03:18:33.601341+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:17.442291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.442291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"332'59","reported_seq":1150,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.604169+0000","last_change":"2024-03-26T03:15:12.789448+0000","last_active":"2024-03-26T03:18:33.604169+0000","last_peered":"2024-03-26T03:18:33.604169+0000","last_clean":"2024-03-26T03:18:33.604169+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:18:33.604169+0000","last_undegraded":"2024-03-26T03:18:33.604169+0000","last_fullsized":"2024-03-26T03:18:33.604169+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'59","last_scrub_stamp":"2024-03-26T03:15:12.789148+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:15:12.789148+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":628,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601839+0000","last_change":"2024-03-26T03:16:50.962056+0000","last_active":"2024-03-26T03:18:33.601839+0000","last_peered":"2024-03-26T03:18:33.601839+0000","last_clean":"2024-03-26T03:18:33.601839+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:18:33.601839+0000","last_undegraded":"2024-03-26T03:18:33.601839+0000","last_fullsized":"2024-03-26T03:18:33.601839+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:50.961975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.961975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":518,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595439+0000","last_change":"2024-03-26T03:14:37.276021+0000","last_active":"2024-03-26T03:18:33.595439+0000","last_peered":"2024-03-26T03:18:33.595439+0000","last_clean":"2024-03-26T03:18:33.595439+0000","last_became_active":"2024-03-26T03:14:37.274135+0000","last_became_peered":"2024-03-26T03:14:37.274135+0000","last_unstale":"2024-03-26T03:18:33.595439+0000","last_undegraded":"2024-03-26T03:18:33.595439+0000","last_fullsized":"2024-03-26T03:18:33.595439+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":553,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597468+0000","last_change":"2024-03-26T03:16:30.089444+0000","last_active":"2024-03-26T03:18:33.597468+0000","last_peered":"2024-03-26T03:18:33.597468+0000","last_clean":"2024-03-26T03:18:33.597468+0000","last_became_active":"2024-03-26T03:13:56.449984+0000","last_became_peered":"2024-03-26T03:13:56.449984+0000","last_unstale":"2024-03-26T03:18:33.597468+0000","last_undegraded":"2024-03-26T03:18:33.597468+0000","last_fullsized":"2024-03-26T03:18:33.597468+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.089341+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.089341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.d","version":"440'8149","reported_seq":8233,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:34.706271+0000","last_change":"2024-03-26T03:16:56.901084+0000","last_active":"2024-03-26T03:18:34.706271+0000","last_peered":"2024-03-26T03:18:34.706271+0000","last_clean":"2024-03-26T03:18:34.706271+0000","last_became_active":"2024-03-26T03:16:56.900700+0000","last_became_peered":"2024-03-26T03:16:56.900700+0000","last_unstale":"2024-03-26T03:18:34.706271+0000","last_undegraded":"2024-03-26T03:18:34.706271+0000","last_fullsized":"2024-03-26T03:18:34.706271+0000","mapping_epoch":403,"log_start":"426'4691","ondisk_log_start":"426'4691","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534052864,"num_objects":8149,"num_object_clones":0,"num_object_copies":24447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8149,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"333'12","reported_seq":372,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589445+0000","last_change":"2024-03-26T03:17:42.576855+0000","last_active":"2024-03-26T03:18:33.589445+0000","last_peered":"2024-03-26T03:18:33.589445+0000","last_clean":"2024-03-26T03:18:33.589445+0000","last_became_active":"2024-03-26T03:14:22.574243+0000","last_became_peered":"2024-03-26T03:14:22.574243+0000","last_unstale":"2024-03-26T03:18:33.589445+0000","last_undegraded":"2024-03-26T03:18:33.589445+0000","last_fullsized":"2024-03-26T03:18:33.589445+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'12","last_scrub_stamp":"2024-03-26T03:17:42.576771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.576771+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:08.322731+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":915,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.603051+0000","last_change":"2024-03-26T03:17:56.581987+0000","last_active":"2024-03-26T03:18:33.603051+0000","last_peered":"2024-03-26T03:18:33.603051+0000","last_clean":"2024-03-26T03:18:33.603051+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:18:33.603051+0000","last_undegraded":"2024-03-26T03:18:33.603051+0000","last_fullsized":"2024-03-26T03:18:33.603051+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:16:40.828476+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.828476+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":840,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.604293+0000","last_change":"2024-03-26T03:14:37.252086+0000","last_active":"2024-03-26T03:18:33.604293+0000","last_peered":"2024-03-26T03:18:33.604293+0000","last_clean":"2024-03-26T03:18:33.604293+0000","last_became_active":"2024-03-26T03:14:37.251746+0000","last_became_peered":"2024-03-26T03:14:37.251746+0000","last_unstale":"2024-03-26T03:18:33.604293+0000","last_undegraded":"2024-03-26T03:18:33.604293+0000","last_fullsized":"2024-03-26T03:18:33.604293+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:19.981413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.981413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":230,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593228+0000","last_change":"2024-03-26T03:14:22.567168+0000","last_active":"2024-03-26T03:18:33.593228+0000","last_peered":"2024-03-26T03:18:33.593228+0000","last_clean":"2024-03-26T03:18:33.593228+0000","last_became_active":"2024-03-26T03:14:22.566823+0000","last_became_peered":"2024-03-26T03:14:22.566823+0000","last_unstale":"2024-03-26T03:18:33.593228+0000","last_undegraded":"2024-03-26T03:18:33.593228+0000","last_fullsized":"2024-03-26T03:18:33.593228+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:24.614199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:05:24.614199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":905,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590052+0000","last_change":"2024-03-26T03:16:30.838677+0000","last_active":"2024-03-26T03:18:33.590052+0000","last_peered":"2024-03-26T03:18:33.590052+0000","last_clean":"2024-03-26T03:18:33.590052+0000","last_became_active":"2024-03-26T03:14:40.178828+0000","last_became_peered":"2024-03-26T03:14:40.178828+0000","last_unstale":"2024-03-26T03:18:33.590052+0000","last_undegraded":"2024-03-26T03:18:33.590052+0000","last_fullsized":"2024-03-26T03:18:33.590052+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.838578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.838578+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"440'8276","reported_seq":8360,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:35.544805+0000","last_change":"2024-03-26T03:16:56.899075+0000","last_active":"2024-03-26T03:18:35.544805+0000","last_peered":"2024-03-26T03:18:35.544805+0000","last_clean":"2024-03-26T03:18:35.544805+0000","last_became_active":"2024-03-26T03:16:56.898139+0000","last_became_peered":"2024-03-26T03:16:56.898139+0000","last_unstale":"2024-03-26T03:18:35.544805+0000","last_undegraded":"2024-03-26T03:18:35.544805+0000","last_fullsized":"2024-03-26T03:18:35.544805+0000","mapping_epoch":403,"log_start":"426'5000","ondisk_log_start":"426'5000","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542375936,"num_objects":8276,"num_object_clones":0,"num_object_copies":24828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":529664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":629,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594139+0000","last_change":"2024-03-26T03:17:59.595756+0000","last_active":"2024-03-26T03:18:33.594139+0000","last_peered":"2024-03-26T03:18:33.594139+0000","last_clean":"2024-03-26T03:18:33.594139+0000","last_became_active":"2024-03-26T03:17:59.594586+0000","last_became_peered":"2024-03-26T03:17:59.594586+0000","last_unstale":"2024-03-26T03:18:33.594139+0000","last_undegraded":"2024-03-26T03:18:33.594139+0000","last_fullsized":"2024-03-26T03:18:33.594139+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.474559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.474559+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":676,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598374+0000","last_change":"2024-03-26T03:17:53.556662+0000","last_active":"2024-03-26T03:18:33.598374+0000","last_peered":"2024-03-26T03:18:33.598374+0000","last_clean":"2024-03-26T03:18:33.598374+0000","last_became_active":"2024-03-26T03:16:50.208194+0000","last_became_peered":"2024-03-26T03:16:50.208194+0000","last_unstale":"2024-03-26T03:18:33.598374+0000","last_undegraded":"2024-03-26T03:18:33.598374+0000","last_fullsized":"2024-03-26T03:18:33.598374+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:53.556429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:53.556429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.094999+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":886,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598254+0000","last_change":"2024-03-26T03:14:37.290458+0000","last_active":"2024-03-26T03:18:33.598254+0000","last_peered":"2024-03-26T03:18:33.598254+0000","last_clean":"2024-03-26T03:18:33.598254+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:18:33.598254+0000","last_undegraded":"2024-03-26T03:18:33.598254+0000","last_fullsized":"2024-03-26T03:18:33.598254+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:14:37.290313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:14:37.290313+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":893,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598883+0000","last_change":"2024-03-26T03:17:37.107175+0000","last_active":"2024-03-26T03:18:33.598883+0000","last_peered":"2024-03-26T03:18:33.598883+0000","last_clean":"2024-03-26T03:18:33.598883+0000","last_became_active":"2024-03-26T03:14:37.249021+0000","last_became_peered":"2024-03-26T03:14:37.249021+0000","last_unstale":"2024-03-26T03:18:33.598883+0000","last_undegraded":"2024-03-26T03:18:33.598883+0000","last_fullsized":"2024-03-26T03:18:33.598883+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:17:37.106951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:37.106951+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:53.197468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":490,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596361+0000","last_change":"2024-03-26T03:14:37.265797+0000","last_active":"2024-03-26T03:18:33.596361+0000","last_peered":"2024-03-26T03:18:33.596361+0000","last_clean":"2024-03-26T03:18:33.596361+0000","last_became_active":"2024-03-26T03:14:37.265041+0000","last_became_peered":"2024-03-26T03:14:37.265041+0000","last_unstale":"2024-03-26T03:18:33.596361+0000","last_undegraded":"2024-03-26T03:18:33.596361+0000","last_fullsized":"2024-03-26T03:18:33.596361+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":952,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600332+0000","last_change":"2024-03-26T03:17:59.607666+0000","last_active":"2024-03-26T03:18:33.600332+0000","last_peered":"2024-03-26T03:18:33.600332+0000","last_clean":"2024-03-26T03:18:33.600332+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:18:33.600332+0000","last_undegraded":"2024-03-26T03:18:33.600332+0000","last_fullsized":"2024-03-26T03:18:33.600332+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:44.879010+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.879010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":315,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595343+0000","last_change":"2024-03-26T03:14:37.323020+0000","last_active":"2024-03-26T03:18:33.595343+0000","last_peered":"2024-03-26T03:18:33.595343+0000","last_clean":"2024-03-26T03:18:33.595343+0000","last_became_active":"2024-03-26T03:14:37.269153+0000","last_became_peered":"2024-03-26T03:14:37.269153+0000","last_unstale":"2024-03-26T03:18:33.595343+0000","last_undegraded":"2024-03-26T03:18:33.595343+0000","last_fullsized":"2024-03-26T03:18:33.595343+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":108,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598859+0000","last_change":"2024-03-26T03:17:56.578376+0000","last_active":"2024-03-26T03:18:33.598859+0000","last_peered":"2024-03-26T03:18:33.598859+0000","last_clean":"2024-03-26T03:18:33.598859+0000","last_became_active":"2024-03-26T03:17:56.571085+0000","last_became_peered":"2024-03-26T03:17:56.571085+0000","last_unstale":"2024-03-26T03:18:33.598859+0000","last_undegraded":"2024-03-26T03:18:33.598859+0000","last_fullsized":"2024-03-26T03:18:33.598859+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:16:49.187353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.187353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":510,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596817+0000","last_change":"2024-03-26T03:14:37.274074+0000","last_active":"2024-03-26T03:18:33.596817+0000","last_peered":"2024-03-26T03:18:33.596817+0000","last_clean":"2024-03-26T03:18:33.596817+0000","last_became_active":"2024-03-26T03:14:37.273658+0000","last_became_peered":"2024-03-26T03:14:37.273658+0000","last_unstale":"2024-03-26T03:18:33.596817+0000","last_undegraded":"2024-03-26T03:18:33.596817+0000","last_fullsized":"2024-03-26T03:18:33.596817+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":720,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598896+0000","last_change":"2024-03-26T03:17:55.099141+0000","last_active":"2024-03-26T03:18:33.598896+0000","last_peered":"2024-03-26T03:18:33.598896+0000","last_clean":"2024-03-26T03:18:33.598896+0000","last_became_active":"2024-03-26T03:16:50.205905+0000","last_became_peered":"2024-03-26T03:16:50.205905+0000","last_unstale":"2024-03-26T03:18:33.598896+0000","last_undegraded":"2024-03-26T03:18:33.598896+0000","last_fullsized":"2024-03-26T03:18:33.598896+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:55.099060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.099060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:17.587668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":347,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597876+0000","last_change":"2024-03-26T03:17:45.079338+0000","last_active":"2024-03-26T03:18:33.597876+0000","last_peered":"2024-03-26T03:18:33.597876+0000","last_clean":"2024-03-26T03:18:33.597876+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:18:33.597876+0000","last_undegraded":"2024-03-26T03:18:33.597876+0000","last_fullsized":"2024-03-26T03:18:33.597876+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:45.079209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.079209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:07.973854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":583,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596596+0000","last_change":"2024-03-26T03:16:40.927377+0000","last_active":"2024-03-26T03:18:33.596596+0000","last_peered":"2024-03-26T03:18:33.596596+0000","last_clean":"2024-03-26T03:18:33.596596+0000","last_became_active":"2024-03-26T03:14:37.269408+0000","last_became_peered":"2024-03-26T03:14:37.269408+0000","last_unstale":"2024-03-26T03:18:33.596596+0000","last_undegraded":"2024-03-26T03:18:33.596596+0000","last_fullsized":"2024-03-26T03:18:33.596596+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'18","last_scrub_stamp":"2024-03-26T03:16:40.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.927260+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":847,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589131+0000","last_change":"2024-03-26T03:17:56.570470+0000","last_active":"2024-03-26T03:18:33.589131+0000","last_peered":"2024-03-26T03:18:33.589131+0000","last_clean":"2024-03-26T03:18:33.589131+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:18:33.589131+0000","last_undegraded":"2024-03-26T03:18:33.589131+0000","last_fullsized":"2024-03-26T03:18:33.589131+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:59.707108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.707108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":811,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.585854+0000","last_change":"2024-03-26T03:17:44.697333+0000","last_active":"2024-03-26T03:18:33.585854+0000","last_peered":"2024-03-26T03:18:33.585854+0000","last_clean":"2024-03-26T03:18:33.585854+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:18:33.585854+0000","last_undegraded":"2024-03-26T03:18:33.585854+0000","last_fullsized":"2024-03-26T03:18:33.585854+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.697255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.697255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.820334+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":217,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.591163+0000","last_change":"2024-03-26T03:16:45.758249+0000","last_active":"2024-03-26T03:18:33.591163+0000","last_peered":"2024-03-26T03:18:33.591163+0000","last_clean":"2024-03-26T03:18:33.591163+0000","last_became_active":"2024-03-26T03:14:37.333430+0000","last_became_peered":"2024-03-26T03:14:37.333430+0000","last_unstale":"2024-03-26T03:18:33.591163+0000","last_undegraded":"2024-03-26T03:18:33.591163+0000","last_fullsized":"2024-03-26T03:18:33.591163+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.758140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.758140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834402+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":588,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.583387+0000","last_change":"2024-03-26T03:17:40.727303+0000","last_active":"2024-03-26T03:18:33.583387+0000","last_peered":"2024-03-26T03:18:33.583387+0000","last_clean":"2024-03-26T03:18:33.583387+0000","last_became_active":"2024-03-26T03:14:24.567937+0000","last_became_peered":"2024-03-26T03:14:24.567937+0000","last_unstale":"2024-03-26T03:18:33.583387+0000","last_undegraded":"2024-03-26T03:18:33.583387+0000","last_fullsized":"2024-03-26T03:18:33.583387+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:40.727084+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.727084+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:03.188531+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":600,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594633+0000","last_change":"2024-03-26T03:16:49.904794+0000","last_active":"2024-03-26T03:18:33.594633+0000","last_peered":"2024-03-26T03:18:33.594633+0000","last_clean":"2024-03-26T03:18:33.594633+0000","last_became_active":"2024-03-26T03:14:37.265466+0000","last_became_peered":"2024-03-26T03:14:37.265466+0000","last_unstale":"2024-03-26T03:18:33.594633+0000","last_undegraded":"2024-03-26T03:18:33.594633+0000","last_fullsized":"2024-03-26T03:18:33.594633+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.904637+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":845,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.584723+0000","last_change":"2024-03-26T03:17:56.569921+0000","last_active":"2024-03-26T03:18:33.584723+0000","last_peered":"2024-03-26T03:18:33.584723+0000","last_clean":"2024-03-26T03:18:33.584723+0000","last_became_active":"2024-03-26T03:17:56.569537+0000","last_became_peered":"2024-03-26T03:17:56.569537+0000","last_unstale":"2024-03-26T03:18:33.584723+0000","last_undegraded":"2024-03-26T03:18:33.584723+0000","last_fullsized":"2024-03-26T03:18:33.584723+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:48.651379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.651379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:01.106930+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"333'13","reported_seq":1103,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.584490+0000","last_change":"2024-03-26T03:18:09.760752+0000","last_active":"2024-03-26T03:18:33.584490+0000","last_peered":"2024-03-26T03:18:33.584490+0000","last_clean":"2024-03-26T03:18:33.584490+0000","last_became_active":"2024-03-26T03:14:22.577229+0000","last_became_peered":"2024-03-26T03:14:22.577229+0000","last_unstale":"2024-03-26T03:18:33.584490+0000","last_undegraded":"2024-03-26T03:18:33.584490+0000","last_fullsized":"2024-03-26T03:18:33.584490+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'13","last_scrub_stamp":"2024-03-26T03:18:09.760609+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:18:09.760609+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:30.051655+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":547,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594338+0000","last_change":"2024-03-26T03:14:37.275365+0000","last_active":"2024-03-26T03:18:33.594338+0000","last_peered":"2024-03-26T03:18:33.594338+0000","last_clean":"2024-03-26T03:18:33.594338+0000","last_became_active":"2024-03-26T03:14:37.273988+0000","last_became_peered":"2024-03-26T03:14:37.273988+0000","last_unstale":"2024-03-26T03:18:33.594338+0000","last_undegraded":"2024-03-26T03:18:33.594338+0000","last_fullsized":"2024-03-26T03:18:33.594338+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":512,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.588814+0000","last_change":"2024-03-26T03:16:28.906151+0000","last_active":"2024-03-26T03:18:33.588814+0000","last_peered":"2024-03-26T03:18:33.588814+0000","last_clean":"2024-03-26T03:18:33.588814+0000","last_became_active":"2024-03-26T03:14:40.173243+0000","last_became_peered":"2024-03-26T03:14:40.173243+0000","last_unstale":"2024-03-26T03:18:33.588814+0000","last_undegraded":"2024-03-26T03:18:33.588814+0000","last_fullsized":"2024-03-26T03:18:33.588814+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:28.906044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.906044+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":648,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593898+0000","last_change":"2024-03-26T03:16:59.931429+0000","last_active":"2024-03-26T03:18:33.593898+0000","last_peered":"2024-03-26T03:18:33.593898+0000","last_clean":"2024-03-26T03:18:33.593898+0000","last_became_active":"2024-03-26T03:14:37.333426+0000","last_became_peered":"2024-03-26T03:14:37.333426+0000","last_unstale":"2024-03-26T03:18:33.593898+0000","last_undegraded":"2024-03-26T03:18:33.593898+0000","last_fullsized":"2024-03-26T03:18:33.593898+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'21","last_scrub_stamp":"2024-03-26T03:16:59.931283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.931283+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":581,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598786+0000","last_change":"2024-03-26T03:16:29.133272+0000","last_active":"2024-03-26T03:18:33.598786+0000","last_peered":"2024-03-26T03:18:33.598786+0000","last_clean":"2024-03-26T03:18:33.598786+0000","last_became_active":"2024-03-26T03:14:37.262766+0000","last_became_peered":"2024-03-26T03:14:37.262766+0000","last_unstale":"2024-03-26T03:18:33.598786+0000","last_undegraded":"2024-03-26T03:18:33.598786+0000","last_fullsized":"2024-03-26T03:18:33.598786+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:29.133174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.133174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"342'11","reported_seq":609,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595103+0000","last_change":"2024-03-26T03:17:50.265634+0000","last_active":"2024-03-26T03:18:33.595103+0000","last_peered":"2024-03-26T03:18:33.595103+0000","last_clean":"2024-03-26T03:18:33.595103+0000","last_became_active":"2024-03-26T03:14:37.264759+0000","last_became_peered":"2024-03-26T03:14:37.264759+0000","last_unstale":"2024-03-26T03:18:33.595103+0000","last_undegraded":"2024-03-26T03:18:33.595103+0000","last_fullsized":"2024-03-26T03:18:33.595103+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'11","last_scrub_stamp":"2024-03-26T03:17:50.265404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:50.265404+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:50.629014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":814,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599380+0000","last_change":"2024-03-26T03:17:48.227168+0000","last_active":"2024-03-26T03:18:33.599380+0000","last_peered":"2024-03-26T03:18:33.599380+0000","last_clean":"2024-03-26T03:18:33.599380+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:18:33.599380+0000","last_undegraded":"2024-03-26T03:18:33.599380+0000","last_fullsized":"2024-03-26T03:18:33.599380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:17:48.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.226947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.196614+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":925,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.584114+0000","last_change":"2024-03-26T03:17:55.577343+0000","last_active":"2024-03-26T03:18:33.584114+0000","last_peered":"2024-03-26T03:18:33.584114+0000","last_clean":"2024-03-26T03:18:33.584114+0000","last_became_active":"2024-03-26T03:14:40.170537+0000","last_became_peered":"2024-03-26T03:14:40.170537+0000","last_unstale":"2024-03-26T03:18:33.584114+0000","last_undegraded":"2024-03-26T03:18:33.584114+0000","last_fullsized":"2024-03-26T03:18:33.584114+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:55.577263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.577263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:21.650938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":852,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.602170+0000","last_change":"2024-03-26T03:13:33.092136+0000","last_active":"2024-03-26T03:18:33.602170+0000","last_peered":"2024-03-26T03:18:33.602170+0000","last_clean":"2024-03-26T03:18:33.602170+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:18:33.602170+0000","last_undegraded":"2024-03-26T03:18:33.602170+0000","last_fullsized":"2024-03-26T03:18:33.602170+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:13:33.091975+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:13:33.091975+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":812,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589699+0000","last_change":"2024-03-26T03:16:33.748393+0000","last_active":"2024-03-26T03:18:33.589699+0000","last_peered":"2024-03-26T03:18:33.589699+0000","last_clean":"2024-03-26T03:18:33.589699+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:18:33.589699+0000","last_undegraded":"2024-03-26T03:18:33.589699+0000","last_fullsized":"2024-03-26T03:18:33.589699+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:33.748295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:33.748295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834656+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":501,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596167+0000","last_change":"2024-03-26T03:16:44.909663+0000","last_active":"2024-03-26T03:18:33.596167+0000","last_peered":"2024-03-26T03:18:33.596167+0000","last_clean":"2024-03-26T03:18:33.596167+0000","last_became_active":"2024-03-26T03:14:22.573232+0000","last_became_peered":"2024-03-26T03:14:22.573232+0000","last_unstale":"2024-03-26T03:18:33.596167+0000","last_undegraded":"2024-03-26T03:18:33.596167+0000","last_fullsized":"2024-03-26T03:18:33.596167+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.909523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.909523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":275,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598462+0000","last_change":"2024-03-26T03:17:35.405461+0000","last_active":"2024-03-26T03:18:33.598462+0000","last_peered":"2024-03-26T03:18:33.598462+0000","last_clean":"2024-03-26T03:18:33.598462+0000","last_became_active":"2024-03-26T03:13:35.278675+0000","last_became_peered":"2024-03-26T03:13:35.278675+0000","last_unstale":"2024-03-26T03:18:33.598462+0000","last_undegraded":"2024-03-26T03:18:33.598462+0000","last_fullsized":"2024-03-26T03:18:33.598462+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.405139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.405139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:00.124612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":593,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600251+0000","last_change":"2024-03-26T03:14:22.578069+0000","last_active":"2024-03-26T03:18:33.600251+0000","last_peered":"2024-03-26T03:18:33.600251+0000","last_clean":"2024-03-26T03:18:33.600251+0000","last_became_active":"2024-03-26T03:14:22.577614+0000","last_became_peered":"2024-03-26T03:14:22.577614+0000","last_unstale":"2024-03-26T03:18:33.600251+0000","last_undegraded":"2024-03-26T03:18:33.600251+0000","last_fullsized":"2024-03-26T03:18:33.600251+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.940687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.940687+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":852,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598641+0000","last_change":"2024-03-26T03:16:53.170995+0000","last_active":"2024-03-26T03:18:33.598641+0000","last_peered":"2024-03-26T03:18:33.598641+0000","last_clean":"2024-03-26T03:18:33.598641+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:18:33.598641+0000","last_undegraded":"2024-03-26T03:18:33.598641+0000","last_fullsized":"2024-03-26T03:18:33.598641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:16:53.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:53.170849+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":849,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601498+0000","last_change":"2024-03-26T03:17:34.417755+0000","last_active":"2024-03-26T03:18:33.601498+0000","last_peered":"2024-03-26T03:18:33.601498+0000","last_clean":"2024-03-26T03:18:33.601498+0000","last_became_active":"2024-03-26T03:13:35.274830+0000","last_became_peered":"2024-03-26T03:13:35.274830+0000","last_unstale":"2024-03-26T03:18:33.601498+0000","last_undegraded":"2024-03-26T03:18:33.601498+0000","last_fullsized":"2024-03-26T03:18:33.601498+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.417610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.417610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:39.383950+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":796,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.603171+0000","last_change":"2024-03-26T03:15:14.862264+0000","last_active":"2024-03-26T03:18:33.603171+0000","last_peered":"2024-03-26T03:18:33.603171+0000","last_clean":"2024-03-26T03:18:33.603171+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:18:33.603171+0000","last_undegraded":"2024-03-26T03:18:33.603171+0000","last_fullsized":"2024-03-26T03:18:33.603171+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:14.862148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:14.862148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":505,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593664+0000","last_change":"2024-03-26T03:14:22.577571+0000","last_active":"2024-03-26T03:18:33.593664+0000","last_peered":"2024-03-26T03:18:33.593664+0000","last_clean":"2024-03-26T03:18:33.593664+0000","last_became_active":"2024-03-26T03:14:22.577265+0000","last_became_peered":"2024-03-26T03:14:22.577265+0000","last_unstale":"2024-03-26T03:18:33.593664+0000","last_undegraded":"2024-03-26T03:18:33.593664+0000","last_fullsized":"2024-03-26T03:18:33.593664+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:19.903253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:19.903253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":516,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600838+0000","last_change":"2024-03-26T03:14:40.950126+0000","last_active":"2024-03-26T03:18:33.600838+0000","last_peered":"2024-03-26T03:18:33.600838+0000","last_clean":"2024-03-26T03:18:33.600838+0000","last_became_active":"2024-03-26T03:14:40.166745+0000","last_became_peered":"2024-03-26T03:14:40.166745+0000","last_unstale":"2024-03-26T03:18:33.600838+0000","last_undegraded":"2024-03-26T03:18:33.600838+0000","last_fullsized":"2024-03-26T03:18:33.600838+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:40.950003+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:14:40.950003+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":522,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593325+0000","last_change":"2024-03-26T03:16:34.843257+0000","last_active":"2024-03-26T03:18:33.593325+0000","last_peered":"2024-03-26T03:18:33.593325+0000","last_clean":"2024-03-26T03:18:33.593325+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:18:33.593325+0000","last_undegraded":"2024-03-26T03:18:33.593325+0000","last_fullsized":"2024-03-26T03:18:33.593325+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'8","last_scrub_stamp":"2024-03-26T03:16:34.843159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:34.843159+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":850,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.585987+0000","last_change":"2024-03-26T03:17:35.886138+0000","last_active":"2024-03-26T03:18:33.585987+0000","last_peered":"2024-03-26T03:18:33.585987+0000","last_clean":"2024-03-26T03:18:33.585987+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:18:33.585987+0000","last_undegraded":"2024-03-26T03:18:33.585987+0000","last_fullsized":"2024-03-26T03:18:33.585987+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.886021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.886021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.652304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":853,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589044+0000","last_change":"2024-03-26T03:17:57.629803+0000","last_active":"2024-03-26T03:18:33.589044+0000","last_peered":"2024-03-26T03:18:33.589044+0000","last_clean":"2024-03-26T03:18:33.589044+0000","last_became_active":"2024-03-26T03:17:56.569576+0000","last_became_peered":"2024-03-26T03:17:56.569576+0000","last_unstale":"2024-03-26T03:18:33.589044+0000","last_undegraded":"2024-03-26T03:18:33.589044+0000","last_fullsized":"2024-03-26T03:18:33.589044+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.629637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.629637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:22.066988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":217,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597973+0000","last_change":"2024-03-26T03:16:45.148252+0000","last_active":"2024-03-26T03:18:33.597973+0000","last_peered":"2024-03-26T03:18:33.597973+0000","last_clean":"2024-03-26T03:18:33.597973+0000","last_became_active":"2024-03-26T03:14:37.260920+0000","last_became_peered":"2024-03-26T03:14:37.260920+0000","last_unstale":"2024-03-26T03:18:33.597973+0000","last_undegraded":"2024-03-26T03:18:33.597973+0000","last_fullsized":"2024-03-26T03:18:33.597973+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.148129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.148129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":589,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598456+0000","last_change":"2024-03-26T03:17:59.621631+0000","last_active":"2024-03-26T03:18:33.598456+0000","last_peered":"2024-03-26T03:18:33.598456+0000","last_clean":"2024-03-26T03:18:33.598456+0000","last_became_active":"2024-03-26T03:17:59.621344+0000","last_became_peered":"2024-03-26T03:17:59.621344+0000","last_unstale":"2024-03-26T03:18:33.598456+0000","last_undegraded":"2024-03-26T03:18:33.598456+0000","last_fullsized":"2024-03-26T03:18:33.598456+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:45.232258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.232258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":585,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593824+0000","last_change":"2024-03-26T03:16:56.991143+0000","last_active":"2024-03-26T03:18:33.593824+0000","last_peered":"2024-03-26T03:18:33.593824+0000","last_clean":"2024-03-26T03:18:33.593824+0000","last_became_active":"2024-03-26T03:14:37.273380+0000","last_became_peered":"2024-03-26T03:14:37.273380+0000","last_unstale":"2024-03-26T03:18:33.593824+0000","last_undegraded":"2024-03-26T03:18:33.593824+0000","last_fullsized":"2024-03-26T03:18:33.593824+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'31","last_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.991031+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":835,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590079+0000","last_change":"2024-03-26T03:17:58.647438+0000","last_active":"2024-03-26T03:18:33.590079+0000","last_peered":"2024-03-26T03:18:33.590079+0000","last_clean":"2024-03-26T03:18:33.590079+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:18:33.590079+0000","last_undegraded":"2024-03-26T03:18:33.590079+0000","last_fullsized":"2024-03-26T03:18:33.590079+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:58.647326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:58.647326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:12.388072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":220,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597308+0000","last_change":"2024-03-26T03:17:32.002736+0000","last_active":"2024-03-26T03:18:33.597308+0000","last_peered":"2024-03-26T03:18:33.597308+0000","last_clean":"2024-03-26T03:18:33.597308+0000","last_became_active":"2024-03-26T03:14:37.249377+0000","last_became_peered":"2024-03-26T03:14:37.249377+0000","last_unstale":"2024-03-26T03:18:33.597308+0000","last_undegraded":"2024-03-26T03:18:33.597308+0000","last_fullsized":"2024-03-26T03:18:33.597308+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.002635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.002635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:00.926149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":491,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595990+0000","last_change":"2024-03-26T03:14:22.603550+0000","last_active":"2024-03-26T03:18:33.595990+0000","last_peered":"2024-03-26T03:18:33.595990+0000","last_clean":"2024-03-26T03:18:33.595990+0000","last_became_active":"2024-03-26T03:14:22.603198+0000","last_became_peered":"2024-03-26T03:14:22.603198+0000","last_unstale":"2024-03-26T03:18:33.595990+0000","last_undegraded":"2024-03-26T03:18:33.595990+0000","last_fullsized":"2024-03-26T03:18:33.595990+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":828,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.586483+0000","last_change":"2024-03-26T03:16:48.688560+0000","last_active":"2024-03-26T03:18:33.586483+0000","last_peered":"2024-03-26T03:18:33.586483+0000","last_clean":"2024-03-26T03:18:33.586483+0000","last_became_active":"2024-03-26T03:13:38.309830+0000","last_became_peered":"2024-03-26T03:13:38.309830+0000","last_unstale":"2024-03-26T03:18:33.586483+0000","last_undegraded":"2024-03-26T03:18:33.586483+0000","last_fullsized":"2024-03-26T03:18:33.586483+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:48.688473+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.688473+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.825160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":696,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594229+0000","last_change":"2024-03-26T03:14:37.333804+0000","last_active":"2024-03-26T03:18:33.594229+0000","last_peered":"2024-03-26T03:18:33.594229+0000","last_clean":"2024-03-26T03:18:33.594229+0000","last_became_active":"2024-03-26T03:14:37.274931+0000","last_became_peered":"2024-03-26T03:14:37.274931+0000","last_unstale":"2024-03-26T03:18:33.594229+0000","last_undegraded":"2024-03-26T03:18:33.594229+0000","last_fullsized":"2024-03-26T03:18:33.594229+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":856,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.583034+0000","last_change":"2024-03-26T03:17:32.885230+0000","last_active":"2024-03-26T03:18:33.583034+0000","last_peered":"2024-03-26T03:18:33.583034+0000","last_clean":"2024-03-26T03:18:33.583034+0000","last_became_active":"2024-03-26T03:14:37.266519+0000","last_became_peered":"2024-03-26T03:14:37.266519+0000","last_unstale":"2024-03-26T03:18:33.583034+0000","last_undegraded":"2024-03-26T03:18:33.583034+0000","last_fullsized":"2024-03-26T03:18:33.583034+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.885069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.885069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:38.557063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"428'11","reported_seq":1106,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590739+0000","last_change":"2024-03-26T03:17:56.576319+0000","last_active":"2024-03-26T03:18:33.590739+0000","last_peered":"2024-03-26T03:18:33.590739+0000","last_clean":"2024-03-26T03:18:33.590739+0000","last_became_active":"2024-03-26T03:17:56.569871+0000","last_became_peered":"2024-03-26T03:17:56.569871+0000","last_unstale":"2024-03-26T03:18:33.590739+0000","last_undegraded":"2024-03-26T03:18:33.590739+0000","last_fullsized":"2024-03-26T03:18:33.590739+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'10","last_scrub_stamp":"2024-03-26T03:16:56.648648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.648648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":599,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593954+0000","last_change":"2024-03-26T03:16:50.207171+0000","last_active":"2024-03-26T03:18:33.593954+0000","last_peered":"2024-03-26T03:18:33.593954+0000","last_clean":"2024-03-26T03:18:33.593954+0000","last_became_active":"2024-03-26T03:16:50.206818+0000","last_became_peered":"2024-03-26T03:16:50.206818+0000","last_unstale":"2024-03-26T03:18:33.593954+0000","last_undegraded":"2024-03-26T03:18:33.593954+0000","last_fullsized":"2024-03-26T03:18:33.593954+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.141917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":814,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597829+0000","last_change":"2024-03-26T03:17:59.611282+0000","last_active":"2024-03-26T03:18:33.597829+0000","last_peered":"2024-03-26T03:18:33.597829+0000","last_clean":"2024-03-26T03:18:33.597829+0000","last_became_active":"2024-03-26T03:17:59.610887+0000","last_became_peered":"2024-03-26T03:17:59.610887+0000","last_unstale":"2024-03-26T03:18:33.597829+0000","last_undegraded":"2024-03-26T03:18:33.597829+0000","last_fullsized":"2024-03-26T03:18:33.597829+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.803243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.803243+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":867,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590899+0000","last_change":"2024-03-26T03:13:32.336994+0000","last_active":"2024-03-26T03:18:33.590899+0000","last_peered":"2024-03-26T03:18:33.590899+0000","last_clean":"2024-03-26T03:18:33.590899+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:18:33.590899+0000","last_undegraded":"2024-03-26T03:18:33.590899+0000","last_fullsized":"2024-03-26T03:18:33.590899+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:13:32.336748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:32.336748+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834288+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":472,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595658+0000","last_change":"2024-03-26T03:17:31.999279+0000","last_active":"2024-03-26T03:18:33.595658+0000","last_peered":"2024-03-26T03:18:33.595658+0000","last_clean":"2024-03-26T03:18:33.595658+0000","last_became_active":"2024-03-26T03:14:22.576816+0000","last_became_peered":"2024-03-26T03:14:22.576816+0000","last_unstale":"2024-03-26T03:18:33.595658+0000","last_undegraded":"2024-03-26T03:18:33.595658+0000","last_fullsized":"2024-03-26T03:18:33.595658+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:17:31.999158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:31.999158+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:59.446120+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":212,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.600446+0000","last_change":"2024-03-26T03:14:37.255762+0000","last_active":"2024-03-26T03:18:33.600446+0000","last_peered":"2024-03-26T03:18:33.600446+0000","last_clean":"2024-03-26T03:18:33.600446+0000","last_became_active":"2024-03-26T03:14:37.255400+0000","last_became_peered":"2024-03-26T03:14:37.255400+0000","last_unstale":"2024-03-26T03:18:33.600446+0000","last_undegraded":"2024-03-26T03:18:33.600446+0000","last_fullsized":"2024-03-26T03:18:33.600446+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:25.571608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:13:25.571608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":826,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601253+0000","last_change":"2024-03-26T03:17:56.570706+0000","last_active":"2024-03-26T03:18:33.601253+0000","last_peered":"2024-03-26T03:18:33.601253+0000","last_clean":"2024-03-26T03:18:33.601253+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:18:33.601253+0000","last_undegraded":"2024-03-26T03:18:33.601253+0000","last_fullsized":"2024-03-26T03:18:33.601253+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.953482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.953482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":717,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596090+0000","last_change":"2024-03-26T03:14:40.162652+0000","last_active":"2024-03-26T03:18:33.596090+0000","last_peered":"2024-03-26T03:18:33.596090+0000","last_clean":"2024-03-26T03:18:33.596090+0000","last_became_active":"2024-03-26T03:14:40.162258+0000","last_became_peered":"2024-03-26T03:14:40.162258+0000","last_unstale":"2024-03-26T03:18:33.596090+0000","last_undegraded":"2024-03-26T03:18:33.596090+0000","last_fullsized":"2024-03-26T03:18:33.596090+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.546156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.546156+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":501,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.604643+0000","last_change":"2024-03-26T03:16:50.211157+0000","last_active":"2024-03-26T03:18:33.604643+0000","last_peered":"2024-03-26T03:18:33.604643+0000","last_clean":"2024-03-26T03:18:33.604643+0000","last_became_active":"2024-03-26T03:16:50.209508+0000","last_became_peered":"2024-03-26T03:16:50.209508+0000","last_unstale":"2024-03-26T03:18:33.604643+0000","last_undegraded":"2024-03-26T03:18:33.604643+0000","last_fullsized":"2024-03-26T03:18:33.604643+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'27","last_scrub_stamp":"2024-03-26T03:16:36.265181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.265181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":848,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589299+0000","last_change":"2024-03-26T03:17:47.637363+0000","last_active":"2024-03-26T03:18:33.589299+0000","last_peered":"2024-03-26T03:18:33.589299+0000","last_clean":"2024-03-26T03:18:33.589299+0000","last_became_active":"2024-03-26T03:14:37.249066+0000","last_became_peered":"2024-03-26T03:14:37.249066+0000","last_unstale":"2024-03-26T03:18:33.589299+0000","last_undegraded":"2024-03-26T03:18:33.589299+0000","last_fullsized":"2024-03-26T03:18:33.589299+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:47.637246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:47.637246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:57.108832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":113,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590152+0000","last_change":"2024-03-26T03:17:56.573995+0000","last_active":"2024-03-26T03:18:33.590152+0000","last_peered":"2024-03-26T03:18:33.590152+0000","last_clean":"2024-03-26T03:18:33.590152+0000","last_became_active":"2024-03-26T03:17:56.571840+0000","last_became_peered":"2024-03-26T03:17:56.571840+0000","last_unstale":"2024-03-26T03:18:33.590152+0000","last_undegraded":"2024-03-26T03:18:33.590152+0000","last_fullsized":"2024-03-26T03:18:33.590152+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.507151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.507151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.755968+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":814,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599902+0000","last_change":"2024-03-26T03:13:29.068724+0000","last_active":"2024-03-26T03:18:33.599902+0000","last_peered":"2024-03-26T03:18:33.599902+0000","last_clean":"2024-03-26T03:18:33.599902+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:18:33.599902+0000","last_undegraded":"2024-03-26T03:18:33.599902+0000","last_fullsized":"2024-03-26T03:18:33.599902+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.068502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.068502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":613,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.583309+0000","last_change":"2024-03-26T03:16:46.720364+0000","last_active":"2024-03-26T03:18:33.583309+0000","last_peered":"2024-03-26T03:18:33.583309+0000","last_clean":"2024-03-26T03:18:33.583309+0000","last_became_active":"2024-03-26T03:14:40.162583+0000","last_became_peered":"2024-03-26T03:14:40.162583+0000","last_unstale":"2024-03-26T03:18:33.583309+0000","last_undegraded":"2024-03-26T03:18:33.583309+0000","last_fullsized":"2024-03-26T03:18:33.583309+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.720272+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.823186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"399'30","reported_seq":563,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594170+0000","last_change":"2024-03-26T03:16:48.957169+0000","last_active":"2024-03-26T03:18:33.594170+0000","last_peered":"2024-03-26T03:18:33.594170+0000","last_clean":"2024-03-26T03:18:33.594170+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:18:33.594170+0000","last_undegraded":"2024-03-26T03:18:33.594170+0000","last_fullsized":"2024-03-26T03:18:33.594170+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'30","last_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.957049+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":530,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596265+0000","last_change":"2024-03-26T03:16:39.889851+0000","last_active":"2024-03-26T03:18:33.596265+0000","last_peered":"2024-03-26T03:18:33.596265+0000","last_clean":"2024-03-26T03:18:33.596265+0000","last_became_active":"2024-03-26T03:14:37.275278+0000","last_became_peered":"2024-03-26T03:14:37.275278+0000","last_unstale":"2024-03-26T03:18:33.596265+0000","last_undegraded":"2024-03-26T03:18:33.596265+0000","last_fullsized":"2024-03-26T03:18:33.596265+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:39.889770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.889770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":793,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599188+0000","last_change":"2024-03-26T03:14:36.271942+0000","last_active":"2024-03-26T03:18:33.599188+0000","last_peered":"2024-03-26T03:18:33.599188+0000","last_clean":"2024-03-26T03:18:33.599188+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:18:33.599188+0000","last_undegraded":"2024-03-26T03:18:33.599188+0000","last_fullsized":"2024-03-26T03:18:33.599188+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.271810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.271810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":486,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.596711+0000","last_change":"2024-03-26T03:14:21.603222+0000","last_active":"2024-03-26T03:18:33.596711+0000","last_peered":"2024-03-26T03:18:33.596711+0000","last_clean":"2024-03-26T03:18:33.596711+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:18:33.596711+0000","last_undegraded":"2024-03-26T03:18:33.596711+0000","last_fullsized":"2024-03-26T03:18:33.596711+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":57,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597772+0000","last_change":"2024-03-26T03:17:57.160567+0000","last_active":"2024-03-26T03:18:33.597772+0000","last_peered":"2024-03-26T03:18:33.597772+0000","last_clean":"2024-03-26T03:18:33.597772+0000","last_became_active":"2024-03-26T03:17:56.567557+0000","last_became_peered":"2024-03-26T03:17:56.567557+0000","last_unstale":"2024-03-26T03:18:33.597772+0000","last_undegraded":"2024-03-26T03:18:33.597772+0000","last_fullsized":"2024-03-26T03:18:33.597772+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'33","last_scrub_stamp":"2024-03-26T03:17:57.160442+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.160442+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:13.145759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":216,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589744+0000","last_change":"2024-03-26T03:16:51.673167+0000","last_active":"2024-03-26T03:18:33.589744+0000","last_peered":"2024-03-26T03:18:33.589744+0000","last_clean":"2024-03-26T03:18:33.589744+0000","last_became_active":"2024-03-26T03:14:37.243678+0000","last_became_peered":"2024-03-26T03:14:37.243678+0000","last_unstale":"2024-03-26T03:18:33.589744+0000","last_undegraded":"2024-03-26T03:18:33.589744+0000","last_fullsized":"2024-03-26T03:18:33.589744+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.673066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.673066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":865,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.586681+0000","last_change":"2024-03-26T03:18:01.638296+0000","last_active":"2024-03-26T03:18:33.586681+0000","last_peered":"2024-03-26T03:18:33.586681+0000","last_clean":"2024-03-26T03:18:33.586681+0000","last_became_active":"2024-03-26T03:17:56.570631+0000","last_became_peered":"2024-03-26T03:17:56.570631+0000","last_unstale":"2024-03-26T03:18:33.586681+0000","last_undegraded":"2024-03-26T03:18:33.586681+0000","last_fullsized":"2024-03-26T03:18:33.586681+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:01.638157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:01.638157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:28.331978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":285,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598273+0000","last_change":"2024-03-26T03:17:56.571755+0000","last_active":"2024-03-26T03:18:33.598273+0000","last_peered":"2024-03-26T03:18:33.598273+0000","last_clean":"2024-03-26T03:18:33.598273+0000","last_became_active":"2024-03-26T03:17:56.571178+0000","last_became_peered":"2024-03-26T03:17:56.571178+0000","last_unstale":"2024-03-26T03:18:33.598273+0000","last_undegraded":"2024-03-26T03:18:33.598273+0000","last_fullsized":"2024-03-26T03:18:33.598273+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.660527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.660527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":560,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.593989+0000","last_change":"2024-03-26T03:17:59.595137+0000","last_active":"2024-03-26T03:18:33.593989+0000","last_peered":"2024-03-26T03:18:33.593989+0000","last_clean":"2024-03-26T03:18:33.593989+0000","last_became_active":"2024-03-26T03:17:59.594759+0000","last_became_peered":"2024-03-26T03:17:59.594759+0000","last_unstale":"2024-03-26T03:18:33.593989+0000","last_undegraded":"2024-03-26T03:18:33.593989+0000","last_fullsized":"2024-03-26T03:18:33.593989+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:32.825864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.825864+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":43,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598319+0000","last_change":"2024-03-26T03:17:56.580380+0000","last_active":"2024-03-26T03:18:33.598319+0000","last_peered":"2024-03-26T03:18:33.598319+0000","last_clean":"2024-03-26T03:18:33.598319+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:18:33.598319+0000","last_undegraded":"2024-03-26T03:18:33.598319+0000","last_fullsized":"2024-03-26T03:18:33.598319+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'13","last_scrub_stamp":"2024-03-26T03:17:45.281375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.281375+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.925165+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":821,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.590084+0000","last_change":"2024-03-26T03:16:38.824092+0000","last_active":"2024-03-26T03:18:33.590084+0000","last_peered":"2024-03-26T03:18:33.590084+0000","last_clean":"2024-03-26T03:18:33.590084+0000","last_became_active":"2024-03-26T03:14:37.373600+0000","last_became_peered":"2024-03-26T03:14:37.373600+0000","last_unstale":"2024-03-26T03:18:33.590084+0000","last_undegraded":"2024-03-26T03:18:33.590084+0000","last_fullsized":"2024-03-26T03:18:33.590084+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:38.823985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.823985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":217,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598117+0000","last_change":"2024-03-26T03:16:51.182163+0000","last_active":"2024-03-26T03:18:33.598117+0000","last_peered":"2024-03-26T03:18:33.598117+0000","last_clean":"2024-03-26T03:18:33.598117+0000","last_became_active":"2024-03-26T03:14:37.253707+0000","last_became_peered":"2024-03-26T03:14:37.253707+0000","last_unstale":"2024-03-26T03:18:33.598117+0000","last_undegraded":"2024-03-26T03:18:33.598117+0000","last_fullsized":"2024-03-26T03:18:33.598117+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.182029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":807,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.583670+0000","last_change":"2024-03-26T03:17:47.711589+0000","last_active":"2024-03-26T03:18:33.583670+0000","last_peered":"2024-03-26T03:18:33.583670+0000","last_clean":"2024-03-26T03:18:33.583670+0000","last_became_active":"2024-03-26T03:14:21.596806+0000","last_became_peered":"2024-03-26T03:14:21.596806+0000","last_unstale":"2024-03-26T03:18:33.583670+0000","last_undegraded":"2024-03-26T03:18:33.583670+0000","last_fullsized":"2024-03-26T03:18:33.583670+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:47.711445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:47.711445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:03.092974+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":620,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.601016+0000","last_change":"2024-03-26T03:17:59.610214+0000","last_active":"2024-03-26T03:18:33.601016+0000","last_peered":"2024-03-26T03:18:33.601016+0000","last_clean":"2024-03-26T03:18:33.601016+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:18:33.601016+0000","last_undegraded":"2024-03-26T03:18:33.601016+0000","last_fullsized":"2024-03-26T03:18:33.601016+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:51.516127+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:17:51.516127+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:09.405345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":858,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597300+0000","last_change":"2024-03-26T03:16:46.178208+0000","last_active":"2024-03-26T03:18:33.597300+0000","last_peered":"2024-03-26T03:18:33.597300+0000","last_clean":"2024-03-26T03:18:33.597300+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:18:33.597300+0000","last_undegraded":"2024-03-26T03:18:33.597300+0000","last_fullsized":"2024-03-26T03:18:33.597300+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:16:46.178099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.178099+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":495,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595587+0000","last_change":"2024-03-26T03:14:22.572664+0000","last_active":"2024-03-26T03:18:33.595587+0000","last_peered":"2024-03-26T03:18:33.595587+0000","last_clean":"2024-03-26T03:18:33.595587+0000","last_became_active":"2024-03-26T03:14:22.571857+0000","last_became_peered":"2024-03-26T03:14:22.571857+0000","last_unstale":"2024-03-26T03:18:33.595587+0000","last_undegraded":"2024-03-26T03:18:33.595587+0000","last_fullsized":"2024-03-26T03:18:33.595587+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":825,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598072+0000","last_change":"2024-03-26T03:15:20.809430+0000","last_active":"2024-03-26T03:18:33.598072+0000","last_peered":"2024-03-26T03:18:33.598072+0000","last_clean":"2024-03-26T03:18:33.598072+0000","last_became_active":"2024-03-26T03:14:37.242540+0000","last_became_peered":"2024-03-26T03:14:37.242540+0000","last_unstale":"2024-03-26T03:18:33.598072+0000","last_undegraded":"2024-03-26T03:18:33.598072+0000","last_fullsized":"2024-03-26T03:18:33.598072+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:20.809176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:20.809176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":820,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.598026+0000","last_change":"2024-03-26T03:17:43.076516+0000","last_active":"2024-03-26T03:18:33.598026+0000","last_peered":"2024-03-26T03:18:33.598026+0000","last_clean":"2024-03-26T03:18:33.598026+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:18:33.598026+0000","last_undegraded":"2024-03-26T03:18:33.598026+0000","last_fullsized":"2024-03-26T03:18:33.598026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.076334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.076334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:49.381353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":516,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.589531+0000","last_change":"2024-03-26T03:16:29.882781+0000","last_active":"2024-03-26T03:18:33.589531+0000","last_peered":"2024-03-26T03:18:33.589531+0000","last_clean":"2024-03-26T03:18:33.589531+0000","last_became_active":"2024-03-26T03:14:40.174111+0000","last_became_peered":"2024-03-26T03:14:40.174111+0000","last_unstale":"2024-03-26T03:18:33.589531+0000","last_undegraded":"2024-03-26T03:18:33.589531+0000","last_fullsized":"2024-03-26T03:18:33.589531+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.882681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.882681+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:36.834431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":583,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.595349+0000","last_change":"2024-03-26T03:14:22.625030+0000","last_active":"2024-03-26T03:18:33.595349+0000","last_peered":"2024-03-26T03:18:33.595349+0000","last_clean":"2024-03-26T03:18:33.595349+0000","last_became_active":"2024-03-26T03:14:22.576815+0000","last_became_peered":"2024-03-26T03:14:22.576815+0000","last_unstale":"2024-03-26T03:18:33.595349+0000","last_undegraded":"2024-03-26T03:18:33.595349+0000","last_fullsized":"2024-03-26T03:18:33.595349+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":65,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":220,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.599241+0000","last_change":"2024-03-26T03:16:48.135457+0000","last_active":"2024-03-26T03:18:33.599241+0000","last_peered":"2024-03-26T03:18:33.599241+0000","last_clean":"2024-03-26T03:18:33.599241+0000","last_became_active":"2024-03-26T03:14:37.248588+0000","last_became_peered":"2024-03-26T03:14:37.248588+0000","last_unstale":"2024-03-26T03:18:33.599241+0000","last_undegraded":"2024-03-26T03:18:33.599241+0000","last_fullsized":"2024-03-26T03:18:33.599241+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:48.135354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.135354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.597472+0000","last_change":"2024-03-26T03:17:56.572431+0000","last_active":"2024-03-26T03:18:33.597472+0000","last_peered":"2024-03-26T03:18:33.597472+0000","last_clean":"2024-03-26T03:18:33.597472+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:18:33.597472+0000","last_undegraded":"2024-03-26T03:18:33.597472+0000","last_fullsized":"2024-03-26T03:18:33.597472+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.615644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.615644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:47.803583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":486,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-03-26T03:18:33.594991+0000","last_change":"2024-03-26T03:14:37.269042+0000","last_active":"2024-03-26T03:18:33.594991+0000","last_peered":"2024-03-26T03:18:33.594991+0000","last_clean":"2024-03-26T03:18:33.594991+0000","last_became_active":"2024-03-26T03:14:37.265472+0000","last_became_peered":"2024-03-26T03:14:37.265472+0000","last_unstale":"2024-03-26T03:18:33.594991+0000","last_undegraded":"2024-03-26T03:18:33.594991+0000","last_fullsized":"2024-03-26T03:18:33.594991+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":8433434624,"num_objects":128684,"num_object_clones":0,"num_object_copies":386052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11445,"num_objects_unfound":0,"num_objects_dirty":128684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128684,"num_write_kb":8235776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10145,"num_bytes_recovered":664862720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26113343488,"data_stored":26113343488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52793,"ondisk_log_size":52793,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603785,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":958300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1167,"num_read_kb":1162,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":3014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":958,"ondisk_log_size":958,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430452,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5805488,"kb_used_data":5074408,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":87939664,"statfs":{"total":95995035648,"available":90050215936,"internally_reserved":0,"allocated":5196193792,"data_stored":5195507921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":748419526},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575117,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6634584,"kb_used_data":5642512,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":87110568,"statfs":{"total":95995035648,"available":89201221632,"internally_reserved":0,"allocated":5777932288,"data_stored":5777207623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738642,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411591,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4422264,"kb_used_data":3610672,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":89322888,"statfs":{"total":95995035648,"available":91466637312,"internally_reserved":0,"allocated":3697328128,"data_stored":3696635514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175028,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4996096,"kb_used_data":4604168,"kb_used_omap":4,"kb_used_meta":391355,"kb_avail":88749056,"statfs":{"total":95995035648,"available":90879033344,"internally_reserved":0,"allocated":4714668032,"data_stored":4713950038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":400748195},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986430,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3617136,"kb_used_data":2841392,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":90128016,"statfs":{"total":95995035648,"available":92291088384,"internally_reserved":0,"allocated":2909585408,"data_stored":2908892145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692090,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4505732,"kb_used_data":3549756,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":89239420,"statfs":{"total":95995035648,"available":91381166080,"internally_reserved":0,"allocated":3634950144,"data_stored":3634244745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5820514304,"data_stored":5820514304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3725000704,"data_stored":3725000704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4715184128,"data_stored":4715184128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2934898688,"data_stored":2934898688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3670999040,"data_stored":3670999040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5246746624,"data_stored":5246746624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:18:39.001 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:18:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:39.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:39.116 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:18:39.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:39.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:39.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:39.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:39.464 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.555 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:39.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:39.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:39.670 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:39.670 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:18:39.670 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:18:39.670 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:39.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:39.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:39.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:39.932 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:39.985 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:39.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:40.060 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:40.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:40.099 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:40.155 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:40.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:40.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:40.316 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:40.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:40.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:40.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:40.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:40.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:18:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:18:40.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:40.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:18:40.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:40.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:40.688 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:40.730 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:40.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:40.832 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:40.877 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:40.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:40.978 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:40.981 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:41.020 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:41.122 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:41.178 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:41.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:41.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:41.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:41.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:41 smithi192 ceph-mon[33110]: pgmap v934: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 767 B/s rd, 67 MiB/s wr, 1.21k op/s; 10351/391146 objects misplaced (2.646%); 23 MiB/s, 371 objects/s recovering 2024-03-26T03:18:41.280 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:41.323 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:41.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:41.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:41.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:41.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[37920]: pgmap v934: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 767 B/s rd, 67 MiB/s wr, 1.21k op/s; 10351/391146 objects misplaced (2.646%); 23 MiB/s, 371 objects/s recovering 2024-03-26T03:18:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:41.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:41 smithi066 ceph-mon[29666]: pgmap v934: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 767 B/s rd, 67 MiB/s wr, 1.21k op/s; 10351/391146 objects misplaced (2.646%); 23 MiB/s, 371 objects/s recovering 2024-03-26T03:18:41.425 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:41.458 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:41.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:41.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:41.697 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:41.737 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:41.839 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:41.867 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:41.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:41.969 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:42.011 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:42.113 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.157 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:42.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:42.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:42.294 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:42.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:42.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:42.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:42.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:42.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:42.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:42.807 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:42.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:42.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:42.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:42.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:43.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:43.133 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:43.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:43.235 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:43.236 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:18:43.236 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:18:43.236 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:18:43.236 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:43.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:43.380 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.415 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:43.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:43.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:43.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:43.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:43.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:43.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:43.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:43.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:43 smithi066 ceph-mon[29666]: pgmap v935: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 1.14k op/s; 8320/390936 objects misplaced (2.128%); 32 MiB/s, 510 objects/s recovering 2024-03-26T03:18:43.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:43 smithi066 ceph-mon[37920]: pgmap v935: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 1.14k op/s; 8320/390936 objects misplaced (2.128%); 32 MiB/s, 510 objects/s recovering 2024-03-26T03:18:43.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:43.949 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:43.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:43.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:43 smithi192 ceph-mon[33110]: pgmap v935: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 1.14k op/s; 8320/390936 objects misplaced (2.128%); 32 MiB/s, 510 objects/s recovering 2024-03-26T03:18:44.051 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:44.081 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:18:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:18:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:18:44.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:44.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:44.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:44.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:44.437 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:44.467 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:44.569 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:44.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:44.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:44.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:44.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:44.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:44 smithi066 ceph-mon[29666]: osdmap e441: 8 total, 8 up, 6 in 2024-03-26T03:18:44.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:44 smithi066 ceph-mon[37920]: osdmap e441: 8 total, 8 up, 6 in 2024-03-26T03:18:44.872 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:44.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:44.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:44.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:44 smithi192 ceph-mon[33110]: osdmap e441: 8 total, 8 up, 6 in 2024-03-26T03:18:45.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:45.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:45.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:45.250 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:45.294 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:45.396 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:45.438 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:45.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:45.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:45.672 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.701 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:45.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:45.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:45.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:45 smithi192 ceph-mon[33110]: pgmap v937: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 1.12k op/s; 8320/390936 objects misplaced (2.128%); 31 MiB/s, 502 objects/s recovering 2024-03-26T03:18:45.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:45 smithi192 ceph-mon[33110]: osdmap e442: 8 total, 8 up, 6 in 2024-03-26T03:18:45.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:45.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:45 smithi066 ceph-mon[29666]: pgmap v937: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 1.12k op/s; 8320/390936 objects misplaced (2.128%); 31 MiB/s, 502 objects/s recovering 2024-03-26T03:18:45.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:45 smithi066 ceph-mon[29666]: osdmap e442: 8 total, 8 up, 6 in 2024-03-26T03:18:45.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:45 smithi066 ceph-mon[37920]: pgmap v937: 178 pgs: 3 active+remapped+backfilling, 175 active+clean; 7.9 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 1.12k op/s; 8320/390936 objects misplaced (2.128%); 31 MiB/s, 502 objects/s recovering 2024-03-26T03:18:45.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:45 smithi066 ceph-mon[37920]: osdmap e442: 8 total, 8 up, 6 in 2024-03-26T03:18:45.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:45.953 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:45.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:46.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:46.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:46.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:46.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:46.267 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:18:46.321 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:46.360 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:46.463 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:46.508 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:46.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:46.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:46.737 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.780 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:46.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3083746663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: from='client.35933 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: from='client.35933 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:18:46.829 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:46 smithi192 ceph-mon[33110]: osdmap e443: 8 total, 8 up, 6 in 2024-03-26T03:18:46.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:18:46.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:18:46.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3083746663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: from='client.35933 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: from='client.35933 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[29666]: osdmap e443: 8 total, 8 up, 6 in 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3083746663' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:18:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: from='client.35933 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:18:46.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:18:46.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: from='client.35933 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:18:46.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:46 smithi066 ceph-mon[37920]: osdmap e443: 8 total, 8 up, 6 in 2024-03-26T03:18:46.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:18:46.881 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:46.924 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:46.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:47.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:47.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:47.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:47.168 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.196 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:47.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:47.219 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:18:47.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:47.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:47.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:47.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:47.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:47.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:47.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:47.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:47 smithi192 ceph-mon[33110]: pgmap v940: 178 pgs: 1 peering, 2 active+remapped+backfilling, 175 active+clean; 7.6 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.7 KiB/s rd, 7.7 MiB/s wr, 1.12k op/s; 6130/375363 objects misplaced (1.633%); 41 MiB/s, 661 objects/s recovering 2024-03-26T03:18:47.610 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:47 smithi192 ceph-mon[33110]: 11.b scrub starts 2024-03-26T03:18:47.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:47.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:47.833 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:47.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:47 smithi066 ceph-mon[29666]: pgmap v940: 178 pgs: 1 peering, 2 active+remapped+backfilling, 175 active+clean; 7.6 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.7 KiB/s rd, 7.7 MiB/s wr, 1.12k op/s; 6130/375363 objects misplaced (1.633%); 41 MiB/s, 661 objects/s recovering 2024-03-26T03:18:47.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:47 smithi066 ceph-mon[29666]: 11.b scrub starts 2024-03-26T03:18:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:47 smithi066 ceph-mon[37920]: pgmap v940: 178 pgs: 1 peering, 2 active+remapped+backfilling, 175 active+clean; 7.6 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.7 KiB/s rd, 7.7 MiB/s wr, 1.12k op/s; 6130/375363 objects misplaced (1.633%); 41 MiB/s, 661 objects/s recovering 2024-03-26T03:18:47.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:47 smithi066 ceph-mon[37920]: 11.b scrub starts 2024-03-26T03:18:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:47.882 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:47.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:47.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:48.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:48.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:48.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:48.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:48.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:48.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:48.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:48.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:48.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:48.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:48.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:48.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:48.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:48.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:48.940 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:48.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:49.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:49.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:49.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:49.199 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:49.301 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:49.347 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:49.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.504 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:49.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:49.604 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:49 smithi192 ceph-mon[33110]: pgmap v941: 178 pgs: 1 peering, 2 active+remapped+backfilling, 175 active+clean; 7.6 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 865 op/s; 6130/375363 objects misplaced (1.633%); 20 MiB/s, 318 objects/s recovering 2024-03-26T03:18:49.604 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:49 smithi192 ceph-mon[33110]: 1.18 scrub starts 2024-03-26T03:18:49.605 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:49 smithi192 ceph-mon[33110]: 1.18 scrub ok 2024-03-26T03:18:49.605 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:49.606 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:18:49.606 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:18:49.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137152 10.c 10:39382015:::benchmark_data_smithi066_148944_object12505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.228227+0000", 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 253.37372144099999, 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0128037700000001, 2024-03-26T03:18:49.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137152 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228227+0000", 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228227+0000", 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228229+0000", 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.606e-06 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228239+0000", 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0379e-05 2024-03-26T03:18:49.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228258+0000", 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8518000000000001e-05 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228269+0000", 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1591e-05 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228326+0000", 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6226e-05 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228495+0000", 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016949099999999999 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228655+0000", 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160063 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.233055+0000", 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044002210000000002 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.239260+0000", 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0062048579999999997 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.264903+0000", 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025642841 2024-03-26T03:18:49.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.264916+0000", 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3325e-05 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241014+0000", 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97609788799999997 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241031+0000", 2024-03-26T03:18:49.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6762999999999999e-05 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137181 10.c 10.72630f0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.286355+0000", 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 253.31559338, 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95471075100000002, 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:49.614 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137181 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286355+0000", 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286355+0000", 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286357+0000", 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1940000000000001e-06 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286364+0000", 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6200000000000001e-06 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286379+0000", 2024-03-26T03:18:49.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5282999999999999e-05 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286387+0000", 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5789999999999998e-06 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286447+0000", 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0154e-05 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286458+0000", 2024-03-26T03:18:49.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0808000000000001e-05 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241057+0000", 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95459924399999996 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241066+0000", 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8689999999999993e-06 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.618 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137196 10.c 10.e56587ac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.308988+0000", 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 253.29296084800001, 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93210979800000004, 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137196 2024-03-26T03:18:49.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308988+0000", 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308988+0000", 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308990+0000", 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.351e-06 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308995+0000", 2024-03-26T03:18:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1710000000000001e-06 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309010+0000", 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4812e-05 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309017+0000", 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1119999999999998e-06 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309078+0000", 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0445999999999997e-05 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309088+0000", 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0752e-05 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241089+0000", 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93200086800000004 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241098+0000", 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2859999999999999e-06 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137219 10.9 e343/342 10:9a86151a:::benchmark_data_smithi066_148944_object12572:head v 343'8542, mlcod=0'0)", 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.326483+0000", 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.275465744, 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47544255400000002, 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.624 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326483+0000", 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326483+0000", 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326485+0000", 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.73e-06 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326493+0000", 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7339999999999997e-06 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326503+0000", 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.006e-05 2024-03-26T03:18:49.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326510+0000", 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8299999999999998e-06 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326583+0000", 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2498999999999998e-05 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326927+0000", 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034401400000000002 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801882+0000", 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47495507399999998 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801925+0000", 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3612999999999997e-05 2024-03-26T03:18:49.628 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137247 10.9 e343/342 10:9cdb27b9:::benchmark_data_smithi066_148944_object12600:head v 343'8543, mlcod=0'0)", 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.335355+0000", 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.26659334600001, 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57069690200000001, 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335355+0000", 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335355+0000", 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335357+0000", 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2860000000000001e-06 2024-03-26T03:18:49.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335364+0000", 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3100000000000003e-06 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335373+0000", 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3009999999999995e-06 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335380+0000", 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4110000000000001e-06 2024-03-26T03:18:49.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.352564+0000", 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017184041000000001 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.352586+0000", 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2751999999999998e-05 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.905930+0000", 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55334367399999995 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906052+0000", 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000122127 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137258 10.8 e343/342 10:1812d821:::benchmark_data_smithi066_148944_object12611:head v 343'8573, mlcod=0'0)", 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.335708+0000", 2024-03-26T03:18:49.633 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.26624020400001, 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46613804800000003, 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335708+0000", 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335708+0000", 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335710+0000", 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0860000000000001e-06 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335719+0000", 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7110000000000007e-06 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335728+0000", 2024-03-26T03:18:49.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0380000000000005e-06 2024-03-26T03:18:49.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335735+0000", 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.537398+0000", 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20166337500000001 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.537427+0000", 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.853e-05 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801809+0000", 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26438231699999998 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801847+0000", 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7389999999999999e-05 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.639 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137254 10.9 e343/342 10:97a448bb:::benchmark_data_smithi066_148944_object12607:head v 343'8544, mlcod=0'0)", 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.337366+0000", 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.26458255599999, 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.568755389, 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337366+0000", 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337366+0000", 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337367+0000", 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.221e-06 2024-03-26T03:18:49.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337375+0000", 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3440000000000003e-06 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337389+0000", 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4479e-05 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337397+0000", 2024-03-26T03:18:49.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5639999999999999e-06 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.356826+0000", 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019429168 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.357212+0000", 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038587099999999998 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906056+0000", 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54884401500000002 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906122+0000", 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5727000000000003e-05 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137266 10.9 e343/342 10:9c7c1621:::benchmark_data_smithi066_148944_object12619:head v 343'8545, mlcod=0'0)", 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.488743+0000", 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.11320569399999, 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55176513699999996, 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488743+0000", 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488743+0000", 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488745+0000", 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.826e-06 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488753+0000", 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6629999999999999e-06 2024-03-26T03:18:49.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488768+0000", 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4498000000000001e-05 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488776+0000", 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3909999999999998e-06 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708819+0000", 2024-03-26T03:18:49.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.220042882 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708835+0000", 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5818e-05 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040459+0000", 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33162439900000001 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040508+0000", 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8659999999999998e-05 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137265 10.1 e343/341 10:8eba7ad0:::benchmark_data_smithi066_148944_object12618:head v 343'8713, mlcod=342'8711)", 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.488970+0000", 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.11297840399999, 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55538486899999995, 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.649 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488970+0000", 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488970+0000", 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488972+0000", 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3990000000000001e-06 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488978+0000", 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6460000000000004e-06 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488992+0000", 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3434e-05 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488999+0000", 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8410000000000002e-06 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801839+0000", 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.31284087700000002 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801863+0000", 2024-03-26T03:18:49.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3876999999999999e-05 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.044306+0000", 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24244289799999999 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.044355+0000", 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8897000000000003e-05 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137271 10.9 e343/342 10:96c90966:::benchmark_data_smithi066_148944_object12624:head v 343'8546, mlcod=0'0)", 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.547439+0000", 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 251.05450961099999, 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49976598, 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547439+0000", 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547439+0000", 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547441+0000", 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-03-26T03:18:49.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547451+0000", 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6339999999999996e-06 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547467+0000", 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6708e-05 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547478+0000", 2024-03-26T03:18:49.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0818e-05 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801958+0000", 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25448008500000002 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801995+0000", 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6918000000000001e-05 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.046398+0000", 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.244402386 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047205+0000", 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000807386 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137274 10.8 e343/342 10:15ded6e2:::benchmark_data_smithi066_148944_object12627:head v 343'8575, mlcod=343'8571)", 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.642726+0000", 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.95922309700001, 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39793152999999998, 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642726+0000", 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642726+0000", 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642727+0000", 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.42e-06 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642734+0000", 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3100000000000003e-06 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642746+0000", 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2091e-05 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642754+0000", 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6860000000000006e-06 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708753+0000", 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.065998860000000006 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708781+0000", 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7880000000000001e-05 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040583+0000", 2024-03-26T03:18:49.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33180187300000002 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040657+0000", 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4410000000000001e-05 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137279 10.1 e343/341 10:8adca3c1:::benchmark_data_smithi066_148944_object12632:head v 343'8714, mlcod=342'8711)", 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.645279+0000", 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.956669334, 2024-03-26T03:18:49.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40479705599999999, 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645279+0000", 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645279+0000", 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645282+0000", 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645291+0000", 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6870000000000006e-06 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645308+0000", 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6716000000000001e-05 2024-03-26T03:18:49.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645317+0000", 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3549999999999994e-06 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.905951+0000", 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26063393499999998 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.905984+0000", 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2678999999999997e-05 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.049973+0000", 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14398904600000001 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050076+0000", 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103428 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137281 10.1 e343/341 10:8598039e:::benchmark_data_smithi066_148944_object12634:head v 343'8715, mlcod=342'8711)", 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.648536+0000", 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.953412703, 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.401635768, 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648536+0000", 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648536+0000", 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648538+0000", 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7349999999999999e-06 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648547+0000", 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0799999999999995e-06 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648575+0000", 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8367000000000001e-05 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648584+0000", 2024-03-26T03:18:49.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2070000000000001e-06 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040465+0000", 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39188030800000001 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040490+0000", 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5731000000000001e-05 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050116+0000", 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0096260900000000003 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050172+0000", 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5250000000000001e-05 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137282 10.1 e343/341 10:86db17fd:::benchmark_data_smithi066_148944_object12635:head v 343'8716, mlcod=342'8711)", 2024-03-26T03:18:49.672 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.650848+0000", 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.951100475, 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39937403100000002, 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650848+0000", 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650848+0000", 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650850+0000", 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9539999999999998e-06 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650859+0000", 2024-03-26T03:18:49.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1290000000000002e-06 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650877+0000", 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7227999999999998e-05 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650885+0000", 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6009999999999995e-06 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042328+0000", 2024-03-26T03:18:49.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39144269500000001 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042348+0000", 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050174+0000", 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078259000000000002 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050222+0000", 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8161999999999997e-05 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137283 10.8 e343/342 10:18321bbd:::benchmark_data_smithi066_148944_object12636:head v 343'8576, mlcod=343'8571)", 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.652601+0000", 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.94934798700001, 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.390188335, 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.677 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652601+0000", 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652601+0000", 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652602+0000", 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5629999999999999e-06 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652611+0000", 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.149e-06 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652627+0000", 2024-03-26T03:18:49.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5875e-05 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652635+0000", 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1040000000000007e-06 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801872+0000", 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.149237114 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801890+0000", 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7972999999999999e-05 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042720+0000", 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.240829876 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042789+0000", 2024-03-26T03:18:49.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8681000000000002e-05 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137284 10.1 e343/341 10:83f5b3bd:::benchmark_data_smithi066_148944_object12637:head v 343'8717, mlcod=342'8711)", 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.654011+0000", 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.94793801599999, 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40273788399999999, 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.682 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654011+0000", 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654011+0000", 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654012+0000", 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4789999999999999e-06 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654021+0000", 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5320000000000006e-06 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654042+0000", 2024-03-26T03:18:49.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1755000000000001e-05 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654050+0000", 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3339999999999996e-06 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.045994+0000", 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39194385999999998 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.046016+0000", 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.232e-05 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.056702+0000", 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010686322999999999 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.056749+0000", 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6281e-05 2024-03-26T03:18:49.686 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137288 10.9 e343/342 10:9d93d421:::benchmark_data_smithi066_148944_object12641:head v 343'8547, mlcod=0'0)", 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.656335+0000", 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.945613466, 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39090719400000001, 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656335+0000", 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656335+0000", 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656337+0000", 2024-03-26T03:18:49.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.494e-06 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656345+0000", 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6109999999999994e-06 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656360+0000", 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4423999999999999e-05 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656368+0000", 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0029999999999994e-06 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906207+0000", 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24983883000000001 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906231+0000", 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4131e-05 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047210+0000", 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14097958099999999 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047242+0000", 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2119999999999997e-05 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137296 10.8 e343/342 10:12c9bc13:::benchmark_data_smithi066_148944_object12649:head v 343'8578, mlcod=343'8572)", 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.811312+0000", 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 250.790636178, 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.417516, 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811312+0000", 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811312+0000", 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811314+0000", 2024-03-26T03:18:49.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811325+0000", 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0745e-05 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811340+0000", 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4419e-05 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811348+0000", 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5760000000000006e-06 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040799+0000", 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22945073699999999 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040824+0000", 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5324999999999999e-05 2024-03-26T03:18:49.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.228786+0000", 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.187961355 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.228828+0000", 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2845000000000001e-05 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:138292 10.5 e345/341 10:a14f9faa:::benchmark_data_smithi066_148944_object13645:head v 345'8656, mlcod=0'0)", 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.188450+0000", 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 249.41349843, 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43925914700000002, 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.697 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188450+0000", 2024-03-26T03:18:49.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188450+0000", 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188451+0000", 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7300000000000004e-07 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188458+0000", 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4869999999999999e-06 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188466+0000", 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1510000000000004e-06 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188472+0000", 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5359999999999998e-06 2024-03-26T03:18:49.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188773+0000", 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000300635 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188795+0000", 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1716e-05 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627678+0000", 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43888337999999999 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627709+0000", 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1269000000000002e-05 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.702 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:138344 10.5 e345/341 10:a8c08bd4:::benchmark_data_smithi066_148944_object13697:head v 345'8657, mlcod=0'0)", 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.236829+0000", 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 249.36511994200001, 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39092764000000002, 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236829+0000", 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236829+0000", 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236831+0000", 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0540000000000002e-06 2024-03-26T03:18:49.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236841+0000", 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0234000000000001e-05 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236860+0000", 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9051000000000001e-05 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236869+0000", 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2890000000000003e-06 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.237100+0000", 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023038600000000001 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:18:49.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.237128+0000", 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8132000000000001e-05 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627731+0000", 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39060298199999999 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627756+0000", 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5511999999999999e-05 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:18:49.708 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:18:49.708 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:18:49.708 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:49.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:49.810 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.846 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:49.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:49.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:49 smithi066 ceph-mon[29666]: pgmap v941: 178 pgs: 1 peering, 2 active+remapped+backfilling, 175 active+clean; 7.6 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 865 op/s; 6130/375363 objects misplaced (1.633%); 20 MiB/s, 318 objects/s recovering 2024-03-26T03:18:49.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:49 smithi066 ceph-mon[29666]: 1.18 scrub starts 2024-03-26T03:18:49.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:49 smithi066 ceph-mon[29666]: 1.18 scrub ok 2024-03-26T03:18:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:49 smithi066 ceph-mon[37920]: pgmap v941: 178 pgs: 1 peering, 2 active+remapped+backfilling, 175 active+clean; 7.6 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 865 op/s; 6130/375363 objects misplaced (1.633%); 20 MiB/s, 318 objects/s recovering 2024-03-26T03:18:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:49 smithi066 ceph-mon[37920]: 1.18 scrub starts 2024-03-26T03:18:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:49 smithi066 ceph-mon[37920]: 1.18 scrub ok 2024-03-26T03:18:49.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:49.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:49.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:50.089 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.134 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:50.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:50.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:50.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:50.368 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:50.416 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:50.429 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:18:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:18:50.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:18:50.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:50.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:50.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:50.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:50 smithi192 ceph-mon[33110]: 6.0 scrub starts 2024-03-26T03:18:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:50 smithi192 ceph-mon[33110]: 6.0 scrub ok 2024-03-26T03:18:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:50 smithi192 ceph-mon[33110]: 11.b scrub ok 2024-03-26T03:18:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:50 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:18:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:50 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:18:50.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:50.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:50.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[29666]: 6.0 scrub starts 2024-03-26T03:18:50.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[29666]: 6.0 scrub ok 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[29666]: 11.b scrub ok 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[37920]: 6.0 scrub starts 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[37920]: 6.0 scrub ok 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[37920]: 11.b scrub ok 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:18:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:50 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:18:50.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:50.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:50.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:51.038 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.093 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:51.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:51.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:51.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:51.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:51.375 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:51.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:51.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:51.611 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:51.660 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:51.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:51.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:51.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:51.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[37920]: pgmap v942: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 1.32k op/s; 6130/363513 objects misplaced (1.686%); 17 MiB/s, 276 objects/s recovering 2024-03-26T03:18:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[37920]: 6.f scrub starts 2024-03-26T03:18:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[37920]: 11.7 scrub starts 2024-03-26T03:18:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[29666]: pgmap v942: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 1.32k op/s; 6130/363513 objects misplaced (1.686%); 17 MiB/s, 276 objects/s recovering 2024-03-26T03:18:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[29666]: 6.f scrub starts 2024-03-26T03:18:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:51 smithi066 ceph-mon[29666]: 11.7 scrub starts 2024-03-26T03:18:51.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:51.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:51.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:51.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:18:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:51 smithi192 ceph-mon[33110]: pgmap v942: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.4 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 1.32k op/s; 6130/363513 objects misplaced (1.686%); 17 MiB/s, 276 objects/s recovering 2024-03-26T03:18:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:51 smithi192 ceph-mon[33110]: 6.f scrub starts 2024-03-26T03:18:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:51 smithi192 ceph-mon[33110]: 11.7 scrub starts 2024-03-26T03:18:52.035 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.081 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:52.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:52.182 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:52.226 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:52.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:52.371 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:52.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:52.523 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:52.626 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.671 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:52.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:52.772 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:52.817 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:52.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:52 smithi066 ceph-mon[29666]: 6.f scrub ok 2024-03-26T03:18:52.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:52 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:18:52.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:52 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:18:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:52 smithi066 ceph-mon[37920]: 6.f scrub ok 2024-03-26T03:18:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:52 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:18:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:52 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:18:52.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:52.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:52.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:52 smithi192 ceph-mon[33110]: 6.f scrub ok 2024-03-26T03:18:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:52 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:18:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:52 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:18:53.053 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.111 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:53.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:53.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:53.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:53.344 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:53.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:53.493 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:53.545 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:53.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.691 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:53.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:53.792 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:53.830 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[29666]: pgmap v943: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 4724/358131 objects misplaced (1.319%); 24 MiB/s, 383 objects/s recovering 2024-03-26T03:18:53.830 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[29666]: 11.7 scrub ok 2024-03-26T03:18:53.830 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[29666]: 1.1b scrub starts 2024-03-26T03:18:53.830 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[29666]: 1.1b scrub ok 2024-03-26T03:18:53.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[37920]: pgmap v943: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 4724/358131 objects misplaced (1.319%); 24 MiB/s, 383 objects/s recovering 2024-03-26T03:18:53.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[37920]: 11.7 scrub ok 2024-03-26T03:18:53.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[37920]: 1.1b scrub starts 2024-03-26T03:18:53.831 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-mon[37920]: 1.1b scrub ok 2024-03-26T03:18:53.837 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:53.939 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:53.975 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:18:53.984 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:53.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:53 smithi192 ceph-mon[33110]: pgmap v943: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 4724/358131 objects misplaced (1.319%); 24 MiB/s, 383 objects/s recovering 2024-03-26T03:18:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:53 smithi192 ceph-mon[33110]: 11.7 scrub ok 2024-03-26T03:18:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:53 smithi192 ceph-mon[33110]: 1.1b scrub starts 2024-03-26T03:18:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:53 smithi192 ceph-mon[33110]: 1.1b scrub ok 2024-03-26T03:18:54.003 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:54.053 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:18:54.087 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:18:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:18:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:18:54.116 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:18:54.161 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:18:54.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:54.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:54.203 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:18:54.248 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:18:54.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:54.289 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:18:54.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:54.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:54.341 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:18:54.383 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:18:54.389 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:18:54.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:54.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:54.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:54.486 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:18:54.493 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:18:54.546 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:54.608 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:54.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:54.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:54.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:54.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:54.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[29666]: 1.6 scrub starts 2024-03-26T03:18:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[29666]: 1.6 scrub ok 2024-03-26T03:18:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[29666]: 11.5 scrub starts 2024-03-26T03:18:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4200794245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:54.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[37920]: 1.6 scrub starts 2024-03-26T03:18:54.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[37920]: 1.6 scrub ok 2024-03-26T03:18:54.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[37920]: 11.5 scrub starts 2024-03-26T03:18:54.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:54 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4200794245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:54.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:54.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:54 smithi192 ceph-mon[33110]: 1.6 scrub starts 2024-03-26T03:18:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:54 smithi192 ceph-mon[33110]: 1.6 scrub ok 2024-03-26T03:18:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:54 smithi192 ceph-mon[33110]: 11.5 scrub starts 2024-03-26T03:18:54.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:54 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4200794245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:18:54.990 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:55.034 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:55.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:55.179 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:55.282 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:55.331 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:55.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:55.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:55.514 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":944,"stamp":"2024-03-26T03:18:54.443021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7808962089,"num_objects":119377,"num_object_clones":0,"num_object_copies":358131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4724,"num_objects_unfound":0,"num_objects_dirty":119377,"num_whiteouts":0,"num_read":2603,"num_read_kb":2326,"num_write":145563,"num_write_kb":8441381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16652,"num_bytes_recovered":1076278497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57789,"ondisk_log_size":57789,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":536,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":28455624,"kb_used_data":23644696,"kb_used_omap":12,"kb_used_meta":4810675,"kb_avail":534015288,"statfs":{"total":575970213888,"available":546831654912,"internally_reserved":0,"allocated":24212168704,"data_stored":24209087704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12419,"internal_metadata":4926132093},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,10,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-716636160,"num_objects":-10935,"num_object_clones":0,"num_object_copies":-32805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3596,"num_objects_unfound":0,"num_objects_dirty":-10935,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3067,"num_bytes_recovered":200998912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.938416"},"pg_stats":[{"pgid":"11.7","version":"443'8988","reported_seq":9078,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.751667+0000","last_change":"2024-03-26T03:16:56.903159+0000","last_active":"2024-03-26T03:18:49.751667+0000","last_peered":"2024-03-26T03:18:49.751667+0000","last_clean":"2024-03-26T03:18:49.751667+0000","last_became_active":"2024-03-26T03:16:56.900984+0000","last_became_peered":"2024-03-26T03:16:56.900984+0000","last_unstale":"2024-03-26T03:18:49.751667+0000","last_undegraded":"2024-03-26T03:18:49.751667+0000","last_fullsized":"2024-03-26T03:18:49.751667+0000","mapping_epoch":403,"log_start":"432'5700","ondisk_log_start":"432'5700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3288,"log_dups_size":928,"ondisk_log_size":3288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486539264,"num_objects":7424,"num_object_clones":0,"num_object_copies":22272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8988,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":741,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249466+0000","last_change":"2024-03-26T03:16:31.121425+0000","last_active":"2024-03-26T03:18:46.249466+0000","last_peered":"2024-03-26T03:18:46.249466+0000","last_clean":"2024-03-26T03:18:46.249466+0000","last_became_active":"2024-03-26T03:14:40.174462+0000","last_became_peered":"2024-03-26T03:14:40.174462+0000","last_unstale":"2024-03-26T03:18:46.249466+0000","last_undegraded":"2024-03-26T03:18:46.249466+0000","last_fullsized":"2024-03-26T03:18:46.249466+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.121331+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.121331+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"342'10","reported_seq":432,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258520+0000","last_change":"2024-03-26T03:17:48.646060+0000","last_active":"2024-03-26T03:18:46.258520+0000","last_peered":"2024-03-26T03:18:46.258520+0000","last_clean":"2024-03-26T03:18:46.258520+0000","last_became_active":"2024-03-26T03:14:37.273000+0000","last_became_peered":"2024-03-26T03:14:37.273000+0000","last_unstale":"2024-03-26T03:18:46.258520+0000","last_undegraded":"2024-03-26T03:18:46.258520+0000","last_fullsized":"2024-03-26T03:18:46.258520+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'10","last_scrub_stamp":"2024-03-26T03:17:48.645977+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.645977+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:53.530710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"230'39","reported_seq":946,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261022+0000","last_change":"2024-03-26T03:16:51.697617+0000","last_active":"2024-03-26T03:18:46.261022+0000","last_peered":"2024-03-26T03:18:46.261022+0000","last_clean":"2024-03-26T03:18:46.261022+0000","last_became_active":"2024-03-26T03:14:37.267205+0000","last_became_peered":"2024-03-26T03:14:37.267205+0000","last_unstale":"2024-03-26T03:18:46.261022+0000","last_undegraded":"2024-03-26T03:18:46.261022+0000","last_fullsized":"2024-03-26T03:18:46.261022+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:16:51.697464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.697464+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.734018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":251,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259219+0000","last_change":"2024-03-26T03:17:57.225859+0000","last_active":"2024-03-26T03:18:46.259219+0000","last_peered":"2024-03-26T03:18:46.259219+0000","last_clean":"2024-03-26T03:18:46.259219+0000","last_became_active":"2024-03-26T03:14:37.264771+0000","last_became_peered":"2024-03-26T03:14:37.264771+0000","last_unstale":"2024-03-26T03:18:46.259219+0000","last_undegraded":"2024-03-26T03:18:46.259219+0000","last_fullsized":"2024-03-26T03:18:46.259219+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:57.708114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":656,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257970+0000","last_change":"2024-03-26T03:18:07.660598+0000","last_active":"2024-03-26T03:18:46.257970+0000","last_peered":"2024-03-26T03:18:46.257970+0000","last_clean":"2024-03-26T03:18:46.257970+0000","last_became_active":"2024-03-26T03:13:35.269469+0000","last_became_peered":"2024-03-26T03:13:35.269469+0000","last_unstale":"2024-03-26T03:18:46.257970+0000","last_undegraded":"2024-03-26T03:18:46.257970+0000","last_fullsized":"2024-03-26T03:18:46.257970+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_clean_scrub_stamp":"2024-03-26T03:18:07.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.629198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"443'8916","reported_seq":9006,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.697191+0000","last_change":"2024-03-26T03:16:56.903838+0000","last_active":"2024-03-26T03:18:49.697191+0000","last_peered":"2024-03-26T03:18:49.697191+0000","last_clean":"2024-03-26T03:18:49.697191+0000","last_became_active":"2024-03-26T03:16:56.903390+0000","last_became_peered":"2024-03-26T03:16:56.903390+0000","last_unstale":"2024-03-26T03:18:49.697191+0000","last_undegraded":"2024-03-26T03:18:49.697191+0000","last_fullsized":"2024-03-26T03:18:49.697191+0000","mapping_epoch":403,"log_start":"431'5400","ondisk_log_start":"431'5400","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481689600,"num_objects":7350,"num_object_clones":0,"num_object_copies":22050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":338,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249978+0000","last_change":"2024-03-26T03:17:59.592471+0000","last_active":"2024-03-26T03:18:46.249978+0000","last_peered":"2024-03-26T03:18:46.249978+0000","last_clean":"2024-03-26T03:18:46.249978+0000","last_became_active":"2024-03-26T03:17:59.592113+0000","last_became_peered":"2024-03-26T03:17:59.592113+0000","last_unstale":"2024-03-26T03:18:46.249978+0000","last_undegraded":"2024-03-26T03:18:46.249978+0000","last_fullsized":"2024-03-26T03:18:46.249978+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.778219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.778219+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:49.247451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"231'26","reported_seq":721,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253762+0000","last_change":"2024-03-26T03:17:38.464630+0000","last_active":"2024-03-26T03:18:46.253762+0000","last_peered":"2024-03-26T03:18:46.253762+0000","last_clean":"2024-03-26T03:18:46.253762+0000","last_became_active":"2024-03-26T03:14:22.579044+0000","last_became_peered":"2024-03-26T03:14:22.579044+0000","last_unstale":"2024-03-26T03:18:46.253762+0000","last_undegraded":"2024-03-26T03:18:46.253762+0000","last_fullsized":"2024-03-26T03:18:46.253762+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:17:38.464358+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:17:38.464358+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:59.515865+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":830,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257001+0000","last_change":"2024-03-26T03:17:56.571530+0000","last_active":"2024-03-26T03:18:46.257001+0000","last_peered":"2024-03-26T03:18:46.257001+0000","last_clean":"2024-03-26T03:18:46.257001+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:18:46.257001+0000","last_undegraded":"2024-03-26T03:18:46.257001+0000","last_fullsized":"2024-03-26T03:18:46.257001+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:16.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:15:16.933002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":41,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248942+0000","last_change":"2024-03-26T03:17:56.568649+0000","last_active":"2024-03-26T03:18:46.248942+0000","last_peered":"2024-03-26T03:18:46.248942+0000","last_clean":"2024-03-26T03:18:46.248942+0000","last_became_active":"2024-03-26T03:17:56.568322+0000","last_became_peered":"2024-03-26T03:17:56.568322+0000","last_unstale":"2024-03-26T03:18:46.248942+0000","last_undegraded":"2024-03-26T03:18:46.248942+0000","last_fullsized":"2024-03-26T03:18:46.248942+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.222175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.222175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:56.779239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":144,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257407+0000","last_change":"2024-03-26T03:16:50.206535+0000","last_active":"2024-03-26T03:18:46.257407+0000","last_peered":"2024-03-26T03:18:46.257407+0000","last_clean":"2024-03-26T03:18:46.257407+0000","last_became_active":"2024-03-26T03:16:50.206224+0000","last_became_peered":"2024-03-26T03:16:50.206224+0000","last_unstale":"2024-03-26T03:18:46.257407+0000","last_undegraded":"2024-03-26T03:18:46.257407+0000","last_fullsized":"2024-03-26T03:18:46.257407+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:26.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:13:26.722546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"443'9046","reported_seq":9136,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.795326+0000","last_change":"2024-03-26T03:16:56.908753+0000","last_active":"2024-03-26T03:18:49.795326+0000","last_peered":"2024-03-26T03:18:49.795326+0000","last_clean":"2024-03-26T03:18:49.795326+0000","last_became_active":"2024-03-26T03:16:56.908452+0000","last_became_peered":"2024-03-26T03:16:56.908452+0000","last_unstale":"2024-03-26T03:18:49.795326+0000","last_undegraded":"2024-03-26T03:18:49.795326+0000","last_fullsized":"2024-03-26T03:18:49.795326+0000","mapping_epoch":403,"log_start":"432'5800","ondisk_log_start":"432'5800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3246,"log_dups_size":986,"ondisk_log_size":3246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486146048,"num_objects":7418,"num_object_clones":0,"num_object_copies":22254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":917,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259090+0000","last_change":"2024-03-26T03:17:32.537859+0000","last_active":"2024-03-26T03:18:46.259090+0000","last_peered":"2024-03-26T03:18:46.259090+0000","last_clean":"2024-03-26T03:18:46.259090+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:18:46.259090+0000","last_undegraded":"2024-03-26T03:18:46.259090+0000","last_fullsized":"2024-03-26T03:18:46.259090+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:32.537761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.537761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"231'41","reported_seq":961,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258245+0000","last_change":"2024-03-26T03:18:02.671958+0000","last_active":"2024-03-26T03:18:46.258245+0000","last_peered":"2024-03-26T03:18:46.258245+0000","last_clean":"2024-03-26T03:18:46.258245+0000","last_became_active":"2024-03-26T03:14:37.266473+0000","last_became_peered":"2024-03-26T03:14:37.266473+0000","last_unstale":"2024-03-26T03:18:46.258245+0000","last_undegraded":"2024-03-26T03:18:46.258245+0000","last_fullsized":"2024-03-26T03:18:46.258245+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:18:02.671821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:02.671821+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:20.660317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":53,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":690,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.254682+0000","last_change":"2024-03-26T03:17:54.517355+0000","last_active":"2024-03-26T03:18:46.254682+0000","last_peered":"2024-03-26T03:18:46.254682+0000","last_clean":"2024-03-26T03:18:46.254682+0000","last_became_active":"2024-03-26T03:16:50.208923+0000","last_became_peered":"2024-03-26T03:16:50.208923+0000","last_unstale":"2024-03-26T03:18:46.254682+0000","last_undegraded":"2024-03-26T03:18:46.254682+0000","last_fullsized":"2024-03-26T03:18:46.254682+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:54.517230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.517230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:09.950125+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":861,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256998+0000","last_change":"2024-03-26T03:18:05.489195+0000","last_active":"2024-03-26T03:18:46.256998+0000","last_peered":"2024-03-26T03:18:46.256998+0000","last_clean":"2024-03-26T03:18:46.256998+0000","last_became_active":"2024-03-26T03:14:37.242217+0000","last_became_peered":"2024-03-26T03:14:37.242217+0000","last_unstale":"2024-03-26T03:18:46.256998+0000","last_undegraded":"2024-03-26T03:18:46.256998+0000","last_fullsized":"2024-03-26T03:18:46.256998+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:05.489069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.227664+0000","last_clean_scrub_stamp":"2024-03-26T03:18:05.489069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:21.866500+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":781,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.260310+0000","last_change":"2024-03-26T03:17:40.190354+0000","last_active":"2024-03-26T03:18:46.260310+0000","last_peered":"2024-03-26T03:18:46.260310+0000","last_clean":"2024-03-26T03:18:46.260310+0000","last_became_active":"2024-03-26T03:14:37.270129+0000","last_became_peered":"2024-03-26T03:14:37.270129+0000","last_unstale":"2024-03-26T03:18:46.260310+0000","last_undegraded":"2024-03-26T03:18:46.260310+0000","last_fullsized":"2024-03-26T03:18:46.260310+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.189873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.189873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:47.578903+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"443'9161","reported_seq":9251,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.492608+0000","last_change":"2024-03-26T03:16:56.905137+0000","last_active":"2024-03-26T03:18:49.492608+0000","last_peered":"2024-03-26T03:18:49.492608+0000","last_clean":"2024-03-26T03:18:49.492608+0000","last_became_active":"2024-03-26T03:16:56.904798+0000","last_became_peered":"2024-03-26T03:16:56.904798+0000","last_unstale":"2024-03-26T03:18:49.492608+0000","last_undegraded":"2024-03-26T03:18:49.492608+0000","last_fullsized":"2024-03-26T03:18:49.492608+0000","mapping_epoch":403,"log_start":"432'6000","ondisk_log_start":"432'6000","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3161,"log_dups_size":2487,"ondisk_log_size":3161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493944832,"num_objects":7537,"num_object_clones":0,"num_object_copies":22611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9161,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"231'39","reported_seq":59,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249370+0000","last_change":"2024-03-26T03:17:56.585807+0000","last_active":"2024-03-26T03:18:46.249370+0000","last_peered":"2024-03-26T03:18:46.249370+0000","last_clean":"2024-03-26T03:18:46.249370+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:18:46.249370+0000","last_undegraded":"2024-03-26T03:18:46.249370+0000","last_fullsized":"2024-03-26T03:18:46.249370+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:17:36.443102+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:17:36.443102+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.831709+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":41,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249314+0000","last_change":"2024-03-26T03:17:56.568076+0000","last_active":"2024-03-26T03:18:46.249314+0000","last_peered":"2024-03-26T03:18:46.249314+0000","last_clean":"2024-03-26T03:18:46.249314+0000","last_became_active":"2024-03-26T03:17:56.567051+0000","last_became_peered":"2024-03-26T03:17:56.567051+0000","last_unstale":"2024-03-26T03:18:46.249314+0000","last_undegraded":"2024-03-26T03:18:46.249314+0000","last_fullsized":"2024-03-26T03:18:46.249314+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.464363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.464363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":831,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259266+0000","last_change":"2024-03-26T03:16:46.947202+0000","last_active":"2024-03-26T03:18:46.259266+0000","last_peered":"2024-03-26T03:18:46.259266+0000","last_clean":"2024-03-26T03:18:46.259266+0000","last_became_active":"2024-03-26T03:13:35.277164+0000","last_became_peered":"2024-03-26T03:13:35.277164+0000","last_unstale":"2024-03-26T03:18:46.259266+0000","last_undegraded":"2024-03-26T03:18:46.259266+0000","last_fullsized":"2024-03-26T03:18:46.259266+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:46.947118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.947118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":237,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257607+0000","last_change":"2024-03-26T03:14:21.588254+0000","last_active":"2024-03-26T03:18:46.257607+0000","last_peered":"2024-03-26T03:18:46.257607+0000","last_clean":"2024-03-26T03:18:46.257607+0000","last_became_active":"2024-03-26T03:14:21.586859+0000","last_became_peered":"2024-03-26T03:14:21.586859+0000","last_unstale":"2024-03-26T03:18:46.257607+0000","last_undegraded":"2024-03-26T03:18:46.257607+0000","last_fullsized":"2024-03-26T03:18:46.257607+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:29.490559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:29.490559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581029+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":540,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255254+0000","last_change":"2024-03-26T03:16:28.866577+0000","last_active":"2024-03-26T03:18:46.255254+0000","last_peered":"2024-03-26T03:18:46.255254+0000","last_clean":"2024-03-26T03:18:46.255254+0000","last_became_active":"2024-03-26T03:14:21.590629+0000","last_became_peered":"2024-03-26T03:14:21.590629+0000","last_unstale":"2024-03-26T03:18:46.255254+0000","last_undegraded":"2024-03-26T03:18:46.255254+0000","last_fullsized":"2024-03-26T03:18:46.255254+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:28.866477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.866477+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.605048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"443'8952","reported_seq":9042,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.669819+0000","last_change":"2024-03-26T03:16:56.903756+0000","last_active":"2024-03-26T03:18:49.669819+0000","last_peered":"2024-03-26T03:18:49.669819+0000","last_clean":"2024-03-26T03:18:49.669819+0000","last_became_active":"2024-03-26T03:16:56.903414+0000","last_became_peered":"2024-03-26T03:16:56.903414+0000","last_unstale":"2024-03-26T03:18:49.669819+0000","last_undegraded":"2024-03-26T03:18:49.669819+0000","last_fullsized":"2024-03-26T03:18:49.669819+0000","mapping_epoch":403,"log_start":"432'5500","ondisk_log_start":"432'5500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486408192,"num_objects":7422,"num_object_clones":0,"num_object_copies":22266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":706,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255341+0000","last_change":"2024-03-26T03:16:30.821771+0000","last_active":"2024-03-26T03:18:46.255341+0000","last_peered":"2024-03-26T03:18:46.255341+0000","last_clean":"2024-03-26T03:18:46.255341+0000","last_became_active":"2024-03-26T03:14:40.169332+0000","last_became_peered":"2024-03-26T03:14:40.169332+0000","last_unstale":"2024-03-26T03:18:46.255341+0000","last_undegraded":"2024-03-26T03:18:46.255341+0000","last_fullsized":"2024-03-26T03:18:46.255341+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.821670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.821670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.605871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"230'20","reported_seq":855,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249804+0000","last_change":"2024-03-26T03:16:39.220608+0000","last_active":"2024-03-26T03:18:46.249804+0000","last_peered":"2024-03-26T03:18:46.249804+0000","last_clean":"2024-03-26T03:18:46.249804+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:18:46.249804+0000","last_undegraded":"2024-03-26T03:18:46.249804+0000","last_fullsized":"2024-03-26T03:18:46.249804+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'20","last_scrub_stamp":"2024-03-26T03:16:39.220501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.220501+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":817,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253226+0000","last_change":"2024-03-26T03:15:09.812769+0000","last_active":"2024-03-26T03:18:46.253226+0000","last_peered":"2024-03-26T03:18:46.253226+0000","last_clean":"2024-03-26T03:18:46.253226+0000","last_became_active":"2024-03-26T03:14:37.251633+0000","last_became_peered":"2024-03-26T03:14:37.251633+0000","last_unstale":"2024-03-26T03:18:46.253226+0000","last_undegraded":"2024-03-26T03:18:46.253226+0000","last_fullsized":"2024-03-26T03:18:46.253226+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:09.812652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:15:09.812652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"401'11","reported_seq":943,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258114+0000","last_change":"2024-03-26T03:16:50.207177+0000","last_active":"2024-03-26T03:18:46.258114+0000","last_peered":"2024-03-26T03:18:46.258114+0000","last_clean":"2024-03-26T03:18:46.258114+0000","last_became_active":"2024-03-26T03:16:50.206797+0000","last_became_peered":"2024-03-26T03:16:50.206797+0000","last_unstale":"2024-03-26T03:18:46.258114+0000","last_undegraded":"2024-03-26T03:18:46.258114+0000","last_fullsized":"2024-03-26T03:18:46.258114+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-26T03:12:28.903038+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:12:28.903038+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":41,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250056+0000","last_change":"2024-03-26T03:17:56.574248+0000","last_active":"2024-03-26T03:18:46.250056+0000","last_peered":"2024-03-26T03:18:46.250056+0000","last_clean":"2024-03-26T03:18:46.250056+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:18:46.250056+0000","last_undegraded":"2024-03-26T03:18:46.250056+0000","last_fullsized":"2024-03-26T03:18:46.250056+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:32.853807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.853807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"443'8981","reported_seq":9070,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:50.552747+0000","last_change":"2024-03-26T03:16:56.909076+0000","last_active":"2024-03-26T03:18:50.552747+0000","last_peered":"2024-03-26T03:18:50.552747+0000","last_clean":"2024-03-26T03:18:50.552747+0000","last_became_active":"2024-03-26T03:16:56.908486+0000","last_became_peered":"2024-03-26T03:16:56.908486+0000","last_unstale":"2024-03-26T03:18:50.552747+0000","last_undegraded":"2024-03-26T03:18:50.552747+0000","last_fullsized":"2024-03-26T03:18:50.552747+0000","mapping_epoch":403,"log_start":"432'5700","ondisk_log_start":"432'5700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479526912,"num_objects":7317,"num_object_clones":0,"num_object_copies":21951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":396,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257925+0000","last_change":"2024-03-26T03:16:50.672364+0000","last_active":"2024-03-26T03:18:46.257925+0000","last_peered":"2024-03-26T03:18:46.257925+0000","last_clean":"2024-03-26T03:18:46.257925+0000","last_became_active":"2024-03-26T03:16:50.205474+0000","last_became_peered":"2024-03-26T03:16:50.205474+0000","last_unstale":"2024-03-26T03:18:46.257925+0000","last_undegraded":"2024-03-26T03:18:46.257925+0000","last_fullsized":"2024-03-26T03:18:46.257925+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:50.672268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.672268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"231'27","reported_seq":880,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253686+0000","last_change":"2024-03-26T03:16:28.788850+0000","last_active":"2024-03-26T03:18:46.253686+0000","last_peered":"2024-03-26T03:18:46.253686+0000","last_clean":"2024-03-26T03:18:46.253686+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:18:46.253686+0000","last_undegraded":"2024-03-26T03:18:46.253686+0000","last_fullsized":"2024-03-26T03:18:46.253686+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'27","last_scrub_stamp":"2024-03-26T03:16:28.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.788749+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":817,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253066+0000","last_change":"2024-03-26T03:14:39.896972+0000","last_active":"2024-03-26T03:18:46.253066+0000","last_peered":"2024-03-26T03:18:46.253066+0000","last_clean":"2024-03-26T03:18:46.253066+0000","last_became_active":"2024-03-26T03:14:37.249790+0000","last_became_peered":"2024-03-26T03:14:37.249790+0000","last_unstale":"2024-03-26T03:18:46.253066+0000","last_undegraded":"2024-03-26T03:18:46.253066+0000","last_fullsized":"2024-03-26T03:18:46.253066+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:39.896823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:14:39.896823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"428'12","reported_seq":57,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250143+0000","last_change":"2024-03-26T03:17:56.575479+0000","last_active":"2024-03-26T03:18:46.250143+0000","last_peered":"2024-03-26T03:18:46.250143+0000","last_clean":"2024-03-26T03:18:46.250143+0000","last_became_active":"2024-03-26T03:17:56.566007+0000","last_became_peered":"2024-03-26T03:17:56.566007+0000","last_unstale":"2024-03-26T03:18:46.250143+0000","last_undegraded":"2024-03-26T03:18:46.250143+0000","last_fullsized":"2024-03-26T03:18:46.250143+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:13:30.083400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:30.083400+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":632,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.254984+0000","last_change":"2024-03-26T03:13:35.276632+0000","last_active":"2024-03-26T03:18:46.254984+0000","last_peered":"2024-03-26T03:18:46.254984+0000","last_clean":"2024-03-26T03:18:46.254984+0000","last_became_active":"2024-03-26T03:13:35.275396+0000","last_became_peered":"2024-03-26T03:13:35.275396+0000","last_unstale":"2024-03-26T03:18:46.254984+0000","last_undegraded":"2024-03-26T03:18:46.254984+0000","last_fullsized":"2024-03-26T03:18:46.254984+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:18.063340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:13:18.063340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"443'9043","reported_seq":9133,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.743614+0000","last_change":"2024-03-26T03:16:56.905918+0000","last_active":"2024-03-26T03:18:49.743614+0000","last_peered":"2024-03-26T03:18:49.743614+0000","last_clean":"2024-03-26T03:18:49.743614+0000","last_became_active":"2024-03-26T03:16:56.905606+0000","last_became_peered":"2024-03-26T03:16:56.905606+0000","last_unstale":"2024-03-26T03:18:49.743614+0000","last_undegraded":"2024-03-26T03:18:49.743614+0000","last_fullsized":"2024-03-26T03:18:49.743614+0000","mapping_epoch":403,"log_start":"431'5500","ondisk_log_start":"431'5500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3543,"log_dups_size":0,"ondisk_log_size":3543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489357312,"num_objects":7467,"num_object_clones":0,"num_object_copies":22401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":957,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253454+0000","last_change":"2024-03-26T03:17:39.489433+0000","last_active":"2024-03-26T03:18:46.253454+0000","last_peered":"2024-03-26T03:18:46.253454+0000","last_clean":"2024-03-26T03:18:46.253454+0000","last_became_active":"2024-03-26T03:13:38.314531+0000","last_became_peered":"2024-03-26T03:13:38.314531+0000","last_unstale":"2024-03-26T03:18:46.253454+0000","last_undegraded":"2024-03-26T03:18:46.253454+0000","last_fullsized":"2024-03-26T03:18:46.253454+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:39.489210+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:17:39.489210+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.198140+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"291'31","reported_seq":884,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250370+0000","last_change":"2024-03-26T03:17:42.061990+0000","last_active":"2024-03-26T03:18:46.250370+0000","last_peered":"2024-03-26T03:18:46.250370+0000","last_clean":"2024-03-26T03:18:46.250370+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:18:46.250370+0000","last_undegraded":"2024-03-26T03:18:46.250370+0000","last_fullsized":"2024-03-26T03:18:46.250370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:17:42.061780+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.061780+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.768905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":785,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257516+0000","last_change":"2024-03-26T03:14:37.253086+0000","last_active":"2024-03-26T03:18:46.257516+0000","last_peered":"2024-03-26T03:18:46.257516+0000","last_clean":"2024-03-26T03:18:46.257516+0000","last_became_active":"2024-03-26T03:14:37.252745+0000","last_became_peered":"2024-03-26T03:14:37.252745+0000","last_unstale":"2024-03-26T03:18:46.257516+0000","last_undegraded":"2024-03-26T03:18:46.257516+0000","last_fullsized":"2024-03-26T03:18:46.257516+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:31.980008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.980008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":637,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250373+0000","last_change":"2024-03-26T03:16:55.201337+0000","last_active":"2024-03-26T03:18:46.250373+0000","last_peered":"2024-03-26T03:18:46.250373+0000","last_clean":"2024-03-26T03:18:46.250373+0000","last_became_active":"2024-03-26T03:14:37.249741+0000","last_became_peered":"2024-03-26T03:14:37.249741+0000","last_unstale":"2024-03-26T03:18:46.250373+0000","last_undegraded":"2024-03-26T03:18:46.250373+0000","last_fullsized":"2024-03-26T03:18:46.250373+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.201190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.201190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":459,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256110+0000","last_change":"2024-03-26T03:17:44.249680+0000","last_active":"2024-03-26T03:18:46.256110+0000","last_peered":"2024-03-26T03:18:46.256110+0000","last_clean":"2024-03-26T03:18:46.256110+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:18:46.256110+0000","last_undegraded":"2024-03-26T03:18:46.256110+0000","last_fullsized":"2024-03-26T03:18:46.256110+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.249561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.249561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:03.829651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"443'8943","reported_seq":21645,"reported_epoch":443,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:18:50.581679+0000","last_change":"2024-03-26T03:17:57.690462+0000","last_active":"2024-03-26T03:18:50.581679+0000","last_peered":"2024-03-26T03:18:50.581679+0000","last_clean":"2024-03-26T03:17:55.558348+0000","last_became_active":"2024-03-26T03:17:57.680052+0000","last_became_peered":"2024-03-26T03:17:57.680052+0000","last_unstale":"2024-03-26T03:18:50.581679+0000","last_undegraded":"2024-03-26T03:18:50.581679+0000","last_fullsized":"2024-03-26T03:18:50.581679+0000","mapping_epoch":428,"log_start":"432'5700","ondisk_log_start":"432'5700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477822976,"num_objects":7291,"num_object_clones":0,"num_object_copies":21873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":647,"num_objects_unfound":0,"num_objects_dirty":7291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8943,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6304,"num_bytes_recovered":413138944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":7291}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":639,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255190+0000","last_change":"2024-03-26T03:17:55.204987+0000","last_active":"2024-03-26T03:18:46.255190+0000","last_peered":"2024-03-26T03:18:46.255190+0000","last_clean":"2024-03-26T03:18:46.255190+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:18:46.255190+0000","last_undegraded":"2024-03-26T03:18:46.255190+0000","last_fullsized":"2024-03-26T03:18:46.255190+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:17:55.204836+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.204836+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:56.241884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"231'34","reported_seq":890,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248846+0000","last_change":"2024-03-26T03:16:56.199180+0000","last_active":"2024-03-26T03:18:46.248846+0000","last_peered":"2024-03-26T03:18:46.248846+0000","last_clean":"2024-03-26T03:18:46.248846+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:18:46.248846+0000","last_undegraded":"2024-03-26T03:18:46.248846+0000","last_fullsized":"2024-03-26T03:18:46.248846+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:16:56.199057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.199057+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":848,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259399+0000","last_change":"2024-03-26T03:17:41.533036+0000","last_active":"2024-03-26T03:18:46.259399+0000","last_peered":"2024-03-26T03:18:46.259399+0000","last_clean":"2024-03-26T03:18:46.259399+0000","last_became_active":"2024-03-26T03:14:37.276008+0000","last_became_peered":"2024-03-26T03:14:37.276008+0000","last_unstale":"2024-03-26T03:18:46.259399+0000","last_undegraded":"2024-03-26T03:18:46.259399+0000","last_fullsized":"2024-03-26T03:18:46.259399+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.532899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.532899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":224,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259072+0000","last_change":"2024-03-26T03:16:40.872720+0000","last_active":"2024-03-26T03:18:46.259072+0000","last_peered":"2024-03-26T03:18:46.259072+0000","last_clean":"2024-03-26T03:18:46.259072+0000","last_became_active":"2024-03-26T03:14:37.248955+0000","last_became_peered":"2024-03-26T03:14:37.248955+0000","last_unstale":"2024-03-26T03:18:46.259072+0000","last_undegraded":"2024-03-26T03:18:46.259072+0000","last_fullsized":"2024-03-26T03:18:46.259072+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:40.872619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.872619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":629,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256765+0000","last_change":"2024-03-26T03:16:29.852173+0000","last_active":"2024-03-26T03:18:46.256765+0000","last_peered":"2024-03-26T03:18:46.256765+0000","last_clean":"2024-03-26T03:18:46.256765+0000","last_became_active":"2024-03-26T03:14:40.166583+0000","last_became_peered":"2024-03-26T03:14:40.166583+0000","last_unstale":"2024-03-26T03:18:46.256765+0000","last_undegraded":"2024-03-26T03:18:46.256765+0000","last_fullsized":"2024-03-26T03:18:46.256765+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.852073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.852073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"443'9220","reported_seq":9310,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:50.560271+0000","last_change":"2024-03-26T03:16:56.900055+0000","last_active":"2024-03-26T03:18:50.560271+0000","last_peered":"2024-03-26T03:18:50.560271+0000","last_clean":"2024-03-26T03:18:50.560271+0000","last_became_active":"2024-03-26T03:16:56.899679+0000","last_became_peered":"2024-03-26T03:16:56.899679+0000","last_unstale":"2024-03-26T03:18:50.560271+0000","last_undegraded":"2024-03-26T03:18:50.560271+0000","last_fullsized":"2024-03-26T03:18:50.560271+0000","mapping_epoch":403,"log_start":"432'5900","ondisk_log_start":"432'5900","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3320,"log_dups_size":0,"ondisk_log_size":3320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":493092864,"num_objects":7524,"num_object_clones":0,"num_object_copies":22572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":535808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":807,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248948+0000","last_change":"2024-03-26T03:16:36.206735+0000","last_active":"2024-03-26T03:18:46.248948+0000","last_peered":"2024-03-26T03:18:46.248948+0000","last_clean":"2024-03-26T03:18:46.248948+0000","last_became_active":"2024-03-26T03:14:37.257004+0000","last_became_peered":"2024-03-26T03:14:37.257004+0000","last_unstale":"2024-03-26T03:18:46.248948+0000","last_undegraded":"2024-03-26T03:18:46.248948+0000","last_fullsized":"2024-03-26T03:18:46.248948+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:36.206634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.206634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"230'54","reported_seq":623,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.260400+0000","last_change":"2024-03-26T03:16:38.931105+0000","last_active":"2024-03-26T03:18:46.260400+0000","last_peered":"2024-03-26T03:18:46.260400+0000","last_clean":"2024-03-26T03:18:46.260400+0000","last_became_active":"2024-03-26T03:14:37.278227+0000","last_became_peered":"2024-03-26T03:14:37.278227+0000","last_unstale":"2024-03-26T03:18:46.260400+0000","last_undegraded":"2024-03-26T03:18:46.260400+0000","last_fullsized":"2024-03-26T03:18:46.260400+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'54","last_scrub_stamp":"2024-03-26T03:16:38.931005+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.931005+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":836,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261913+0000","last_change":"2024-03-26T03:13:29.030426+0000","last_active":"2024-03-26T03:18:46.261913+0000","last_peered":"2024-03-26T03:18:46.261913+0000","last_clean":"2024-03-26T03:18:46.261913+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:18:46.261913+0000","last_undegraded":"2024-03-26T03:18:46.261913+0000","last_fullsized":"2024-03-26T03:18:46.261913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:13:29.030298+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.030298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249012+0000","last_change":"2024-03-26T03:17:56.567975+0000","last_active":"2024-03-26T03:18:46.249012+0000","last_peered":"2024-03-26T03:18:46.249012+0000","last_clean":"2024-03-26T03:18:46.249012+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:18:46.249012+0000","last_undegraded":"2024-03-26T03:18:46.249012+0000","last_fullsized":"2024-03-26T03:18:46.249012+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.179410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.179410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:09.819892+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":940,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249493+0000","last_change":"2024-03-26T03:17:54.095189+0000","last_active":"2024-03-26T03:18:46.249493+0000","last_peered":"2024-03-26T03:18:46.249493+0000","last_clean":"2024-03-26T03:18:46.249493+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:18:46.249493+0000","last_undegraded":"2024-03-26T03:18:46.249493+0000","last_fullsized":"2024-03-26T03:18:46.249493+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:54.095058+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.095058+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:01.417606+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"443'8944","reported_seq":15736,"reported_epoch":443,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:18:50.581251+0000","last_change":"2024-03-26T03:18:29.644500+0000","last_active":"2024-03-26T03:18:50.581251+0000","last_peered":"2024-03-26T03:18:50.581251+0000","last_clean":"2024-03-26T03:17:55.548043+0000","last_became_active":"2024-03-26T03:17:57.671666+0000","last_became_peered":"2024-03-26T03:17:57.671666+0000","last_unstale":"2024-03-26T03:18:50.581251+0000","last_undegraded":"2024-03-26T03:18:50.581251+0000","last_fullsized":"2024-03-26T03:18:50.581251+0000","mapping_epoch":428,"log_start":"432'5700","ondisk_log_start":"432'5700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478806016,"num_objects":7306,"num_object_clones":0,"num_object_copies":21918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4077,"num_objects_unfound":0,"num_objects_dirty":7306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":219152384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":7306}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":840,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258346+0000","last_change":"2024-03-26T03:17:34.573811+0000","last_active":"2024-03-26T03:18:46.258346+0000","last_peered":"2024-03-26T03:18:46.258346+0000","last_clean":"2024-03-26T03:18:46.258346+0000","last_became_active":"2024-03-26T03:13:35.269721+0000","last_became_peered":"2024-03-26T03:13:35.269721+0000","last_unstale":"2024-03-26T03:18:46.258346+0000","last_undegraded":"2024-03-26T03:18:46.258346+0000","last_fullsized":"2024-03-26T03:18:46.258346+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.573719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.573719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"231'35","reported_seq":772,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262337+0000","last_change":"2024-03-26T03:16:56.710503+0000","last_active":"2024-03-26T03:18:46.262337+0000","last_peered":"2024-03-26T03:18:46.262337+0000","last_clean":"2024-03-26T03:18:46.262337+0000","last_became_active":"2024-03-26T03:14:37.258677+0000","last_became_peered":"2024-03-26T03:14:37.258677+0000","last_unstale":"2024-03-26T03:18:46.262337+0000","last_undegraded":"2024-03-26T03:18:46.262337+0000","last_fullsized":"2024-03-26T03:18:46.262337+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:16:56.710408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.710408+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.734131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":47,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":771,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249666+0000","last_change":"2024-03-26T03:14:37.257762+0000","last_active":"2024-03-26T03:18:46.249666+0000","last_peered":"2024-03-26T03:18:46.249666+0000","last_clean":"2024-03-26T03:18:46.249666+0000","last_became_active":"2024-03-26T03:14:37.257422+0000","last_became_peered":"2024-03-26T03:14:37.257422+0000","last_unstale":"2024-03-26T03:18:46.249666+0000","last_undegraded":"2024-03-26T03:18:46.249666+0000","last_fullsized":"2024-03-26T03:18:46.249666+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:35.954172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:35.954172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":825,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253262+0000","last_change":"2024-03-26T03:13:24.055056+0000","last_active":"2024-03-26T03:18:46.253262+0000","last_peered":"2024-03-26T03:18:46.253262+0000","last_clean":"2024-03-26T03:18:46.253262+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:18:46.253262+0000","last_undegraded":"2024-03-26T03:18:46.253262+0000","last_fullsized":"2024-03-26T03:18:46.253262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:24.054721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:13:24.054721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":910,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259060+0000","last_change":"2024-03-26T03:16:27.907348+0000","last_active":"2024-03-26T03:18:46.259060+0000","last_peered":"2024-03-26T03:18:46.259060+0000","last_clean":"2024-03-26T03:18:46.259060+0000","last_became_active":"2024-03-26T03:14:22.569851+0000","last_became_peered":"2024-03-26T03:14:22.569851+0000","last_unstale":"2024-03-26T03:18:46.259060+0000","last_undegraded":"2024-03-26T03:18:46.259060+0000","last_fullsized":"2024-03-26T03:18:46.259060+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:27.907229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:27.907229+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"443'8998","reported_seq":9088,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.504172+0000","last_change":"2024-03-26T03:16:56.907700+0000","last_active":"2024-03-26T03:18:49.504172+0000","last_peered":"2024-03-26T03:18:49.504172+0000","last_clean":"2024-03-26T03:18:49.504172+0000","last_became_active":"2024-03-26T03:16:56.907183+0000","last_became_peered":"2024-03-26T03:16:56.907183+0000","last_unstale":"2024-03-26T03:18:49.504172+0000","last_undegraded":"2024-03-26T03:18:49.504172+0000","last_fullsized":"2024-03-26T03:18:49.504172+0000","mapping_epoch":403,"log_start":"432'5800","ondisk_log_start":"432'5800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3198,"log_dups_size":2387,"ondisk_log_size":3198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489553920,"num_objects":7470,"num_object_clones":0,"num_object_copies":22410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":526976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":855,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.263043+0000","last_change":"2024-03-26T03:17:42.743976+0000","last_active":"2024-03-26T03:18:46.263043+0000","last_peered":"2024-03-26T03:18:46.263043+0000","last_clean":"2024-03-26T03:18:46.263043+0000","last_became_active":"2024-03-26T03:13:35.271824+0000","last_became_peered":"2024-03-26T03:13:35.271824+0000","last_unstale":"2024-03-26T03:18:46.263043+0000","last_undegraded":"2024-03-26T03:18:46.263043+0000","last_fullsized":"2024-03-26T03:18:46.263043+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:42.743853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.743853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:57.009889+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"230'30","reported_seq":729,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262200+0000","last_change":"2024-03-26T03:14:37.277935+0000","last_active":"2024-03-26T03:18:46.262200+0000","last_peered":"2024-03-26T03:18:46.262200+0000","last_clean":"2024-03-26T03:18:46.262200+0000","last_became_active":"2024-03-26T03:14:37.246803+0000","last_became_peered":"2024-03-26T03:14:37.246803+0000","last_unstale":"2024-03-26T03:18:46.262200+0000","last_undegraded":"2024-03-26T03:18:46.262200+0000","last_fullsized":"2024-03-26T03:18:46.262200+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:13:15.467975+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.467975+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":760,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257629+0000","last_change":"2024-03-26T03:16:56.019977+0000","last_active":"2024-03-26T03:18:46.257629+0000","last_peered":"2024-03-26T03:18:46.257629+0000","last_clean":"2024-03-26T03:18:46.257629+0000","last_became_active":"2024-03-26T03:14:37.265520+0000","last_became_peered":"2024-03-26T03:14:37.265520+0000","last_unstale":"2024-03-26T03:18:46.257629+0000","last_undegraded":"2024-03-26T03:18:46.257629+0000","last_fullsized":"2024-03-26T03:18:46.257629+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:56.019845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.019845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248684+0000","last_change":"2024-03-26T03:17:56.568202+0000","last_active":"2024-03-26T03:18:46.248684+0000","last_peered":"2024-03-26T03:18:46.248684+0000","last_clean":"2024-03-26T03:18:46.248684+0000","last_became_active":"2024-03-26T03:17:56.567590+0000","last_became_peered":"2024-03-26T03:17:56.567590+0000","last_unstale":"2024-03-26T03:18:46.248684+0000","last_undegraded":"2024-03-26T03:18:46.248684+0000","last_fullsized":"2024-03-26T03:18:46.248684+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":807,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259355+0000","last_change":"2024-03-26T03:15:17.827310+0000","last_active":"2024-03-26T03:18:46.259355+0000","last_peered":"2024-03-26T03:18:46.259355+0000","last_clean":"2024-03-26T03:18:46.259355+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:18:46.259355+0000","last_undegraded":"2024-03-26T03:18:46.259355+0000","last_fullsized":"2024-03-26T03:18:46.259355+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:17.827186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:15:17.827186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":946,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253556+0000","last_change":"2024-03-26T03:17:59.603321+0000","last_active":"2024-03-26T03:18:46.253556+0000","last_peered":"2024-03-26T03:18:46.253556+0000","last_clean":"2024-03-26T03:18:46.253556+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:18:46.253556+0000","last_undegraded":"2024-03-26T03:18:46.253556+0000","last_fullsized":"2024-03-26T03:18:46.253556+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:43.863351+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:43.863351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"443'8960","reported_seq":9050,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.489413+0000","last_change":"2024-03-26T03:16:56.907179+0000","last_active":"2024-03-26T03:18:49.489413+0000","last_peered":"2024-03-26T03:18:49.489413+0000","last_clean":"2024-03-26T03:18:49.489413+0000","last_became_active":"2024-03-26T03:16:56.906797+0000","last_became_peered":"2024-03-26T03:16:56.906797+0000","last_unstale":"2024-03-26T03:18:49.489413+0000","last_undegraded":"2024-03-26T03:18:49.489413+0000","last_fullsized":"2024-03-26T03:18:49.489413+0000","mapping_epoch":403,"log_start":"432'5800","ondisk_log_start":"432'5800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3160,"log_dups_size":2387,"ondisk_log_size":3160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489684992,"num_objects":7472,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":98,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.254591+0000","last_change":"2024-03-26T03:16:50.209616+0000","last_active":"2024-03-26T03:18:46.254591+0000","last_peered":"2024-03-26T03:18:46.254591+0000","last_clean":"2024-03-26T03:18:46.254591+0000","last_became_active":"2024-03-26T03:16:50.208883+0000","last_became_peered":"2024-03-26T03:16:50.208883+0000","last_unstale":"2024-03-26T03:18:46.254591+0000","last_undegraded":"2024-03-26T03:18:46.254591+0000","last_fullsized":"2024-03-26T03:18:46.254591+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.231346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.231346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"342'29","reported_seq":1141,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.260039+0000","last_change":"2024-03-26T03:16:54.697496+0000","last_active":"2024-03-26T03:18:46.260039+0000","last_peered":"2024-03-26T03:18:46.260039+0000","last_clean":"2024-03-26T03:18:46.260039+0000","last_became_active":"2024-03-26T03:14:37.265178+0000","last_became_peered":"2024-03-26T03:14:37.265178+0000","last_unstale":"2024-03-26T03:18:46.260039+0000","last_undegraded":"2024-03-26T03:18:46.260039+0000","last_fullsized":"2024-03-26T03:18:46.260039+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'29","last_scrub_stamp":"2024-03-26T03:16:54.697359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:54.697359+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":519,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257894+0000","last_change":"2024-03-26T03:16:31.804230+0000","last_active":"2024-03-26T03:18:46.257894+0000","last_peered":"2024-03-26T03:18:46.257894+0000","last_clean":"2024-03-26T03:18:46.257894+0000","last_became_active":"2024-03-26T03:14:40.180729+0000","last_became_peered":"2024-03-26T03:14:40.180729+0000","last_unstale":"2024-03-26T03:18:46.257894+0000","last_undegraded":"2024-03-26T03:18:46.257894+0000","last_fullsized":"2024-03-26T03:18:46.257894+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.804125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.804125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":877,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262088+0000","last_change":"2024-03-26T03:17:52.570959+0000","last_active":"2024-03-26T03:18:46.262088+0000","last_peered":"2024-03-26T03:18:46.262088+0000","last_clean":"2024-03-26T03:18:46.262088+0000","last_became_active":"2024-03-26T03:14:37.274568+0000","last_became_peered":"2024-03-26T03:14:37.274568+0000","last_unstale":"2024-03-26T03:18:46.262088+0000","last_undegraded":"2024-03-26T03:18:46.262088+0000","last_fullsized":"2024-03-26T03:18:46.262088+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.570835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.570835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:19.356483+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":513,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257145+0000","last_change":"2024-03-26T03:14:21.657126+0000","last_active":"2024-03-26T03:18:46.257145+0000","last_peered":"2024-03-26T03:18:46.257145+0000","last_clean":"2024-03-26T03:18:46.257145+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:18:46.257145+0000","last_undegraded":"2024-03-26T03:18:46.257145+0000","last_fullsized":"2024-03-26T03:18:46.257145+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:05:26.989192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:26.989192+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":814,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.263065+0000","last_change":"2024-03-26T03:17:41.822116+0000","last_active":"2024-03-26T03:18:46.263065+0000","last_peered":"2024-03-26T03:18:46.263065+0000","last_clean":"2024-03-26T03:18:46.263065+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:18:46.263065+0000","last_undegraded":"2024-03-26T03:18:46.263065+0000","last_fullsized":"2024-03-26T03:18:46.263065+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.822032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.822032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:06.248445+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"443'9132","reported_seq":9222,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.488221+0000","last_change":"2024-03-26T03:16:56.908716+0000","last_active":"2024-03-26T03:18:49.488221+0000","last_peered":"2024-03-26T03:18:49.488221+0000","last_clean":"2024-03-26T03:18:49.488221+0000","last_became_active":"2024-03-26T03:16:56.908005+0000","last_became_peered":"2024-03-26T03:16:56.908005+0000","last_unstale":"2024-03-26T03:18:49.488221+0000","last_undegraded":"2024-03-26T03:18:49.488221+0000","last_fullsized":"2024-03-26T03:18:49.488221+0000","mapping_epoch":403,"log_start":"432'5900","ondisk_log_start":"432'5900","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":2487,"ondisk_log_size":3232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494141440,"num_objects":7540,"num_object_clones":0,"num_object_copies":22620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9132,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":731,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262283+0000","last_change":"2024-03-26T03:16:52.721328+0000","last_active":"2024-03-26T03:18:46.262283+0000","last_peered":"2024-03-26T03:18:46.262283+0000","last_clean":"2024-03-26T03:18:46.262283+0000","last_became_active":"2024-03-26T03:16:50.206619+0000","last_became_peered":"2024-03-26T03:16:50.206619+0000","last_unstale":"2024-03-26T03:18:46.262283+0000","last_undegraded":"2024-03-26T03:18:46.262283+0000","last_fullsized":"2024-03-26T03:18:46.262283+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.721247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.734103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"230'23","reported_seq":915,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253856+0000","last_change":"2024-03-26T03:14:37.294831+0000","last_active":"2024-03-26T03:18:46.253856+0000","last_peered":"2024-03-26T03:18:46.253856+0000","last_clean":"2024-03-26T03:18:46.253856+0000","last_became_active":"2024-03-26T03:14:37.272426+0000","last_became_peered":"2024-03-26T03:14:37.272426+0000","last_unstale":"2024-03-26T03:18:46.253856+0000","last_undegraded":"2024-03-26T03:18:46.253856+0000","last_fullsized":"2024-03-26T03:18:46.253856+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:13:23.013434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:23.013434+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255514+0000","last_change":"2024-03-26T03:16:50.211076+0000","last_active":"2024-03-26T03:18:46.255514+0000","last_peered":"2024-03-26T03:18:46.255514+0000","last_clean":"2024-03-26T03:18:46.255514+0000","last_became_active":"2024-03-26T03:16:50.210676+0000","last_became_peered":"2024-03-26T03:16:50.210676+0000","last_unstale":"2024-03-26T03:18:46.255514+0000","last_undegraded":"2024-03-26T03:18:46.255514+0000","last_fullsized":"2024-03-26T03:18:46.255514+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:41.390700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:41.390700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.605951+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2034,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.106163+0000","last_change":"2024-03-26T03:13:31.075605+0000","last_active":"2024-03-26T03:18:49.106163+0000","last_peered":"2024-03-26T03:18:49.106163+0000","last_clean":"2024-03-26T03:18:49.106163+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:18:49.106163+0000","last_undegraded":"2024-03-26T03:18:49.106163+0000","last_fullsized":"2024-03-26T03:18:49.106163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:13:31.074419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.074419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1182,"num_read_kb":1177,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":790,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259952+0000","last_change":"2024-03-26T03:16:37.826030+0000","last_active":"2024-03-26T03:18:46.259952+0000","last_peered":"2024-03-26T03:18:46.259952+0000","last_clean":"2024-03-26T03:18:46.259952+0000","last_became_active":"2024-03-26T03:14:37.373684+0000","last_became_peered":"2024-03-26T03:14:37.373684+0000","last_unstale":"2024-03-26T03:18:46.259952+0000","last_undegraded":"2024-03-26T03:18:46.259952+0000","last_fullsized":"2024-03-26T03:18:46.259952+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.825906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.825906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":882,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249267+0000","last_change":"2024-03-26T03:17:59.594483+0000","last_active":"2024-03-26T03:18:46.249267+0000","last_peered":"2024-03-26T03:18:46.249267+0000","last_clean":"2024-03-26T03:18:46.249267+0000","last_became_active":"2024-03-26T03:17:59.594123+0000","last_became_peered":"2024-03-26T03:17:59.594123+0000","last_unstale":"2024-03-26T03:18:46.249267+0000","last_undegraded":"2024-03-26T03:18:46.249267+0000","last_fullsized":"2024-03-26T03:18:46.249267+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:50.701574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.701574+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.b","version":"443'8968","reported_seq":22667,"reported_epoch":443,"state":"active+clean+scrubbing","last_fresh":"2024-03-26T03:18:49.494495+0000","last_change":"2024-03-26T03:18:46.587372+0000","last_active":"2024-03-26T03:18:49.494495+0000","last_peered":"2024-03-26T03:18:49.494495+0000","last_clean":"2024-03-26T03:18:49.494495+0000","last_became_active":"2024-03-26T03:18:44.514105+0000","last_became_peered":"2024-03-26T03:18:44.514105+0000","last_unstale":"2024-03-26T03:18:49.494495+0000","last_undegraded":"2024-03-26T03:18:49.494495+0000","last_fullsized":"2024-03-26T03:18:49.494495+0000","mapping_epoch":441,"log_start":"432'5800","ondisk_log_start":"432'5800","created":403,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":7074,"log_size":3168,"log_dups_size":2387,"ondisk_log_size":3168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485490688,"num_objects":7408,"num_object_clones":0,"num_object_copies":22224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8968,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":443023360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":284,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256690+0000","last_change":"2024-03-26T03:17:56.570342+0000","last_active":"2024-03-26T03:18:46.256690+0000","last_peered":"2024-03-26T03:18:46.256690+0000","last_clean":"2024-03-26T03:18:46.256690+0000","last_became_active":"2024-03-26T03:17:56.570006+0000","last_became_peered":"2024-03-26T03:17:56.570006+0000","last_unstale":"2024-03-26T03:18:46.256690+0000","last_undegraded":"2024-03-26T03:18:46.256690+0000","last_fullsized":"2024-03-26T03:18:46.256690+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:17.442291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.442291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"332'59","reported_seq":1159,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.254957+0000","last_change":"2024-03-26T03:15:12.789448+0000","last_active":"2024-03-26T03:18:46.254957+0000","last_peered":"2024-03-26T03:18:46.254957+0000","last_clean":"2024-03-26T03:18:46.254957+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:18:46.254957+0000","last_undegraded":"2024-03-26T03:18:46.254957+0000","last_fullsized":"2024-03-26T03:18:46.254957+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'59","last_scrub_stamp":"2024-03-26T03:15:12.789148+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:15:12.789148+0000","objects_scrubbed":11,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":635,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253676+0000","last_change":"2024-03-26T03:16:50.962056+0000","last_active":"2024-03-26T03:18:46.253676+0000","last_peered":"2024-03-26T03:18:46.253676+0000","last_clean":"2024-03-26T03:18:46.253676+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:18:46.253676+0000","last_undegraded":"2024-03-26T03:18:46.253676+0000","last_fullsized":"2024-03-26T03:18:46.253676+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:50.961975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.961975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":525,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258944+0000","last_change":"2024-03-26T03:14:37.276021+0000","last_active":"2024-03-26T03:18:46.258944+0000","last_peered":"2024-03-26T03:18:46.258944+0000","last_clean":"2024-03-26T03:18:46.258944+0000","last_became_active":"2024-03-26T03:14:37.274135+0000","last_became_peered":"2024-03-26T03:14:37.274135+0000","last_unstale":"2024-03-26T03:18:46.258944+0000","last_undegraded":"2024-03-26T03:18:46.258944+0000","last_fullsized":"2024-03-26T03:18:46.258944+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:19.157956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:06:19.157956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":559,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248776+0000","last_change":"2024-03-26T03:16:30.089444+0000","last_active":"2024-03-26T03:18:46.248776+0000","last_peered":"2024-03-26T03:18:46.248776+0000","last_clean":"2024-03-26T03:18:46.248776+0000","last_became_active":"2024-03-26T03:13:56.449984+0000","last_became_peered":"2024-03-26T03:13:56.449984+0000","last_unstale":"2024-03-26T03:18:46.248776+0000","last_undegraded":"2024-03-26T03:18:46.248776+0000","last_fullsized":"2024-03-26T03:18:46.248776+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.089341+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.089341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.d","version":"443'9108","reported_seq":9199,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.703007+0000","last_change":"2024-03-26T03:16:56.901084+0000","last_active":"2024-03-26T03:18:49.703007+0000","last_peered":"2024-03-26T03:18:49.703007+0000","last_clean":"2024-03-26T03:18:49.703007+0000","last_became_active":"2024-03-26T03:16:56.900700+0000","last_became_peered":"2024-03-26T03:16:56.900700+0000","last_unstale":"2024-03-26T03:18:49.703007+0000","last_undegraded":"2024-03-26T03:18:49.703007+0000","last_fullsized":"2024-03-26T03:18:49.703007+0000","mapping_epoch":403,"log_start":"432'5691","ondisk_log_start":"432'5691","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499187736,"num_objects":7618,"num_object_clones":0,"num_object_copies":22854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7618,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9108,"num_write_kb":535169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"333'12","reported_seq":381,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258930+0000","last_change":"2024-03-26T03:17:42.576855+0000","last_active":"2024-03-26T03:18:46.258930+0000","last_peered":"2024-03-26T03:18:46.258930+0000","last_clean":"2024-03-26T03:18:46.258930+0000","last_became_active":"2024-03-26T03:14:22.574243+0000","last_became_peered":"2024-03-26T03:14:22.574243+0000","last_unstale":"2024-03-26T03:18:46.258930+0000","last_undegraded":"2024-03-26T03:18:46.258930+0000","last_fullsized":"2024-03-26T03:18:46.258930+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'12","last_scrub_stamp":"2024-03-26T03:17:42.576771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.576771+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:08.322731+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"231'32","reported_seq":921,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253467+0000","last_change":"2024-03-26T03:17:56.581987+0000","last_active":"2024-03-26T03:18:46.253467+0000","last_peered":"2024-03-26T03:18:46.253467+0000","last_clean":"2024-03-26T03:18:46.253467+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:18:46.253467+0000","last_undegraded":"2024-03-26T03:18:46.253467+0000","last_fullsized":"2024-03-26T03:18:46.253467+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:16:40.828476+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.828476+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":846,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255016+0000","last_change":"2024-03-26T03:14:37.252086+0000","last_active":"2024-03-26T03:18:46.255016+0000","last_peered":"2024-03-26T03:18:46.255016+0000","last_clean":"2024-03-26T03:18:46.255016+0000","last_became_active":"2024-03-26T03:14:37.251746+0000","last_became_peered":"2024-03-26T03:14:37.251746+0000","last_unstale":"2024-03-26T03:18:46.255016+0000","last_undegraded":"2024-03-26T03:18:46.255016+0000","last_fullsized":"2024-03-26T03:18:46.255016+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:19.981413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.981413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":244,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:49.512584+0000","last_change":"2024-03-26T03:18:49.512742+0000","last_active":"2024-03-26T03:18:49.512584+0000","last_peered":"2024-03-26T03:18:49.512584+0000","last_clean":"2024-03-26T03:18:49.512584+0000","last_became_active":"2024-03-26T03:14:22.566823+0000","last_became_peered":"2024-03-26T03:14:22.566823+0000","last_unstale":"2024-03-26T03:18:49.512584+0000","last_undegraded":"2024-03-26T03:18:49.512584+0000","last_fullsized":"2024-03-26T03:18:49.512584+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:49.512502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.512502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:52.569595+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":911,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259187+0000","last_change":"2024-03-26T03:16:30.838677+0000","last_active":"2024-03-26T03:18:46.259187+0000","last_peered":"2024-03-26T03:18:46.259187+0000","last_clean":"2024-03-26T03:18:46.259187+0000","last_became_active":"2024-03-26T03:14:40.178828+0000","last_became_peered":"2024-03-26T03:14:40.178828+0000","last_unstale":"2024-03-26T03:18:46.259187+0000","last_undegraded":"2024-03-26T03:18:46.259187+0000","last_fullsized":"2024-03-26T03:18:46.259187+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.838578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.838578+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"443'9285","reported_seq":9375,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:50.565669+0000","last_change":"2024-03-26T03:16:56.899075+0000","last_active":"2024-03-26T03:18:50.565669+0000","last_peered":"2024-03-26T03:18:50.565669+0000","last_clean":"2024-03-26T03:18:50.565669+0000","last_became_active":"2024-03-26T03:16:56.898139+0000","last_became_peered":"2024-03-26T03:16:56.898139+0000","last_unstale":"2024-03-26T03:18:50.565669+0000","last_undegraded":"2024-03-26T03:18:50.565669+0000","last_fullsized":"2024-03-26T03:18:50.565669+0000","mapping_epoch":403,"log_start":"432'6000","ondisk_log_start":"432'6000","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":496959488,"num_objects":7583,"num_object_clones":0,"num_object_copies":22749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9285,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":635,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255778+0000","last_change":"2024-03-26T03:17:59.595756+0000","last_active":"2024-03-26T03:18:46.255778+0000","last_peered":"2024-03-26T03:18:46.255778+0000","last_clean":"2024-03-26T03:18:46.255778+0000","last_became_active":"2024-03-26T03:17:59.594586+0000","last_became_peered":"2024-03-26T03:17:59.594586+0000","last_unstale":"2024-03-26T03:18:46.255778+0000","last_undegraded":"2024-03-26T03:18:46.255778+0000","last_fullsized":"2024-03-26T03:18:46.255778+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.474559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.474559+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581276+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":682,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257274+0000","last_change":"2024-03-26T03:17:53.556662+0000","last_active":"2024-03-26T03:18:46.257274+0000","last_peered":"2024-03-26T03:18:46.257274+0000","last_clean":"2024-03-26T03:18:46.257274+0000","last_became_active":"2024-03-26T03:16:50.208194+0000","last_became_peered":"2024-03-26T03:16:50.208194+0000","last_unstale":"2024-03-26T03:18:46.257274+0000","last_undegraded":"2024-03-26T03:18:46.257274+0000","last_fullsized":"2024-03-26T03:18:46.257274+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:53.556429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:53.556429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.094999+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"230'30","reported_seq":892,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248871+0000","last_change":"2024-03-26T03:14:37.290458+0000","last_active":"2024-03-26T03:18:46.248871+0000","last_peered":"2024-03-26T03:18:46.248871+0000","last_clean":"2024-03-26T03:18:46.248871+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:18:46.248871+0000","last_undegraded":"2024-03-26T03:18:46.248871+0000","last_fullsized":"2024-03-26T03:18:46.248871+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:14:37.290313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:14:37.290313+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":899,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250262+0000","last_change":"2024-03-26T03:17:37.107175+0000","last_active":"2024-03-26T03:18:46.250262+0000","last_peered":"2024-03-26T03:18:46.250262+0000","last_clean":"2024-03-26T03:18:46.250262+0000","last_became_active":"2024-03-26T03:14:37.249021+0000","last_became_peered":"2024-03-26T03:14:37.249021+0000","last_unstale":"2024-03-26T03:18:46.250262+0000","last_undegraded":"2024-03-26T03:18:46.250262+0000","last_fullsized":"2024-03-26T03:18:46.250262+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:17:37.106951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:37.106951+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:53.197468+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":497,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256536+0000","last_change":"2024-03-26T03:14:37.265797+0000","last_active":"2024-03-26T03:18:46.256536+0000","last_peered":"2024-03-26T03:18:46.256536+0000","last_clean":"2024-03-26T03:18:46.256536+0000","last_became_active":"2024-03-26T03:14:37.265041+0000","last_became_peered":"2024-03-26T03:14:37.265041+0000","last_unstale":"2024-03-26T03:18:46.256536+0000","last_undegraded":"2024-03-26T03:18:46.256536+0000","last_fullsized":"2024-03-26T03:18:46.256536+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:20.934219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:20.934219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":958,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253345+0000","last_change":"2024-03-26T03:17:59.607666+0000","last_active":"2024-03-26T03:18:46.253345+0000","last_peered":"2024-03-26T03:18:46.253345+0000","last_clean":"2024-03-26T03:18:46.253345+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:18:46.253345+0000","last_undegraded":"2024-03-26T03:18:46.253345+0000","last_fullsized":"2024-03-26T03:18:46.253345+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:44.879010+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.879010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"230'27","reported_seq":321,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258701+0000","last_change":"2024-03-26T03:14:37.323020+0000","last_active":"2024-03-26T03:18:46.258701+0000","last_peered":"2024-03-26T03:18:46.258701+0000","last_clean":"2024-03-26T03:18:46.258701+0000","last_became_active":"2024-03-26T03:14:37.269153+0000","last_became_peered":"2024-03-26T03:14:37.269153+0000","last_unstale":"2024-03-26T03:18:46.258701+0000","last_undegraded":"2024-03-26T03:18:46.258701+0000","last_fullsized":"2024-03-26T03:18:46.258701+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'21","last_scrub_stamp":"2024-03-26T03:05:35.652432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:35.652432+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581164+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":114,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257783+0000","last_change":"2024-03-26T03:17:56.578376+0000","last_active":"2024-03-26T03:18:46.257783+0000","last_peered":"2024-03-26T03:18:46.257783+0000","last_clean":"2024-03-26T03:18:46.257783+0000","last_became_active":"2024-03-26T03:17:56.571085+0000","last_became_peered":"2024-03-26T03:17:56.571085+0000","last_unstale":"2024-03-26T03:18:46.257783+0000","last_undegraded":"2024-03-26T03:18:46.257783+0000","last_fullsized":"2024-03-26T03:18:46.257783+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:16:49.187353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.187353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":516,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257852+0000","last_change":"2024-03-26T03:14:37.274074+0000","last_active":"2024-03-26T03:18:46.257852+0000","last_peered":"2024-03-26T03:18:46.257852+0000","last_clean":"2024-03-26T03:18:46.257852+0000","last_became_active":"2024-03-26T03:14:37.273658+0000","last_became_peered":"2024-03-26T03:14:37.273658+0000","last_unstale":"2024-03-26T03:18:46.257852+0000","last_undegraded":"2024-03-26T03:18:46.257852+0000","last_fullsized":"2024-03-26T03:18:46.257852+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:34.054798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:05:34.054798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":726,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249411+0000","last_change":"2024-03-26T03:17:55.099141+0000","last_active":"2024-03-26T03:18:46.249411+0000","last_peered":"2024-03-26T03:18:46.249411+0000","last_clean":"2024-03-26T03:18:46.249411+0000","last_became_active":"2024-03-26T03:16:50.205905+0000","last_became_peered":"2024-03-26T03:16:50.205905+0000","last_unstale":"2024-03-26T03:18:46.249411+0000","last_undegraded":"2024-03-26T03:18:46.249411+0000","last_fullsized":"2024-03-26T03:18:46.249411+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:55.099060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.099060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:17.587668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":353,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249135+0000","last_change":"2024-03-26T03:17:45.079338+0000","last_active":"2024-03-26T03:18:46.249135+0000","last_peered":"2024-03-26T03:18:46.249135+0000","last_clean":"2024-03-26T03:18:46.249135+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:18:46.249135+0000","last_undegraded":"2024-03-26T03:18:46.249135+0000","last_fullsized":"2024-03-26T03:18:46.249135+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:45.079209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.079209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:07.973854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"230'18","reported_seq":589,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257426+0000","last_change":"2024-03-26T03:16:40.927377+0000","last_active":"2024-03-26T03:18:46.257426+0000","last_peered":"2024-03-26T03:18:46.257426+0000","last_clean":"2024-03-26T03:18:46.257426+0000","last_became_active":"2024-03-26T03:14:37.269408+0000","last_became_peered":"2024-03-26T03:14:37.269408+0000","last_unstale":"2024-03-26T03:18:46.257426+0000","last_undegraded":"2024-03-26T03:18:46.257426+0000","last_fullsized":"2024-03-26T03:18:46.257426+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'18","last_scrub_stamp":"2024-03-26T03:16:40.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.927260+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":853,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258337+0000","last_change":"2024-03-26T03:17:56.570470+0000","last_active":"2024-03-26T03:18:46.258337+0000","last_peered":"2024-03-26T03:18:46.258337+0000","last_clean":"2024-03-26T03:18:46.258337+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:18:46.258337+0000","last_undegraded":"2024-03-26T03:18:46.258337+0000","last_fullsized":"2024-03-26T03:18:46.258337+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:59.707108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.707108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":817,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261990+0000","last_change":"2024-03-26T03:17:44.697333+0000","last_active":"2024-03-26T03:18:46.261990+0000","last_peered":"2024-03-26T03:18:46.261990+0000","last_clean":"2024-03-26T03:18:46.261990+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:18:46.261990+0000","last_undegraded":"2024-03-26T03:18:46.261990+0000","last_fullsized":"2024-03-26T03:18:46.261990+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.697255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.697255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.820334+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":223,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259993+0000","last_change":"2024-03-26T03:16:45.758249+0000","last_active":"2024-03-26T03:18:46.259993+0000","last_peered":"2024-03-26T03:18:46.259993+0000","last_clean":"2024-03-26T03:18:46.259993+0000","last_became_active":"2024-03-26T03:14:37.333430+0000","last_became_peered":"2024-03-26T03:14:37.333430+0000","last_unstale":"2024-03-26T03:18:46.259993+0000","last_undegraded":"2024-03-26T03:18:46.259993+0000","last_fullsized":"2024-03-26T03:18:46.259993+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.758140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.758140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":594,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261187+0000","last_change":"2024-03-26T03:17:40.727303+0000","last_active":"2024-03-26T03:18:46.261187+0000","last_peered":"2024-03-26T03:18:46.261187+0000","last_clean":"2024-03-26T03:18:46.261187+0000","last_became_active":"2024-03-26T03:14:24.567937+0000","last_became_peered":"2024-03-26T03:14:24.567937+0000","last_unstale":"2024-03-26T03:18:46.261187+0000","last_undegraded":"2024-03-26T03:18:46.261187+0000","last_fullsized":"2024-03-26T03:18:46.261187+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:40.727084+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.727084+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:03.188531+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"231'39","reported_seq":606,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255956+0000","last_change":"2024-03-26T03:16:49.904794+0000","last_active":"2024-03-26T03:18:46.255956+0000","last_peered":"2024-03-26T03:18:46.255956+0000","last_clean":"2024-03-26T03:18:46.255956+0000","last_became_active":"2024-03-26T03:14:37.265466+0000","last_became_peered":"2024-03-26T03:14:37.265466+0000","last_unstale":"2024-03-26T03:18:46.255956+0000","last_undegraded":"2024-03-26T03:18:46.255956+0000","last_fullsized":"2024-03-26T03:18:46.255956+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.904637+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":851,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261730+0000","last_change":"2024-03-26T03:17:56.569921+0000","last_active":"2024-03-26T03:18:46.261730+0000","last_peered":"2024-03-26T03:18:46.261730+0000","last_clean":"2024-03-26T03:18:46.261730+0000","last_became_active":"2024-03-26T03:17:56.569537+0000","last_became_peered":"2024-03-26T03:17:56.569537+0000","last_unstale":"2024-03-26T03:18:46.261730+0000","last_undegraded":"2024-03-26T03:18:46.261730+0000","last_fullsized":"2024-03-26T03:18:46.261730+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:48.651379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.651379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:01.106930+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"333'13","reported_seq":1112,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262413+0000","last_change":"2024-03-26T03:18:09.760752+0000","last_active":"2024-03-26T03:18:46.262413+0000","last_peered":"2024-03-26T03:18:46.262413+0000","last_clean":"2024-03-26T03:18:46.262413+0000","last_became_active":"2024-03-26T03:14:22.577229+0000","last_became_peered":"2024-03-26T03:14:22.577229+0000","last_unstale":"2024-03-26T03:18:46.262413+0000","last_undegraded":"2024-03-26T03:18:46.262413+0000","last_fullsized":"2024-03-26T03:18:46.262413+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'13","last_scrub_stamp":"2024-03-26T03:18:09.760609+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:18:09.760609+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:30.051655+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":553,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258801+0000","last_change":"2024-03-26T03:14:37.275365+0000","last_active":"2024-03-26T03:18:46.258801+0000","last_peered":"2024-03-26T03:18:46.258801+0000","last_clean":"2024-03-26T03:18:46.258801+0000","last_became_active":"2024-03-26T03:14:37.273988+0000","last_became_peered":"2024-03-26T03:14:37.273988+0000","last_unstale":"2024-03-26T03:18:46.258801+0000","last_undegraded":"2024-03-26T03:18:46.258801+0000","last_fullsized":"2024-03-26T03:18:46.258801+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:16.678750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:16.678750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581073+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":518,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257806+0000","last_change":"2024-03-26T03:16:28.906151+0000","last_active":"2024-03-26T03:18:46.257806+0000","last_peered":"2024-03-26T03:18:46.257806+0000","last_clean":"2024-03-26T03:18:46.257806+0000","last_became_active":"2024-03-26T03:14:40.173243+0000","last_became_peered":"2024-03-26T03:14:40.173243+0000","last_unstale":"2024-03-26T03:18:46.257806+0000","last_undegraded":"2024-03-26T03:18:46.257806+0000","last_fullsized":"2024-03-26T03:18:46.257806+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:28.906044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.906044+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"231'21","reported_seq":654,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256205+0000","last_change":"2024-03-26T03:16:59.931429+0000","last_active":"2024-03-26T03:18:46.256205+0000","last_peered":"2024-03-26T03:18:46.256205+0000","last_clean":"2024-03-26T03:18:46.256205+0000","last_became_active":"2024-03-26T03:14:37.333426+0000","last_became_peered":"2024-03-26T03:14:37.333426+0000","last_unstale":"2024-03-26T03:18:46.256205+0000","last_undegraded":"2024-03-26T03:18:46.256205+0000","last_fullsized":"2024-03-26T03:18:46.256205+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'21","last_scrub_stamp":"2024-03-26T03:16:59.931283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.931283+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":587,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249632+0000","last_change":"2024-03-26T03:16:29.133272+0000","last_active":"2024-03-26T03:18:46.249632+0000","last_peered":"2024-03-26T03:18:46.249632+0000","last_clean":"2024-03-26T03:18:46.249632+0000","last_became_active":"2024-03-26T03:14:37.262766+0000","last_became_peered":"2024-03-26T03:14:37.262766+0000","last_unstale":"2024-03-26T03:18:46.249632+0000","last_undegraded":"2024-03-26T03:18:46.249632+0000","last_fullsized":"2024-03-26T03:18:46.249632+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:29.133174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.133174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"342'11","reported_seq":618,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256280+0000","last_change":"2024-03-26T03:17:50.265634+0000","last_active":"2024-03-26T03:18:46.256280+0000","last_peered":"2024-03-26T03:18:46.256280+0000","last_clean":"2024-03-26T03:18:46.256280+0000","last_became_active":"2024-03-26T03:14:37.264759+0000","last_became_peered":"2024-03-26T03:14:37.264759+0000","last_unstale":"2024-03-26T03:18:46.256280+0000","last_undegraded":"2024-03-26T03:18:46.256280+0000","last_fullsized":"2024-03-26T03:18:46.256280+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'11","last_scrub_stamp":"2024-03-26T03:17:50.265404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:50.265404+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:50.629014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":820,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250402+0000","last_change":"2024-03-26T03:17:48.227168+0000","last_active":"2024-03-26T03:18:46.250402+0000","last_peered":"2024-03-26T03:18:46.250402+0000","last_clean":"2024-03-26T03:18:46.250402+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:18:46.250402+0000","last_undegraded":"2024-03-26T03:18:46.250402+0000","last_fullsized":"2024-03-26T03:18:46.250402+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:17:48.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.226947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.196614+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":931,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262616+0000","last_change":"2024-03-26T03:17:55.577343+0000","last_active":"2024-03-26T03:18:46.262616+0000","last_peered":"2024-03-26T03:18:46.262616+0000","last_clean":"2024-03-26T03:18:46.262616+0000","last_became_active":"2024-03-26T03:14:40.170537+0000","last_became_peered":"2024-03-26T03:14:40.170537+0000","last_unstale":"2024-03-26T03:18:46.262616+0000","last_undegraded":"2024-03-26T03:18:46.262616+0000","last_fullsized":"2024-03-26T03:18:46.262616+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:55.577263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.577263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:21.650938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":858,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253361+0000","last_change":"2024-03-26T03:13:33.092136+0000","last_active":"2024-03-26T03:18:46.253361+0000","last_peered":"2024-03-26T03:18:46.253361+0000","last_clean":"2024-03-26T03:18:46.253361+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:18:46.253361+0000","last_undegraded":"2024-03-26T03:18:46.253361+0000","last_fullsized":"2024-03-26T03:18:46.253361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:13:33.091975+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:13:33.091975+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":818,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258625+0000","last_change":"2024-03-26T03:16:33.748393+0000","last_active":"2024-03-26T03:18:46.258625+0000","last_peered":"2024-03-26T03:18:46.258625+0000","last_clean":"2024-03-26T03:18:46.258625+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:18:46.258625+0000","last_undegraded":"2024-03-26T03:18:46.258625+0000","last_fullsized":"2024-03-26T03:18:46.258625+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:33.748295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:33.748295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":507,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256113+0000","last_change":"2024-03-26T03:16:44.909663+0000","last_active":"2024-03-26T03:18:46.256113+0000","last_peered":"2024-03-26T03:18:46.256113+0000","last_clean":"2024-03-26T03:18:46.256113+0000","last_became_active":"2024-03-26T03:14:22.573232+0000","last_became_peered":"2024-03-26T03:14:22.573232+0000","last_unstale":"2024-03-26T03:18:46.256113+0000","last_undegraded":"2024-03-26T03:18:46.256113+0000","last_fullsized":"2024-03-26T03:18:46.256113+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.909523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.909523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":281,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257364+0000","last_change":"2024-03-26T03:17:35.405461+0000","last_active":"2024-03-26T03:18:46.257364+0000","last_peered":"2024-03-26T03:18:46.257364+0000","last_clean":"2024-03-26T03:18:46.257364+0000","last_became_active":"2024-03-26T03:13:35.278675+0000","last_became_peered":"2024-03-26T03:13:35.278675+0000","last_unstale":"2024-03-26T03:18:46.257364+0000","last_undegraded":"2024-03-26T03:18:46.257364+0000","last_fullsized":"2024-03-26T03:18:46.257364+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.405139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.405139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:00.124612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":599,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261802+0000","last_change":"2024-03-26T03:14:22.578069+0000","last_active":"2024-03-26T03:18:46.261802+0000","last_peered":"2024-03-26T03:18:46.261802+0000","last_clean":"2024-03-26T03:18:46.261802+0000","last_became_active":"2024-03-26T03:14:22.577614+0000","last_became_peered":"2024-03-26T03:14:22.577614+0000","last_unstale":"2024-03-26T03:18:46.261802+0000","last_undegraded":"2024-03-26T03:18:46.261802+0000","last_fullsized":"2024-03-26T03:18:46.261802+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.940687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.940687+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"231'19","reported_seq":858,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249485+0000","last_change":"2024-03-26T03:16:53.170995+0000","last_active":"2024-03-26T03:18:46.249485+0000","last_peered":"2024-03-26T03:18:46.249485+0000","last_clean":"2024-03-26T03:18:46.249485+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:18:46.249485+0000","last_undegraded":"2024-03-26T03:18:46.249485+0000","last_fullsized":"2024-03-26T03:18:46.249485+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:16:53.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:53.170849+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":855,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256841+0000","last_change":"2024-03-26T03:17:34.417755+0000","last_active":"2024-03-26T03:18:46.256841+0000","last_peered":"2024-03-26T03:18:46.256841+0000","last_clean":"2024-03-26T03:18:46.256841+0000","last_became_active":"2024-03-26T03:13:35.274830+0000","last_became_peered":"2024-03-26T03:13:35.274830+0000","last_unstale":"2024-03-26T03:18:46.256841+0000","last_undegraded":"2024-03-26T03:18:46.256841+0000","last_fullsized":"2024-03-26T03:18:46.256841+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.417610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.417610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":802,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253569+0000","last_change":"2024-03-26T03:15:14.862264+0000","last_active":"2024-03-26T03:18:46.253569+0000","last_peered":"2024-03-26T03:18:46.253569+0000","last_clean":"2024-03-26T03:18:46.253569+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:18:46.253569+0000","last_undegraded":"2024-03-26T03:18:46.253569+0000","last_fullsized":"2024-03-26T03:18:46.253569+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:14.862148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:14.862148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":511,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256008+0000","last_change":"2024-03-26T03:14:22.577571+0000","last_active":"2024-03-26T03:18:46.256008+0000","last_peered":"2024-03-26T03:18:46.256008+0000","last_clean":"2024-03-26T03:18:46.256008+0000","last_became_active":"2024-03-26T03:14:22.577265+0000","last_became_peered":"2024-03-26T03:14:22.577265+0000","last_unstale":"2024-03-26T03:18:46.256008+0000","last_undegraded":"2024-03-26T03:18:46.256008+0000","last_fullsized":"2024-03-26T03:18:46.256008+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:19.903253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:19.903253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":522,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.252699+0000","last_change":"2024-03-26T03:14:40.950126+0000","last_active":"2024-03-26T03:18:46.252699+0000","last_peered":"2024-03-26T03:18:46.252699+0000","last_clean":"2024-03-26T03:18:46.252699+0000","last_became_active":"2024-03-26T03:14:40.166745+0000","last_became_peered":"2024-03-26T03:14:40.166745+0000","last_unstale":"2024-03-26T03:18:46.252699+0000","last_undegraded":"2024-03-26T03:18:46.252699+0000","last_fullsized":"2024-03-26T03:18:46.252699+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:40.950003+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:14:40.950003+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"230'8","reported_seq":528,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.260223+0000","last_change":"2024-03-26T03:16:34.843257+0000","last_active":"2024-03-26T03:18:46.260223+0000","last_peered":"2024-03-26T03:18:46.260223+0000","last_clean":"2024-03-26T03:18:46.260223+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:18:46.260223+0000","last_undegraded":"2024-03-26T03:18:46.260223+0000","last_fullsized":"2024-03-26T03:18:46.260223+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'8","last_scrub_stamp":"2024-03-26T03:16:34.843159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:34.843159+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.605921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":856,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262202+0000","last_change":"2024-03-26T03:17:35.886138+0000","last_active":"2024-03-26T03:18:46.262202+0000","last_peered":"2024-03-26T03:18:46.262202+0000","last_clean":"2024-03-26T03:18:46.262202+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:18:46.262202+0000","last_undegraded":"2024-03-26T03:18:46.262202+0000","last_fullsized":"2024-03-26T03:18:46.262202+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.886021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.886021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:04.652304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":859,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258229+0000","last_change":"2024-03-26T03:17:57.629803+0000","last_active":"2024-03-26T03:18:46.258229+0000","last_peered":"2024-03-26T03:18:46.258229+0000","last_clean":"2024-03-26T03:18:46.258229+0000","last_became_active":"2024-03-26T03:17:56.569576+0000","last_became_peered":"2024-03-26T03:17:56.569576+0000","last_unstale":"2024-03-26T03:18:46.258229+0000","last_undegraded":"2024-03-26T03:18:46.258229+0000","last_fullsized":"2024-03-26T03:18:46.258229+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.629637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.629637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:22.066988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":223,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249570+0000","last_change":"2024-03-26T03:16:45.148252+0000","last_active":"2024-03-26T03:18:46.249570+0000","last_peered":"2024-03-26T03:18:46.249570+0000","last_clean":"2024-03-26T03:18:46.249570+0000","last_became_active":"2024-03-26T03:14:37.260920+0000","last_became_peered":"2024-03-26T03:14:37.260920+0000","last_unstale":"2024-03-26T03:18:46.249570+0000","last_undegraded":"2024-03-26T03:18:46.249570+0000","last_fullsized":"2024-03-26T03:18:46.249570+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.148129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.148129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":595,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249313+0000","last_change":"2024-03-26T03:17:59.621631+0000","last_active":"2024-03-26T03:18:46.249313+0000","last_peered":"2024-03-26T03:18:46.249313+0000","last_clean":"2024-03-26T03:18:46.249313+0000","last_became_active":"2024-03-26T03:17:59.621344+0000","last_became_peered":"2024-03-26T03:17:59.621344+0000","last_unstale":"2024-03-26T03:18:46.249313+0000","last_undegraded":"2024-03-26T03:18:46.249313+0000","last_fullsized":"2024-03-26T03:18:46.249313+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:45.232258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.232258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"231'31","reported_seq":591,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256670+0000","last_change":"2024-03-26T03:16:56.991143+0000","last_active":"2024-03-26T03:18:46.256670+0000","last_peered":"2024-03-26T03:18:46.256670+0000","last_clean":"2024-03-26T03:18:46.256670+0000","last_became_active":"2024-03-26T03:14:37.273380+0000","last_became_peered":"2024-03-26T03:14:37.273380+0000","last_unstale":"2024-03-26T03:18:46.256670+0000","last_undegraded":"2024-03-26T03:18:46.256670+0000","last_fullsized":"2024-03-26T03:18:46.256670+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'31","last_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.991031+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":841,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259290+0000","last_change":"2024-03-26T03:17:58.647438+0000","last_active":"2024-03-26T03:18:46.259290+0000","last_peered":"2024-03-26T03:18:46.259290+0000","last_clean":"2024-03-26T03:18:46.259290+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:18:46.259290+0000","last_undegraded":"2024-03-26T03:18:46.259290+0000","last_fullsized":"2024-03-26T03:18:46.259290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:58.647326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:58.647326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:12.388072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":226,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248678+0000","last_change":"2024-03-26T03:17:32.002736+0000","last_active":"2024-03-26T03:18:46.248678+0000","last_peered":"2024-03-26T03:18:46.248678+0000","last_clean":"2024-03-26T03:18:46.248678+0000","last_became_active":"2024-03-26T03:14:37.249377+0000","last_became_peered":"2024-03-26T03:14:37.249377+0000","last_unstale":"2024-03-26T03:18:46.248678+0000","last_undegraded":"2024-03-26T03:18:46.248678+0000","last_fullsized":"2024-03-26T03:18:46.248678+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.002635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.002635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:00.926149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":497,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255946+0000","last_change":"2024-03-26T03:14:22.603550+0000","last_active":"2024-03-26T03:18:46.255946+0000","last_peered":"2024-03-26T03:18:46.255946+0000","last_clean":"2024-03-26T03:18:46.255946+0000","last_became_active":"2024-03-26T03:14:22.603198+0000","last_became_peered":"2024-03-26T03:14:22.603198+0000","last_unstale":"2024-03-26T03:18:46.255946+0000","last_undegraded":"2024-03-26T03:18:46.255946+0000","last_fullsized":"2024-03-26T03:18:46.255946+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:38.188061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:38.188061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":834,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261817+0000","last_change":"2024-03-26T03:16:48.688560+0000","last_active":"2024-03-26T03:18:46.261817+0000","last_peered":"2024-03-26T03:18:46.261817+0000","last_clean":"2024-03-26T03:18:46.261817+0000","last_became_active":"2024-03-26T03:13:38.309830+0000","last_became_peered":"2024-03-26T03:13:38.309830+0000","last_unstale":"2024-03-26T03:18:46.261817+0000","last_undegraded":"2024-03-26T03:18:46.261817+0000","last_fullsized":"2024-03-26T03:18:46.261817+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:48.688473+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.688473+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.734046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"231'26","reported_seq":703,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258601+0000","last_change":"2024-03-26T03:14:37.333804+0000","last_active":"2024-03-26T03:18:46.258601+0000","last_peered":"2024-03-26T03:18:46.258601+0000","last_clean":"2024-03-26T03:18:46.258601+0000","last_became_active":"2024-03-26T03:14:37.274931+0000","last_became_peered":"2024-03-26T03:14:37.274931+0000","last_unstale":"2024-03-26T03:18:46.258601+0000","last_undegraded":"2024-03-26T03:18:46.258601+0000","last_fullsized":"2024-03-26T03:18:46.258601+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'16","last_scrub_stamp":"2024-03-26T03:05:43.709077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:43.709077+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581196+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":32,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":862,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261191+0000","last_change":"2024-03-26T03:17:32.885230+0000","last_active":"2024-03-26T03:18:46.261191+0000","last_peered":"2024-03-26T03:18:46.261191+0000","last_clean":"2024-03-26T03:18:46.261191+0000","last_became_active":"2024-03-26T03:14:37.266519+0000","last_became_peered":"2024-03-26T03:14:37.266519+0000","last_unstale":"2024-03-26T03:18:46.261191+0000","last_undegraded":"2024-03-26T03:18:46.261191+0000","last_fullsized":"2024-03-26T03:18:46.261191+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.885069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.885069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:46.734156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"428'11","reported_seq":1115,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259444+0000","last_change":"2024-03-26T03:17:56.576319+0000","last_active":"2024-03-26T03:18:46.259444+0000","last_peered":"2024-03-26T03:18:46.259444+0000","last_clean":"2024-03-26T03:18:46.259444+0000","last_became_active":"2024-03-26T03:17:56.569871+0000","last_became_peered":"2024-03-26T03:17:56.569871+0000","last_unstale":"2024-03-26T03:18:46.259444+0000","last_undegraded":"2024-03-26T03:18:46.259444+0000","last_fullsized":"2024-03-26T03:18:46.259444+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'10","last_scrub_stamp":"2024-03-26T03:16:56.648648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.648648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":605,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255477+0000","last_change":"2024-03-26T03:16:50.207171+0000","last_active":"2024-03-26T03:18:46.255477+0000","last_peered":"2024-03-26T03:18:46.255477+0000","last_clean":"2024-03-26T03:18:46.255477+0000","last_became_active":"2024-03-26T03:16:50.206818+0000","last_became_peered":"2024-03-26T03:16:50.206818+0000","last_unstale":"2024-03-26T03:18:46.255477+0000","last_undegraded":"2024-03-26T03:18:46.255477+0000","last_fullsized":"2024-03-26T03:18:46.255477+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:12:34.141917+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581244+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":820,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249114+0000","last_change":"2024-03-26T03:17:59.611282+0000","last_active":"2024-03-26T03:18:46.249114+0000","last_peered":"2024-03-26T03:18:46.249114+0000","last_clean":"2024-03-26T03:18:46.249114+0000","last_became_active":"2024-03-26T03:17:59.610887+0000","last_became_peered":"2024-03-26T03:17:59.610887+0000","last_unstale":"2024-03-26T03:18:46.249114+0000","last_undegraded":"2024-03-26T03:18:46.249114+0000","last_fullsized":"2024-03-26T03:18:46.249114+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.803243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.803243+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"231'25","reported_seq":873,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259514+0000","last_change":"2024-03-26T03:13:32.336994+0000","last_active":"2024-03-26T03:18:46.259514+0000","last_peered":"2024-03-26T03:18:46.259514+0000","last_clean":"2024-03-26T03:18:46.259514+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:18:46.259514+0000","last_undegraded":"2024-03-26T03:18:46.259514+0000","last_fullsized":"2024-03-26T03:18:46.259514+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:13:32.336748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:32.336748+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":478,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257038+0000","last_change":"2024-03-26T03:17:31.999279+0000","last_active":"2024-03-26T03:18:46.257038+0000","last_peered":"2024-03-26T03:18:46.257038+0000","last_clean":"2024-03-26T03:18:46.257038+0000","last_became_active":"2024-03-26T03:14:22.576816+0000","last_became_peered":"2024-03-26T03:14:22.576816+0000","last_unstale":"2024-03-26T03:18:46.257038+0000","last_undegraded":"2024-03-26T03:18:46.257038+0000","last_fullsized":"2024-03-26T03:18:46.257038+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:17:31.999158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:31.999158+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:59.446120+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":219,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.262037+0000","last_change":"2024-03-26T03:14:37.255762+0000","last_active":"2024-03-26T03:18:46.262037+0000","last_peered":"2024-03-26T03:18:46.262037+0000","last_clean":"2024-03-26T03:18:46.262037+0000","last_became_active":"2024-03-26T03:14:37.255400+0000","last_became_peered":"2024-03-26T03:14:37.255400+0000","last_unstale":"2024-03-26T03:18:46.262037+0000","last_undegraded":"2024-03-26T03:18:46.262037+0000","last_fullsized":"2024-03-26T03:18:46.262037+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:25.571608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:13:25.571608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":832,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256589+0000","last_change":"2024-03-26T03:17:56.570706+0000","last_active":"2024-03-26T03:18:46.256589+0000","last_peered":"2024-03-26T03:18:46.256589+0000","last_clean":"2024-03-26T03:18:46.256589+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:18:46.256589+0000","last_undegraded":"2024-03-26T03:18:46.256589+0000","last_fullsized":"2024-03-26T03:18:46.256589+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.953482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.953482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":723,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257126+0000","last_change":"2024-03-26T03:14:40.162652+0000","last_active":"2024-03-26T03:18:46.257126+0000","last_peered":"2024-03-26T03:18:46.257126+0000","last_clean":"2024-03-26T03:18:46.257126+0000","last_became_active":"2024-03-26T03:14:40.162258+0000","last_became_peered":"2024-03-26T03:14:40.162258+0000","last_unstale":"2024-03-26T03:18:46.257126+0000","last_undegraded":"2024-03-26T03:18:46.257126+0000","last_fullsized":"2024-03-26T03:18:46.257126+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.546156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.546156+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581292+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"230'27","reported_seq":507,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258873+0000","last_change":"2024-03-26T03:16:50.211157+0000","last_active":"2024-03-26T03:18:46.258873+0000","last_peered":"2024-03-26T03:18:46.258873+0000","last_clean":"2024-03-26T03:18:46.258873+0000","last_became_active":"2024-03-26T03:16:50.209508+0000","last_became_peered":"2024-03-26T03:16:50.209508+0000","last_unstale":"2024-03-26T03:18:46.258873+0000","last_undegraded":"2024-03-26T03:18:46.258873+0000","last_fullsized":"2024-03-26T03:18:46.258873+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'27","last_scrub_stamp":"2024-03-26T03:16:36.265181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.265181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":854,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258268+0000","last_change":"2024-03-26T03:17:47.637363+0000","last_active":"2024-03-26T03:18:46.258268+0000","last_peered":"2024-03-26T03:18:46.258268+0000","last_clean":"2024-03-26T03:18:46.258268+0000","last_became_active":"2024-03-26T03:14:37.249066+0000","last_became_peered":"2024-03-26T03:14:37.249066+0000","last_unstale":"2024-03-26T03:18:46.258268+0000","last_undegraded":"2024-03-26T03:18:46.258268+0000","last_fullsized":"2024-03-26T03:18:46.258268+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:47.637246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:47.637246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:57.108832+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259753+0000","last_change":"2024-03-26T03:17:56.573995+0000","last_active":"2024-03-26T03:18:46.259753+0000","last_peered":"2024-03-26T03:18:46.259753+0000","last_clean":"2024-03-26T03:18:46.259753+0000","last_became_active":"2024-03-26T03:17:56.571840+0000","last_became_peered":"2024-03-26T03:17:56.571840+0000","last_unstale":"2024-03-26T03:18:46.259753+0000","last_undegraded":"2024-03-26T03:18:46.259753+0000","last_fullsized":"2024-03-26T03:18:46.259753+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.507151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.507151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.755968+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":820,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250428+0000","last_change":"2024-03-26T03:13:29.068724+0000","last_active":"2024-03-26T03:18:46.250428+0000","last_peered":"2024-03-26T03:18:46.250428+0000","last_clean":"2024-03-26T03:18:46.250428+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:18:46.250428+0000","last_undegraded":"2024-03-26T03:18:46.250428+0000","last_fullsized":"2024-03-26T03:18:46.250428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.068502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.068502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":627,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:48.870772+0000","last_change":"2024-03-26T03:18:48.870805+0000","last_active":"2024-03-26T03:18:48.870772+0000","last_peered":"2024-03-26T03:18:48.870772+0000","last_clean":"2024-03-26T03:18:48.870772+0000","last_became_active":"2024-03-26T03:14:40.162583+0000","last_became_peered":"2024-03-26T03:14:40.162583+0000","last_unstale":"2024-03-26T03:18:48.870772+0000","last_undegraded":"2024-03-26T03:18:48.870772+0000","last_fullsized":"2024-03-26T03:18:48.870772+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:18:48.870709+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:18:48.870709+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:01.732773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"399'30","reported_seq":569,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258169+0000","last_change":"2024-03-26T03:16:48.957169+0000","last_active":"2024-03-26T03:18:46.258169+0000","last_peered":"2024-03-26T03:18:46.258169+0000","last_clean":"2024-03-26T03:18:46.258169+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:18:46.258169+0000","last_undegraded":"2024-03-26T03:18:46.258169+0000","last_fullsized":"2024-03-26T03:18:46.258169+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'30","last_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.957049+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.606419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":536,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.256227+0000","last_change":"2024-03-26T03:16:39.889851+0000","last_active":"2024-03-26T03:18:46.256227+0000","last_peered":"2024-03-26T03:18:46.256227+0000","last_clean":"2024-03-26T03:18:46.256227+0000","last_became_active":"2024-03-26T03:14:37.275278+0000","last_became_peered":"2024-03-26T03:14:37.275278+0000","last_unstale":"2024-03-26T03:18:46.256227+0000","last_undegraded":"2024-03-26T03:18:46.256227+0000","last_fullsized":"2024-03-26T03:18:46.256227+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:39.889770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.889770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.605159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":799,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249494+0000","last_change":"2024-03-26T03:14:36.271942+0000","last_active":"2024-03-26T03:18:46.249494+0000","last_peered":"2024-03-26T03:18:46.249494+0000","last_clean":"2024-03-26T03:18:46.249494+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:18:46.249494+0000","last_undegraded":"2024-03-26T03:18:46.249494+0000","last_fullsized":"2024-03-26T03:18:46.249494+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.271810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.271810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":492,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257703+0000","last_change":"2024-03-26T03:14:21.603222+0000","last_active":"2024-03-26T03:18:46.257703+0000","last_peered":"2024-03-26T03:18:46.257703+0000","last_clean":"2024-03-26T03:18:46.257703+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:18:46.257703+0000","last_undegraded":"2024-03-26T03:18:46.257703+0000","last_fullsized":"2024-03-26T03:18:46.257703+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:22.998081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:22.998081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"230'33","reported_seq":63,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249041+0000","last_change":"2024-03-26T03:17:57.160567+0000","last_active":"2024-03-26T03:18:46.249041+0000","last_peered":"2024-03-26T03:18:46.249041+0000","last_clean":"2024-03-26T03:18:46.249041+0000","last_became_active":"2024-03-26T03:17:56.567557+0000","last_became_peered":"2024-03-26T03:17:56.567557+0000","last_unstale":"2024-03-26T03:18:46.249041+0000","last_undegraded":"2024-03-26T03:18:46.249041+0000","last_fullsized":"2024-03-26T03:18:46.249041+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'33","last_scrub_stamp":"2024-03-26T03:17:57.160442+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.160442+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:13.145759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":222,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258781+0000","last_change":"2024-03-26T03:16:51.673167+0000","last_active":"2024-03-26T03:18:46.258781+0000","last_peered":"2024-03-26T03:18:46.258781+0000","last_clean":"2024-03-26T03:18:46.258781+0000","last_became_active":"2024-03-26T03:14:37.243678+0000","last_became_peered":"2024-03-26T03:14:37.243678+0000","last_unstale":"2024-03-26T03:18:46.258781+0000","last_undegraded":"2024-03-26T03:18:46.258781+0000","last_fullsized":"2024-03-26T03:18:46.258781+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.673066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.673066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":871,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261989+0000","last_change":"2024-03-26T03:18:01.638296+0000","last_active":"2024-03-26T03:18:46.261989+0000","last_peered":"2024-03-26T03:18:46.261989+0000","last_clean":"2024-03-26T03:18:46.261989+0000","last_became_active":"2024-03-26T03:17:56.570631+0000","last_became_peered":"2024-03-26T03:17:56.570631+0000","last_unstale":"2024-03-26T03:18:46.261989+0000","last_undegraded":"2024-03-26T03:18:46.261989+0000","last_fullsized":"2024-03-26T03:18:46.261989+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:01.638157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:01.638157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:28.331978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":291,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.252417+0000","last_change":"2024-03-26T03:17:56.571755+0000","last_active":"2024-03-26T03:18:46.252417+0000","last_peered":"2024-03-26T03:18:46.252417+0000","last_clean":"2024-03-26T03:18:46.252417+0000","last_became_active":"2024-03-26T03:17:56.571178+0000","last_became_peered":"2024-03-26T03:17:56.571178+0000","last_unstale":"2024-03-26T03:18:46.252417+0000","last_undegraded":"2024-03-26T03:18:46.252417+0000","last_fullsized":"2024-03-26T03:18:46.252417+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.660527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.660527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":566,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258328+0000","last_change":"2024-03-26T03:17:59.595137+0000","last_active":"2024-03-26T03:18:46.258328+0000","last_peered":"2024-03-26T03:18:46.258328+0000","last_clean":"2024-03-26T03:18:46.258328+0000","last_became_active":"2024-03-26T03:17:59.594759+0000","last_became_peered":"2024-03-26T03:17:59.594759+0000","last_unstale":"2024-03-26T03:18:46.258328+0000","last_undegraded":"2024-03-26T03:18:46.258328+0000","last_fullsized":"2024-03-26T03:18:46.258328+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:32.825864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.825864+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.605265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"230'13","reported_seq":49,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249198+0000","last_change":"2024-03-26T03:17:56.580380+0000","last_active":"2024-03-26T03:18:46.249198+0000","last_peered":"2024-03-26T03:18:46.249198+0000","last_clean":"2024-03-26T03:18:46.249198+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:18:46.249198+0000","last_undegraded":"2024-03-26T03:18:46.249198+0000","last_fullsized":"2024-03-26T03:18:46.249198+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'13","last_scrub_stamp":"2024-03-26T03:17:45.281375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.281375+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:51.925165+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":17,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":827,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259887+0000","last_change":"2024-03-26T03:16:38.824092+0000","last_active":"2024-03-26T03:18:46.259887+0000","last_peered":"2024-03-26T03:18:46.259887+0000","last_clean":"2024-03-26T03:18:46.259887+0000","last_became_active":"2024-03-26T03:14:37.373600+0000","last_became_peered":"2024-03-26T03:14:37.373600+0000","last_unstale":"2024-03-26T03:18:46.259887+0000","last_undegraded":"2024-03-26T03:18:46.259887+0000","last_fullsized":"2024-03-26T03:18:46.259887+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:38.823985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.823985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":223,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249896+0000","last_change":"2024-03-26T03:16:51.182163+0000","last_active":"2024-03-26T03:18:46.249896+0000","last_peered":"2024-03-26T03:18:46.249896+0000","last_clean":"2024-03-26T03:18:46.249896+0000","last_became_active":"2024-03-26T03:14:37.253707+0000","last_became_peered":"2024-03-26T03:14:37.253707+0000","last_unstale":"2024-03-26T03:18:46.249896+0000","last_undegraded":"2024-03-26T03:18:46.249896+0000","last_fullsized":"2024-03-26T03:18:46.249896+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.182029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":813,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.261413+0000","last_change":"2024-03-26T03:17:47.711589+0000","last_active":"2024-03-26T03:18:46.261413+0000","last_peered":"2024-03-26T03:18:46.261413+0000","last_clean":"2024-03-26T03:18:46.261413+0000","last_became_active":"2024-03-26T03:14:21.596806+0000","last_became_peered":"2024-03-26T03:14:21.596806+0000","last_unstale":"2024-03-26T03:18:46.261413+0000","last_undegraded":"2024-03-26T03:18:46.261413+0000","last_fullsized":"2024-03-26T03:18:46.261413+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:47.711445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:47.711445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:03.092974+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":626,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.254872+0000","last_change":"2024-03-26T03:17:59.610214+0000","last_active":"2024-03-26T03:18:46.254872+0000","last_peered":"2024-03-26T03:18:46.254872+0000","last_clean":"2024-03-26T03:18:46.254872+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:18:46.254872+0000","last_undegraded":"2024-03-26T03:18:46.254872+0000","last_fullsized":"2024-03-26T03:18:46.254872+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:51.516127+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:17:51.516127+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:09.405345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"230'21","reported_seq":864,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248673+0000","last_change":"2024-03-26T03:16:46.178208+0000","last_active":"2024-03-26T03:18:46.248673+0000","last_peered":"2024-03-26T03:18:46.248673+0000","last_clean":"2024-03-26T03:18:46.248673+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:18:46.248673+0000","last_undegraded":"2024-03-26T03:18:46.248673+0000","last_fullsized":"2024-03-26T03:18:46.248673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:16:46.178099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.178099+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":501,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.259088+0000","last_change":"2024-03-26T03:14:22.572664+0000","last_active":"2024-03-26T03:18:46.259088+0000","last_peered":"2024-03-26T03:18:46.259088+0000","last_clean":"2024-03-26T03:18:46.259088+0000","last_became_active":"2024-03-26T03:14:22.571857+0000","last_became_peered":"2024-03-26T03:14:22.571857+0000","last_unstale":"2024-03-26T03:18:46.259088+0000","last_undegraded":"2024-03-26T03:18:46.259088+0000","last_fullsized":"2024-03-26T03:18:46.259088+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:25.976630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:05:25.976630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":831,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.253166+0000","last_change":"2024-03-26T03:15:20.809430+0000","last_active":"2024-03-26T03:18:46.253166+0000","last_peered":"2024-03-26T03:18:46.253166+0000","last_clean":"2024-03-26T03:18:46.253166+0000","last_became_active":"2024-03-26T03:14:37.242540+0000","last_became_peered":"2024-03-26T03:14:37.242540+0000","last_unstale":"2024-03-26T03:18:46.253166+0000","last_undegraded":"2024-03-26T03:18:46.253166+0000","last_fullsized":"2024-03-26T03:18:46.253166+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:20.809176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:20.809176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":826,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.249747+0000","last_change":"2024-03-26T03:17:43.076516+0000","last_active":"2024-03-26T03:18:46.249747+0000","last_peered":"2024-03-26T03:18:46.249747+0000","last_clean":"2024-03-26T03:18:46.249747+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:18:46.249747+0000","last_undegraded":"2024-03-26T03:18:46.249747+0000","last_fullsized":"2024-03-26T03:18:46.249747+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.076334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.076334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:49.381353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":522,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.258669+0000","last_change":"2024-03-26T03:16:29.882781+0000","last_active":"2024-03-26T03:18:46.258669+0000","last_peered":"2024-03-26T03:18:46.258669+0000","last_clean":"2024-03-26T03:18:46.258669+0000","last_became_active":"2024-03-26T03:14:40.174111+0000","last_became_peered":"2024-03-26T03:14:40.174111+0000","last_unstale":"2024-03-26T03:18:46.258669+0000","last_undegraded":"2024-03-26T03:18:46.258669+0000","last_fullsized":"2024-03-26T03:18:46.258669+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.882681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.882681+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"231'52","reported_seq":589,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.257019+0000","last_change":"2024-03-26T03:14:22.625030+0000","last_active":"2024-03-26T03:18:46.257019+0000","last_peered":"2024-03-26T03:18:46.257019+0000","last_clean":"2024-03-26T03:18:46.257019+0000","last_became_active":"2024-03-26T03:14:22.576815+0000","last_became_peered":"2024-03-26T03:14:22.576815+0000","last_unstale":"2024-03-26T03:18:46.257019+0000","last_undegraded":"2024-03-26T03:18:46.257019+0000","last_fullsized":"2024-03-26T03:18:46.257019+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'40","last_scrub_stamp":"2024-03-26T03:05:32.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:05:32.082581+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":65,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":227,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.250287+0000","last_change":"2024-03-26T03:16:48.135457+0000","last_active":"2024-03-26T03:18:46.250287+0000","last_peered":"2024-03-26T03:18:46.250287+0000","last_clean":"2024-03-26T03:18:46.250287+0000","last_became_active":"2024-03-26T03:14:37.248588+0000","last_became_peered":"2024-03-26T03:14:37.248588+0000","last_unstale":"2024-03-26T03:18:46.250287+0000","last_undegraded":"2024-03-26T03:18:46.250287+0000","last_fullsized":"2024-03-26T03:18:46.250287+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:48.135354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.135354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":41,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.248827+0000","last_change":"2024-03-26T03:17:56.572431+0000","last_active":"2024-03-26T03:18:46.248827+0000","last_peered":"2024-03-26T03:18:46.248827+0000","last_clean":"2024-03-26T03:18:46.248827+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:18:46.248827+0000","last_undegraded":"2024-03-26T03:18:46.248827+0000","last_fullsized":"2024-03-26T03:18:46.248827+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.615644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.615644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:47.803583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":492,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-03-26T03:18:46.255308+0000","last_change":"2024-03-26T03:14:37.269042+0000","last_active":"2024-03-26T03:18:46.255308+0000","last_peered":"2024-03-26T03:18:46.255308+0000","last_clean":"2024-03-26T03:18:46.255308+0000","last_became_active":"2024-03-26T03:14:37.265472+0000","last_became_peered":"2024-03-26T03:14:37.265472+0000","last_unstale":"2024-03-26T03:18:46.255308+0000","last_undegraded":"2024-03-26T03:18:46.255308+0000","last_fullsized":"2024-03-26T03:18:46.255308+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:05:17.983183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:05:17.983183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:18:48.581058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7808352280,"num_objects":119147,"num_object_clones":0,"num_object_copies":357441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4724,"num_objects_unfound":0,"num_objects_dirty":119147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144645,"num_write_kb":8441281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16408,"num_bytes_recovered":1075314688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24059064320,"data_stored":24059052104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52754,"ondisk_log_size":52754,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603785,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":4,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":958300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1182,"num_read_kb":1177,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":3014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":958,"ondisk_log_size":958,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430455,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5598972,"kb_used_data":4809780,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":88146180,"statfs":{"total":95995035648,"available":90261688320,"internally_reserved":0,"allocated":4925214720,"data_stored":4924716614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":808122822},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575120,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6246176,"kb_used_data":5236120,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":87498976,"statfs":{"total":95995035648,"available":89598951424,"internally_reserved":0,"allocated":5361786880,"data_stored":5361254052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1034156485},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738645,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411594,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4196164,"kb_used_data":3370812,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":89548988,"statfs":{"total":95995035648,"available":91698163712,"internally_reserved":0,"allocated":3451711488,"data_stored":3451206639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175031,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4708888,"kb_used_data":4299472,"kb_used_omap":4,"kb_used_meta":409403,"kb_avail":89036264,"statfs":{"total":95995035648,"available":91173134336,"internally_reserved":0,"allocated":4402659328,"data_stored":4402133171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":419229347},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986433,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3857220,"kb_used_data":3045820,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":89887932,"statfs":{"total":95995035648,"available":92045242368,"internally_reserved":0,"allocated":3118919680,"data_stored":3118414182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692093,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3848204,"kb_used_data":2882692,"kb_used_omap":1,"kb_used_meta":965502,"kb_avail":89896948,"statfs":{"total":95995035648,"available":92054474752,"internally_reserved":0,"allocated":2951876608,"data_stored":2951363046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":988674501},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5321785344,"data_stored":5321785344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3425308672,"data_stored":3425304600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4400283648,"data_stored":4400283648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3109949440,"data_stored":3109945368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2922184704,"data_stored":2922184704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4879552512,"data_stored":4879548440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:18:55.521 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph status --format=json 2024-03-26T03:18:55.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:55.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:55.700 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:55.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:55.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: pgmap v944: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 4724/358131 objects misplaced (1.319%); 19 MiB/s, 308 objects/s recovering 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: from='client.35945 v1:172.21.15.66:0/3397246165' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: 1.10 scrub starts 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: 1.10 scrub ok 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[29666]: 11.5 scrub ok 2024-03-26T03:18:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: pgmap v944: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 4724/358131 objects misplaced (1.319%); 19 MiB/s, 308 objects/s recovering 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: from='client.35945 v1:172.21.15.66:0/3397246165' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: 1.10 scrub starts 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: 1.10 scrub ok 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:55 smithi066 ceph-mon[37920]: 11.5 scrub ok 2024-03-26T03:18:55.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:55.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:55.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: pgmap v944: 178 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfilling, 175 active+clean; 7.3 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s; 4724/358131 objects misplaced (1.319%); 19 MiB/s, 308 objects/s recovering 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: from='client.35945 v1:172.21.15.66:0/3397246165' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: 1.10 scrub starts 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: 1.10 scrub ok 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:18:55.939 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:18:55.940 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:18:55.940 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:55 smithi192 ceph-mon[33110]: 11.5 scrub ok 2024-03-26T03:18:56.000 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:56.030 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:56.131 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:56.160 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:56.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:56.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:56.387 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.416 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:56.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:56.517 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:56.546 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[29666]: osdmap e444: 8 total, 8 up, 6 in 2024-03-26T03:18:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[29666]: 6.5 scrub starts 2024-03-26T03:18:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[29666]: 6.5 scrub ok 2024-03-26T03:18:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[29666]: pgmap v946: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 174 active+clean; 6.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 3138/338730 objects misplaced (0.926%); 16 MiB/s, 252 objects/s recovering 2024-03-26T03:18:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[37920]: osdmap e444: 8 total, 8 up, 6 in 2024-03-26T03:18:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[37920]: 6.5 scrub starts 2024-03-26T03:18:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[37920]: 6.5 scrub ok 2024-03-26T03:18:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:56 smithi066 ceph-mon[37920]: pgmap v946: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 174 active+clean; 6.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 3138/338730 objects misplaced (0.926%); 16 MiB/s, 252 objects/s recovering 2024-03-26T03:18:56.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:56.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:56.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:56.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:56.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:56.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:56.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:56 smithi192 ceph-mon[33110]: osdmap e444: 8 total, 8 up, 6 in 2024-03-26T03:18:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:56 smithi192 ceph-mon[33110]: 6.5 scrub starts 2024-03-26T03:18:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:56 smithi192 ceph-mon[33110]: 6.5 scrub ok 2024-03-26T03:18:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:56 smithi192 ceph-mon[33110]: pgmap v946: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 174 active+clean; 6.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 3138/338730 objects misplaced (0.926%); 16 MiB/s, 252 objects/s recovering 2024-03-26T03:18:57.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:18:57.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:18:57.172 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:57.226 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:57.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:57.372 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:57.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:57.518 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:57.620 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.664 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:57.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:57.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:57.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[29666]: osdmap e445: 8 total, 8 up, 6 in 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[37920]: osdmap e445: 8 total, 8 up, 6 in 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:18:57.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:18:57.863 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:57 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:57.905 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:57.949 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:57.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:57 smithi192 ceph-mon[33110]: osdmap e445: 8 total, 8 up, 6 in 2024-03-26T03:18:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:57 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:18:57.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:57 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:18:57.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:57 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:18:58.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:18:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:58.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:18:58.185 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:58.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:58.327 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:58.370 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:58.472 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:58.518 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:58.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:58.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:58.754 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:58.794 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:58.801 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:18:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:58 smithi066 ceph-mon[29666]: 6.1f scrub starts 2024-03-26T03:18:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:58 smithi066 ceph-mon[29666]: 6.1f scrub ok 2024-03-26T03:18:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:58 smithi066 ceph-mon[29666]: pgmap v948: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 174 active+clean; 6.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 3138/338730 objects misplaced (0.926%); 20 MiB/s, 315 objects/s recovering 2024-03-26T03:18:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:58 smithi066 ceph-mon[37920]: 6.1f scrub starts 2024-03-26T03:18:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:58 smithi066 ceph-mon[37920]: 6.1f scrub ok 2024-03-26T03:18:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:58 smithi066 ceph-mon[37920]: pgmap v948: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 174 active+clean; 6.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 3138/338730 objects misplaced (0.926%); 20 MiB/s, 315 objects/s recovering 2024-03-26T03:18:58.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:58.921 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:58.959 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:18:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:58 smithi192 ceph-mon[33110]: 6.1f scrub starts 2024-03-26T03:18:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:58 smithi192 ceph-mon[33110]: 6.1f scrub ok 2024-03-26T03:18:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:58 smithi192 ceph-mon[33110]: pgmap v948: 178 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+remapped+backfilling, 174 active+clean; 6.9 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 3138/338730 objects misplaced (0.926%); 20 MiB/s, 315 objects/s recovering 2024-03-26T03:18:58.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:58.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:59.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:18:59.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:18:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:59.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:18:59.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:59.292 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:59.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:59.394 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:18:59.440 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:59.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:18:59.542 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:18:59.588 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:59.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:18:59.690 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:18:59.733 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:59.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:18:59.835 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:18:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[29666]: 6.1b scrub starts 2024-03-26T03:18:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[29666]: 6.1b scrub ok 2024-03-26T03:18:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:18:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:18:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[37920]: 6.1b scrub starts 2024-03-26T03:18:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[37920]: 6.1b scrub ok 2024-03-26T03:18:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:18:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:18:59 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:18:59.882 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:18:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:18:59.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:18:59.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:18:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:59 smithi192 ceph-mon[33110]: 6.1b scrub starts 2024-03-26T03:18:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:59 smithi192 ceph-mon[33110]: 6.1b scrub ok 2024-03-26T03:18:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:59 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:18:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:18:59 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:19:00.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:00.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:00.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:00.252 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:00.283 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:00.385 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:00.415 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:00.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:19:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:19:00.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:00.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:00.654 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:00.696 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:00.798 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.845 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:00.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:00.869 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[29666]: 6.16 scrub starts 2024-03-26T03:19:00.869 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[29666]: 6.16 scrub ok 2024-03-26T03:19:00.869 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:00.869 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[29666]: pgmap v949: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.37k op/s; 3138/325152 objects misplaced (0.965%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:19:00.869 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[37920]: 6.16 scrub starts 2024-03-26T03:19:00.869 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[37920]: 6.16 scrub ok 2024-03-26T03:19:00.870 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:00.870 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:00 smithi066 ceph-mon[37920]: pgmap v949: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.37k op/s; 3138/325152 objects misplaced (0.965%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:19:00.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:00 smithi192 ceph-mon[33110]: 6.16 scrub starts 2024-03-26T03:19:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:00 smithi192 ceph-mon[33110]: 6.16 scrub ok 2024-03-26T03:19:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:00 smithi192 ceph-mon[33110]: pgmap v949: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.6 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.37k op/s; 3138/325152 objects misplaced (0.965%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:19:00.994 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:00.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:01.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:01.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:01.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:01.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:01.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:01.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:01.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:01.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:01.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:01.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:01.693 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[29666]: 6.b scrub starts 2024-03-26T03:19:01.693 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:19:01.693 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:19:01.693 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[29666]: 6.b scrub ok 2024-03-26T03:19:01.693 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[37920]: 6.b scrub starts 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[37920]: 6.b scrub ok 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:01.694 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:01 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:01.770 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:01.821 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:01.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:01.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:01.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:01 smithi192 ceph-mon[33110]: 6.b scrub starts 2024-03-26T03:19:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:01 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:19:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:01 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:19:01.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:01 smithi192 ceph-mon[33110]: 6.b scrub ok 2024-03-26T03:19:01.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:01 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:01.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:01 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:02.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:02.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:02.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:02.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:02.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:02.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:02.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:02.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:02.599 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:02.647 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:02.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:02.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:02.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:02.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:02.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:02 smithi192 ceph-mon[33110]: osdmap e446: 8 total, 8 up, 6 in 2024-03-26T03:19:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:02 smithi192 ceph-mon[33110]: 1.1c scrub starts 2024-03-26T03:19:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:02 smithi192 ceph-mon[33110]: 5.17 scrub starts 2024-03-26T03:19:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:02 smithi192 ceph-mon[33110]: 1.1c scrub ok 2024-03-26T03:19:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:02 smithi192 ceph-mon[33110]: 5.17 scrub ok 2024-03-26T03:19:02.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:02 smithi192 ceph-mon[33110]: pgmap v951: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s; 1978/320676 objects misplaced (0.617%); 22 MiB/s, 352 objects/s recovering 2024-03-26T03:19:03.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:03.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:03.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[29666]: osdmap e446: 8 total, 8 up, 6 in 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[29666]: 1.1c scrub starts 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[29666]: 5.17 scrub starts 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[29666]: 1.1c scrub ok 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[29666]: 5.17 scrub ok 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[29666]: pgmap v951: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s; 1978/320676 objects misplaced (0.617%); 22 MiB/s, 352 objects/s recovering 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[37920]: osdmap e446: 8 total, 8 up, 6 in 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[37920]: 1.1c scrub starts 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[37920]: 5.17 scrub starts 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[37920]: 1.1c scrub ok 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[37920]: 5.17 scrub ok 2024-03-26T03:19:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:02 smithi066 ceph-mon[37920]: pgmap v951: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.18k op/s; 1978/320676 objects misplaced (0.617%); 22 MiB/s, 352 objects/s recovering 2024-03-26T03:19:03.151 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.192 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:03.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:03.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:03.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:03.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:03.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:03.561 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.606 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:03.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:03.707 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:03.751 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:03.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:03.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:03.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:03.919 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:19:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:19:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:19:03.985 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:04.028 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:04.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:04.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:04.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:03 smithi192 ceph-mon[33110]: 3.2 scrub starts 2024-03-26T03:19:04.237 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:03 smithi192 ceph-mon[33110]: 3.2 scrub ok 2024-03-26T03:19:04.266 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.308 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:04.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:04.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:03 smithi066 ceph-mon[29666]: 3.2 scrub starts 2024-03-26T03:19:04.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:03 smithi066 ceph-mon[29666]: 3.2 scrub ok 2024-03-26T03:19:04.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:03 smithi066 ceph-mon[37920]: 3.2 scrub starts 2024-03-26T03:19:04.353 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:03 smithi066 ceph-mon[37920]: 3.2 scrub ok 2024-03-26T03:19:04.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:04.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:04.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:04.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:04.680 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:04.723 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:04.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:04.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:04.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:04.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:04.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:05.085 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:05.131 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:05.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:05.232 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:05.276 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:05.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:05.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:05.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:05.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:05.510 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:05.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:05.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:05.590 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:05 smithi066 ceph-mon[29666]: pgmap v952: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s; 1978/320676 objects misplaced (0.617%); 8.1 MiB/s, 129 objects/s recovering 2024-03-26T03:19:05.590 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:05 smithi066 ceph-mon[29666]: 6.13 scrub starts 2024-03-26T03:19:05.591 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:05 smithi066 ceph-mon[29666]: 6.13 scrub ok 2024-03-26T03:19:05.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:05 smithi066 ceph-mon[37920]: pgmap v952: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s; 1978/320676 objects misplaced (0.617%); 8.1 MiB/s, 129 objects/s recovering 2024-03-26T03:19:05.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:05 smithi066 ceph-mon[37920]: 6.13 scrub starts 2024-03-26T03:19:05.591 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:05 smithi066 ceph-mon[37920]: 6.13 scrub ok 2024-03-26T03:19:05.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:05.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:05.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:05 smithi192 ceph-mon[33110]: pgmap v952: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s; 1978/320676 objects misplaced (0.617%); 8.1 MiB/s, 129 objects/s recovering 2024-03-26T03:19:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:05 smithi192 ceph-mon[33110]: 6.13 scrub starts 2024-03-26T03:19:05.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:05 smithi192 ceph-mon[33110]: 6.13 scrub ok 2024-03-26T03:19:05.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:05.843 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:05.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:05.945 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:05.976 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:19:05.976 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:19:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.000 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:06.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:06.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:06.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:06.235 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:06.283 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:06.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:06.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:06.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:06.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:06.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:19:06.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:19:06.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:19:06.563 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:19:06.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:19:06.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:19:06.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:19:06.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:06 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:19:06.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:06.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:06 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:19:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:06 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:19:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:06 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:19:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:06 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:19:06.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:06.838 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:06.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:06.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:06.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:07.071 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.130 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:07.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:07.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:07.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:07.276 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:19:07.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: pgmap v953: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.1 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 15 KiB/s rd, 0 B/s wr, 1.39k op/s; 1215/298314 objects misplaced (0.407%); 11 MiB/s, 172 objects/s recovering 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: 1.7 scrub starts 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: 1.7 scrub ok 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: 2.1e scrub starts 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: 2.1e scrub ok 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3365180855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:19:07.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:19:07.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: pgmap v953: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.1 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 15 KiB/s rd, 0 B/s wr, 1.39k op/s; 1215/298314 objects misplaced (0.407%); 11 MiB/s, 172 objects/s recovering 2024-03-26T03:19:07.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: 1.7 scrub starts 2024-03-26T03:19:07.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: 1.7 scrub ok 2024-03-26T03:19:07.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: 2.1e scrub starts 2024-03-26T03:19:07.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: 2.1e scrub ok 2024-03-26T03:19:07.360 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3365180855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:07.365 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:07.414 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:07.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:07.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:07.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:07.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:19:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:19:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: pgmap v953: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.1 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 15 KiB/s rd, 0 B/s wr, 1.39k op/s; 1215/298314 objects misplaced (0.407%); 11 MiB/s, 172 objects/s recovering 2024-03-26T03:19:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: 1.7 scrub starts 2024-03-26T03:19:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: 1.7 scrub ok 2024-03-26T03:19:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: 2.1e scrub starts 2024-03-26T03:19:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: 2.1e scrub ok 2024-03-26T03:19:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3365180855' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:07.739 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:19:07.782 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:07.832 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:07.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:07.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:07.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:08.067 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.143 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:08.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:08.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.275 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:08.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:08.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:08.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3365180855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: osdmap e447: 8 total, 8 up, 6 in 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: 4.16 scrub starts 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[29666]: 4.16 scrub ok 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:19:08.419 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:19:08.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3365180855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:19:08.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: osdmap e447: 8 total, 8 up, 6 in 2024-03-26T03:19:08.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:08.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:08.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: 4.16 scrub starts 2024-03-26T03:19:08.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:08 smithi066 ceph-mon[37920]: 4.16 scrub ok 2024-03-26T03:19:08.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:08.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:08.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:08.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:19:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:19:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3365180855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:19:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: osdmap e447: 8 total, 8 up, 6 in 2024-03-26T03:19:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: 4.16 scrub starts 2024-03-26T03:19:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:08 smithi192 ceph-mon[33110]: 4.16 scrub ok 2024-03-26T03:19:08.773 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:08.822 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:08.924 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:08.966 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:08.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:09.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:09.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:09.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:09.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:09.336 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:09.381 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:09.484 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.531 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:09.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[29666]: osdmap e448: 8 total, 8 up, 6 in 2024-03-26T03:19:09.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[29666]: pgmap v956: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.1 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 1.13k op/s; 1215/298314 objects misplaced (0.407%); 6.0 MiB/s, 96 objects/s recovering 2024-03-26T03:19:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:19:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:19:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[37920]: osdmap e448: 8 total, 8 up, 6 in 2024-03-26T03:19:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[37920]: pgmap v956: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.1 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 1.13k op/s; 1215/298314 objects misplaced (0.407%); 6.0 MiB/s, 96 objects/s recovering 2024-03-26T03:19:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:19:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:09 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:19:09.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:09.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:09 smithi192 ceph-mon[33110]: osdmap e448: 8 total, 8 up, 6 in 2024-03-26T03:19:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:09 smithi192 ceph-mon[33110]: pgmap v956: 178 pgs: 1 active+remapped+backfilling, 177 active+clean; 6.1 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 1.13k op/s; 1215/298314 objects misplaced (0.407%); 6.0 MiB/s, 96 objects/s recovering 2024-03-26T03:19:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:09 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:19:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:09 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:19:09.768 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:09.796 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:09.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:09.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:09.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:10.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:10.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:10.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:10.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:10.243 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:10.300 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:10.375 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:10.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:10 smithi192 ceph-mon[33110]: 4.19 scrub starts 2024-03-26T03:19:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:10 smithi192 ceph-mon[33110]: 4.19 scrub ok 2024-03-26T03:19:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:10.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:10.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:19:10 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:19:10.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:10.608 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[29666]: 4.19 scrub starts 2024-03-26T03:19:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[29666]: 4.19 scrub ok 2024-03-26T03:19:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:10.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[37920]: 4.19 scrub starts 2024-03-26T03:19:10.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[37920]: 4.19 scrub ok 2024-03-26T03:19:10.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:10.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.653 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:10.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:10.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:10.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:10.880 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:10.929 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:10.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:11.029 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:19:11.029 INFO:teuthology.orchestra.run.smithi066.stdout:{"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1433,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":449,"num_osds":8,"num_up_osds":8,"osd_up_since":1711422860,"num_in_osds":6,"osd_in_since":1711422876,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":177},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":178,"num_pools":7,"num_objects":99438,"data_bytes":6502239785,"bytes_used":25391972352,"bytes_avail":550578241536,"bytes_total":575970213888,"misplaced_objects":1215,"misplaced_total":298314,"misplaced_ratio":0.0040728896397755386,"recovering_objects_per_sec":96,"recovering_bytes_per_sec":6332703,"recovering_keys_per_sec":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"read_bytes_sec":20870,"write_bytes_sec":0,"read_op_per_sec":20,"write_op_per_sec":1106},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.66:8443/","prometheus":"http://172.21.15.66:9283/"}},"servicemap":{"epoch":16,"modified":"2024-03-26T03:15:40.364401+0000","services":{"rgw":{"daemons":{"summary":"","34346":{"start_epoch":12,"start_stamp":"2024-03-26T02:59:04.062332+0000","gid":34346,"addr":"172.21.15.66:0/2004587607","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)","ceph_version_short":"19.0.0-2475-ge42b8ea3","container_hostname":"smithi066","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi066","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024","kernel_version":"5.14.0-432.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488316","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"57c86d59-a969-4489-a362-518b8ef1fb92","zone_name":"default","zonegroup_id":"a30734c1-8553-48eb-b7d7-d8249c7a77bc","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-26T03:19:11.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:11.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:11.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:11.190 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:11.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:11.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:11.425 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:11.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:19:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:11.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:11.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: osdmap e449: 8 total, 8 up, 6 in 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: pgmap v958: 178 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 175 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 66 KiB/s rd, 0 B/s wr, 2.22k op/s; 1215/282612 objects misplaced (0.430%); 6.9 MiB/s, 109 objects/s recovering 2024-03-26T03:19:11.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1392123144' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: osdmap e449: 8 total, 8 up, 6 in 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: pgmap v958: 178 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 175 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 66 KiB/s rd, 0 B/s wr, 2.22k op/s; 1215/282612 objects misplaced (0.430%); 6.9 MiB/s, 109 objects/s recovering 2024-03-26T03:19:11.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1392123144' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-26T03:19:11.628 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:11.730 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:19:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:19:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:11.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: osdmap e449: 8 total, 8 up, 6 in 2024-03-26T03:19:11.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: pgmap v958: 178 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 175 active+clean; 5.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 66 KiB/s rd, 0 B/s wr, 2.22k op/s; 1215/282612 objects misplaced (0.430%); 6.9 MiB/s, 109 objects/s recovering 2024-03-26T03:19:11.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1392123144' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-26T03:19:11.781 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.782 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:11.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:11.883 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:11.940 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:11.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:12.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:12.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:12.181 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:12.225 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:12.328 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:12.377 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:12.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:12.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:12 smithi066 ceph-mon[29666]: 4.1e scrub starts 2024-03-26T03:19:12.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:12 smithi066 ceph-mon[29666]: 4.1e scrub ok 2024-03-26T03:19:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:12 smithi066 ceph-mon[37920]: 4.1e scrub starts 2024-03-26T03:19:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:12 smithi066 ceph-mon[37920]: 4.1e scrub ok 2024-03-26T03:19:12.616 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:12.661 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:12.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:12 smithi192 ceph-mon[33110]: 4.1e scrub starts 2024-03-26T03:19:12.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:12 smithi192 ceph-mon[33110]: 4.1e scrub ok 2024-03-26T03:19:12.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:12.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:12.899 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:12.945 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:12.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:13.047 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:13.090 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:13.181 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:13.192 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:13.250 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:13.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:13.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:13.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:13.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:13.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:19:13.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:19:13.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[29666]: 11.4 scrub starts 2024-03-26T03:19:13.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[29666]: pgmap v959: 178 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 175 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 50 KiB/s rd, 0 B/s wr, 1.25k op/s; 548/277305 objects misplaced (0.198%); 6.2 MiB/s, 99 objects/s recovering 2024-03-26T03:19:13.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[29666]: 6.6 deep-scrub starts 2024-03-26T03:19:13.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[29666]: 6.6 deep-scrub ok 2024-03-26T03:19:13.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:19:13.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:19:13.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[37920]: 11.4 scrub starts 2024-03-26T03:19:13.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[37920]: pgmap v959: 178 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 175 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 50 KiB/s rd, 0 B/s wr, 1.25k op/s; 548/277305 objects misplaced (0.198%); 6.2 MiB/s, 99 objects/s recovering 2024-03-26T03:19:13.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[37920]: 6.6 deep-scrub starts 2024-03-26T03:19:13.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-mon[37920]: 6.6 deep-scrub ok 2024-03-26T03:19:13.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:13.637 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:13 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:19:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:13 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:19:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:13 smithi192 ceph-mon[33110]: 11.4 scrub starts 2024-03-26T03:19:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:13 smithi192 ceph-mon[33110]: pgmap v959: 178 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 175 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 50 KiB/s rd, 0 B/s wr, 1.25k op/s; 548/277305 objects misplaced (0.198%); 6.2 MiB/s, 99 objects/s recovering 2024-03-26T03:19:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:13 smithi192 ceph-mon[33110]: 6.6 deep-scrub starts 2024-03-26T03:19:13.638 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:13 smithi192 ceph-mon[33110]: 6.6 deep-scrub ok 2024-03-26T03:19:13.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:13.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:13.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:13.859 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:19:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:19:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:19:13.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:13.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:13.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:13.994 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-03-26T03:19:13.994 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:19:14.038 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.100 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:14.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:14.201 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:14.263 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:14.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:14.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:14.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:14.459 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:14 smithi066 ceph-mon[29666]: 11.4 scrub ok 2024-03-26T03:19:14.460 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:14 smithi066 ceph-mon[37920]: 11.4 scrub ok 2024-03-26T03:19:14.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:14.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:14.627 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:14.675 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:14 smithi192 ceph-mon[33110]: 11.4 scrub ok 2024-03-26T03:19:14.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:14.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:14.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:14.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:14.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:15.041 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:15.084 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:15.136 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:19:15.136 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:19:15.136 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:19:15.136 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:19:15.137 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:19:15.138 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:19:15.186 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:15.234 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:15.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:15.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:15.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:15.486 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[29666]: pgmap v960: 178 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 175 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.04k op/s; 548/277305 objects misplaced (0.198%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/903872307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[37920]: pgmap v960: 178 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 175 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.04k op/s; 548/277305 objects misplaced (0.198%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T03:19:15.487 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:15 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/903872307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.509 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:15.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:15.610 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:15.637 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:15 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:19:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:15 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:19:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:15 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:19:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:15 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:19:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:15 smithi192 ceph-mon[33110]: pgmap v960: 178 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 175 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.04k op/s; 548/277305 objects misplaced (0.198%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T03:19:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:15 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/903872307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:15.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:15.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:15.864 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:15.892 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:15.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:15.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:16.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:16.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:16.148 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:16.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:16.277 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:16.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:16.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:16.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:16.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:16.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:16.641 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:16.685 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:16.687 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:16 smithi066 ceph-mon[29666]: osdmap e450: 8 total, 8 up, 6 in 2024-03-26T03:19:16.687 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:16 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:19:16.687 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:16 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:19:16.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:16 smithi066 ceph-mon[37920]: osdmap e450: 8 total, 8 up, 6 in 2024-03-26T03:19:16.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:16 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:19:16.688 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:16 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:19:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:16 smithi192 ceph-mon[33110]: osdmap e450: 8 total, 8 up, 6 in 2024-03-26T03:19:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:16 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:19:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:16 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:19:16.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:16.846 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:16.949 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:16.997 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:16.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:17.099 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:17.145 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:17.247 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.310 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:17.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:17.411 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:17.436 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[29666]: osdmap e451: 8 total, 8 up, 6 in 2024-03-26T03:19:17.436 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[29666]: 1.e deep-scrub starts 2024-03-26T03:19:17.436 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[29666]: 1.e deep-scrub ok 2024-03-26T03:19:17.436 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[29666]: pgmap v963: 178 pgs: 1 peering, 177 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 1.43k op/s; 11 MiB/s, 182 objects/s recovering 2024-03-26T03:19:17.438 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[37920]: osdmap e451: 8 total, 8 up, 6 in 2024-03-26T03:19:17.438 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[37920]: 1.e deep-scrub starts 2024-03-26T03:19:17.438 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[37920]: 1.e deep-scrub ok 2024-03-26T03:19:17.438 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:17 smithi066 ceph-mon[37920]: pgmap v963: 178 pgs: 1 peering, 177 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 1.43k op/s; 11 MiB/s, 182 objects/s recovering 2024-03-26T03:19:17.461 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:17.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:17.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:17.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:17.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:17 smithi192 ceph-mon[33110]: osdmap e451: 8 total, 8 up, 6 in 2024-03-26T03:19:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:17 smithi192 ceph-mon[33110]: 1.e deep-scrub starts 2024-03-26T03:19:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:17 smithi192 ceph-mon[33110]: 1.e deep-scrub ok 2024-03-26T03:19:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:17 smithi192 ceph-mon[33110]: pgmap v963: 178 pgs: 1 peering, 177 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 8.1 KiB/s rd, 0 B/s wr, 1.43k op/s; 11 MiB/s, 182 objects/s recovering 2024-03-26T03:19:17.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:17.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:17.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:17.964 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:19:17.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:17.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:18.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:18.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:18.125 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:19:18.125 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:19:18.125 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:19:18.126 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:19:18.126 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:19:18.218 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:18.280 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:18.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:18.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:18.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:18.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:19:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:19:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:19:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:18 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:19:18.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:18.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:18 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:19:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:18 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:19:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:18 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:19:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:18 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:19:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:18 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:19:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:18 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:19:18.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:18.833 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:18.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:18.945 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:18.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:18.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:18.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:19:19.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:19.121 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:19.223 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:19.284 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:19.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:19.413 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:19:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:19.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:19.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:19.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[29666]: 11.1 scrub starts 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[29666]: pgmap v964: 178 pgs: 1 peering, 177 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 8.7 MiB/s, 139 objects/s recovering 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3403544313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[29666]: from='client.25906 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[37920]: 11.1 scrub starts 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[37920]: pgmap v964: 178 pgs: 1 peering, 177 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 8.7 MiB/s, 139 objects/s recovering 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3403544313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[37920]: from='client.25906 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:19:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:19 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:19:19.655 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:19.702 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:19 smithi192 ceph-mon[33110]: 11.1 scrub starts 2024-03-26T03:19:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:19 smithi192 ceph-mon[33110]: pgmap v964: 178 pgs: 1 peering, 177 active+clean; 5.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 1.09k op/s; 8.7 MiB/s, 139 objects/s recovering 2024-03-26T03:19:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:19 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3403544313' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:19:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:19 smithi192 ceph-mon[33110]: from='client.25906 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:19:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:19 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:19:19.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:19 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:19:19.804 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:19.850 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:19.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:19.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:19.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:20.022 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:20.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:20.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:20.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:20.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:20.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:20.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:20.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:19:20 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:19:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:20 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:19:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:20 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:19:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:20 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:20 smithi192 ceph-mon[33110]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:19:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:20 smithi192 ceph-mon[33110]: osdmap e452: 8 total, 8 up, 6 in 2024-03-26T03:19:20.491 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[29666]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[29666]: osdmap e452: 8 total, 8 up, 6 in 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:19:20.513 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:19:20.514 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:20.514 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[37920]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:19:20.514 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:20 smithi066 ceph-mon[37920]: osdmap e452: 8 total, 8 up, 6 in 2024-03-26T03:19:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.538 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:20.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:20.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:20.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:20.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:20.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:20.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:20.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:20.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:21.043 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.087 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:21.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:21.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:21.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:21.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:21.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:21.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:21.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:21.583 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:21.625 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:21.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:21 smithi192 ceph-mon[33110]: pgmap v966: 178 pgs: 1 peering, 177 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 1.89k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-03-26T03:19:21.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:21.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:21.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:21 smithi066 ceph-mon[29666]: pgmap v966: 178 pgs: 1 peering, 177 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 1.89k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-03-26T03:19:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:21.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:21 smithi066 ceph-mon[37920]: pgmap v966: 178 pgs: 1 peering, 177 active+clean; 4.9 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 1.89k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-03-26T03:19:21.860 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:21.905 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:21.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:22.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:22.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:22.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:22.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:22.269 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:22.315 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:22.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:22.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:22.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:22.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:22 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:22 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:22.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:22 smithi066 ceph-mon[29666]: osdmap e453: 8 total, 8 up, 6 in 2024-03-26T03:19:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:22 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:22 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:22.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:22 smithi066 ceph-mon[37920]: osdmap e453: 8 total, 8 up, 6 in 2024-03-26T03:19:22.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:22 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:22 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:22 smithi192 ceph-mon[33110]: osdmap e453: 8 total, 8 up, 6 in 2024-03-26T03:19:22.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:22.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:22.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:22.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:22.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:23.080 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:23.126 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:23.228 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:23.248 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:19:23.254 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:19:23.276 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:23.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:23.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:23.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:23.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:23.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:23.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:23 smithi192 ceph-mon[33110]: pgmap v968: 178 pgs: 178 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 985 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:23.738 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:23 smithi066 ceph-mon[29666]: pgmap v968: 178 pgs: 178 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 985 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:23.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:23 smithi066 ceph-mon[37920]: pgmap v968: 178 pgs: 178 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 985 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:23.772 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:23.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:23.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:23.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:23.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:24.001 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:19:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:19:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:19:24.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:24.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:24.173 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:24.219 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:24.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:24.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:24.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:24.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:24.581 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:24.623 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:24.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:24 smithi192 ceph-mon[33110]: from='client.25912 v1:172.21.15.66:0/4034980413' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:19:24.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:24.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:24.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:24 smithi066 ceph-mon[29666]: from='client.25912 v1:172.21.15.66:0/4034980413' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:19:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:24 smithi066 ceph-mon[37920]: from='client.25912 v1:172.21.15.66:0/4034980413' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:19:24.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:24.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:24.991 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:25.035 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:25.136 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:25.169 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:25.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:25.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:25.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:25.404 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.432 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:25.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:25.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:25.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:25.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:25.702 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:25 smithi192 ceph-mon[33110]: pgmap v969: 178 pgs: 178 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 749 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:25.804 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:25.846 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:25 smithi066 ceph-mon[29666]: pgmap v969: 178 pgs: 178 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 749 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:25 smithi066 ceph-mon[37920]: pgmap v969: 178 pgs: 178 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s rd, 0 B/s wr, 749 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:25.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:25.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:25.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:26.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:26.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:26.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:26.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:26.350 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:26.383 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:26.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:26.533 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:26.635 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:26.676 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:26.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:26.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:26.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:26.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:26.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:26.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:27.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:27.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:27.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:27.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:27.302 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.350 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:27.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:27.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:27.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:27.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:27.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:27.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:27.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:27.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:27.842 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:27.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:27 smithi066 ceph-mon[29666]: pgmap v970: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.70k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:27.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:27 smithi066 ceph-mon[37920]: pgmap v970: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.70k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:27.885 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:27.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:27.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:27.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:27 smithi192 ceph-mon[33110]: pgmap v970: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 383 B/s rd, 0 B/s wr, 1.70k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:28.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:28.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:28.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:28.260 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.302 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:28.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:28.403 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.448 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:28.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:28.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.595 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:28.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:28.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:28.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:28.770 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:28.831 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:28.898 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:28.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:29.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:29.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:29.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:29.180 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:29.240 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":968,"stamp":"2024-03-26T03:19:22.451910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5197142464,"num_objects":79524,"num_object_clones":0,"num_object_copies":238572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79524,"num_whiteouts":0,"num_read":3058,"num_read_kb":2778,"num_write":185698,"num_write_kb":8441326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20849,"num_bytes_recovered":1351161765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58428,"ondisk_log_size":58428,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":535,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":20778760,"kb_used_data":15752320,"kb_used_omap":12,"kb_used_meta":5025843,"kb_avail":541692152,"statfs":{"total":575970213888,"available":554692763648,"internally_reserved":0,"allocated":16130375680,"data_stored":16127891635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12419,"internal_metadata":5146464125},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-393032809,"num_objects":-5997,"num_object_clones":0,"num_object_copies":-17991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5997,"num_whiteouts":0,"num_read":1,"num_read_kb":-2,"num_write":5993,"num_write_kb":-55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":-40252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.081663"},"pg_stats":[{"pgid":"11.7","version":"452'11438","reported_seq":11628,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.782146+0000","last_change":"2024-03-26T03:18:52.598875+0000","last_active":"2024-03-26T03:19:19.782146+0000","last_peered":"2024-03-26T03:19:19.782146+0000","last_clean":"2024-03-26T03:19:19.782146+0000","last_became_active":"2024-03-26T03:16:56.900984+0000","last_became_peered":"2024-03-26T03:16:56.900984+0000","last_unstale":"2024-03-26T03:19:19.782146+0000","last_undegraded":"2024-03-26T03:19:19.782146+0000","last_fullsized":"2024-03-26T03:19:19.782146+0000","mapping_epoch":403,"log_start":"440'8100","ondisk_log_start":"440'8100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'9185","last_scrub_stamp":"2024-03-26T03:18:52.598640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:18:52.598640+0000","objects_scrubbed":7305,"log_size":3338,"log_dups_size":928,"ondisk_log_size":3338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.644456+0000","scrub_duration":1807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325976064,"num_objects":4974,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":760,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405594+0000","last_change":"2024-03-26T03:16:31.121425+0000","last_active":"2024-03-26T03:19:19.405594+0000","last_peered":"2024-03-26T03:19:19.405594+0000","last_clean":"2024-03-26T03:19:19.405594+0000","last_became_active":"2024-03-26T03:14:40.174462+0000","last_became_peered":"2024-03-26T03:14:40.174462+0000","last_unstale":"2024-03-26T03:19:19.405594+0000","last_undegraded":"2024-03-26T03:19:19.405594+0000","last_fullsized":"2024-03-26T03:19:19.405594+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.121331+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.121331+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"342'10","reported_seq":457,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408921+0000","last_change":"2024-03-26T03:17:48.646060+0000","last_active":"2024-03-26T03:19:19.408921+0000","last_peered":"2024-03-26T03:19:19.408921+0000","last_clean":"2024-03-26T03:19:19.408921+0000","last_became_active":"2024-03-26T03:14:37.273000+0000","last_became_peered":"2024-03-26T03:14:37.273000+0000","last_unstale":"2024-03-26T03:19:19.408921+0000","last_undegraded":"2024-03-26T03:19:19.408921+0000","last_fullsized":"2024-03-26T03:19:19.408921+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'10","last_scrub_stamp":"2024-03-26T03:17:48.645977+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.645977+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"446'51","reported_seq":990,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412445+0000","last_change":"2024-03-26T03:18:51.773314+0000","last_active":"2024-03-26T03:19:19.412445+0000","last_peered":"2024-03-26T03:19:19.412445+0000","last_clean":"2024-03-26T03:19:19.412445+0000","last_became_active":"2024-03-26T03:14:37.267205+0000","last_became_peered":"2024-03-26T03:14:37.267205+0000","last_unstale":"2024-03-26T03:19:19.412445+0000","last_undegraded":"2024-03-26T03:19:19.412445+0000","last_fullsized":"2024-03-26T03:19:19.412445+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:18:51.773171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:51.773171+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:52.240058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":270,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417386+0000","last_change":"2024-03-26T03:17:57.225859+0000","last_active":"2024-03-26T03:19:19.417386+0000","last_peered":"2024-03-26T03:19:19.417386+0000","last_clean":"2024-03-26T03:19:19.417386+0000","last_became_active":"2024-03-26T03:14:37.264771+0000","last_became_peered":"2024-03-26T03:14:37.264771+0000","last_unstale":"2024-03-26T03:19:19.417386+0000","last_undegraded":"2024-03-26T03:19:19.417386+0000","last_fullsized":"2024-03-26T03:19:19.417386+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":674,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407748+0000","last_change":"2024-03-26T03:18:07.660598+0000","last_active":"2024-03-26T03:19:19.407748+0000","last_peered":"2024-03-26T03:19:19.407748+0000","last_clean":"2024-03-26T03:19:19.407748+0000","last_became_active":"2024-03-26T03:13:35.269469+0000","last_became_peered":"2024-03-26T03:13:35.269469+0000","last_unstale":"2024-03-26T03:19:19.407748+0000","last_undegraded":"2024-03-26T03:19:19.407748+0000","last_fullsized":"2024-03-26T03:19:19.407748+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_clean_scrub_stamp":"2024-03-26T03:18:07.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.629198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"452'11415","reported_seq":11585,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.758681+0000","last_change":"2024-03-26T03:19:14.031564+0000","last_active":"2024-03-26T03:19:19.758681+0000","last_peered":"2024-03-26T03:19:19.758681+0000","last_clean":"2024-03-26T03:19:19.758681+0000","last_became_active":"2024-03-26T03:16:56.903390+0000","last_became_peered":"2024-03-26T03:16:56.903390+0000","last_unstale":"2024-03-26T03:19:19.758681+0000","last_undegraded":"2024-03-26T03:19:19.758681+0000","last_fullsized":"2024-03-26T03:19:19.758681+0000","mapping_epoch":403,"log_start":"440'7900","ondisk_log_start":"440'7900","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'10985","last_scrub_stamp":"2024-03-26T03:19:14.031321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:19:14.031321+0000","objects_scrubbed":5362,"log_size":3515,"log_dups_size":0,"ondisk_log_size":3515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:19.550278+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317915136,"num_objects":4851,"num_object_clones":0,"num_object_copies":14553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11415,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":364,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406389+0000","last_change":"2024-03-26T03:19:16.437899+0000","last_active":"2024-03-26T03:19:19.406389+0000","last_peered":"2024-03-26T03:19:19.406389+0000","last_clean":"2024-03-26T03:19:19.406389+0000","last_became_active":"2024-03-26T03:17:59.592113+0000","last_became_peered":"2024-03-26T03:17:59.592113+0000","last_unstale":"2024-03-26T03:19:19.406389+0000","last_undegraded":"2024-03-26T03:19:19.406389+0000","last_fullsized":"2024-03-26T03:19:19.406389+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_clean_scrub_stamp":"2024-03-26T03:19:16.437810+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:44.087975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"446'32","reported_seq":748,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418721+0000","last_change":"2024-03-26T03:17:38.464630+0000","last_active":"2024-03-26T03:19:19.418721+0000","last_peered":"2024-03-26T03:19:19.418721+0000","last_clean":"2024-03-26T03:19:19.418721+0000","last_became_active":"2024-03-26T03:14:22.579044+0000","last_became_peered":"2024-03-26T03:14:22.579044+0000","last_unstale":"2024-03-26T03:19:19.418721+0000","last_undegraded":"2024-03-26T03:19:19.418721+0000","last_fullsized":"2024-03-26T03:19:19.418721+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:17:38.464358+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:17:38.464358+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":849,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.421210+0000","last_change":"2024-03-26T03:17:56.571530+0000","last_active":"2024-03-26T03:19:19.421210+0000","last_peered":"2024-03-26T03:19:19.421210+0000","last_clean":"2024-03-26T03:19:19.421210+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:19:19.421210+0000","last_undegraded":"2024-03-26T03:19:19.421210+0000","last_fullsized":"2024-03-26T03:19:19.421210+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:16.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:15:16.933002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.402491+0000","last_change":"2024-03-26T03:17:56.568649+0000","last_active":"2024-03-26T03:19:19.402491+0000","last_peered":"2024-03-26T03:19:19.402491+0000","last_clean":"2024-03-26T03:19:19.402491+0000","last_became_active":"2024-03-26T03:17:56.568322+0000","last_became_peered":"2024-03-26T03:17:56.568322+0000","last_unstale":"2024-03-26T03:19:19.402491+0000","last_undegraded":"2024-03-26T03:19:19.402491+0000","last_fullsized":"2024-03-26T03:19:19.402491+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.222175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.222175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":163,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.419584+0000","last_change":"2024-03-26T03:16:50.206535+0000","last_active":"2024-03-26T03:19:19.419584+0000","last_peered":"2024-03-26T03:19:19.419584+0000","last_clean":"2024-03-26T03:19:19.419584+0000","last_became_active":"2024-03-26T03:16:50.206224+0000","last_became_peered":"2024-03-26T03:16:50.206224+0000","last_unstale":"2024-03-26T03:19:19.419584+0000","last_undegraded":"2024-03-26T03:19:19.419584+0000","last_fullsized":"2024-03-26T03:19:19.419584+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:26.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:13:26.722546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"452'11531","reported_seq":11718,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.800727+0000","last_change":"2024-03-26T03:18:55.362846+0000","last_active":"2024-03-26T03:19:19.800727+0000","last_peered":"2024-03-26T03:19:19.800727+0000","last_clean":"2024-03-26T03:19:19.800727+0000","last_became_active":"2024-03-26T03:16:56.908452+0000","last_became_peered":"2024-03-26T03:16:56.908452+0000","last_unstale":"2024-03-26T03:19:19.800727+0000","last_undegraded":"2024-03-26T03:19:19.800727+0000","last_fullsized":"2024-03-26T03:19:19.800727+0000","mapping_epoch":403,"log_start":"440'8200","ondisk_log_start":"440'8200","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'9482","last_scrub_stamp":"2024-03-26T03:18:55.362409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:18:55.362409+0000","objects_scrubbed":7039,"log_size":3331,"log_dups_size":986,"ondisk_log_size":3331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:20.391650+0000","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289088,"num_objects":4933,"num_object_clones":0,"num_object_copies":14799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11531,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":935,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408642+0000","last_change":"2024-03-26T03:17:32.537859+0000","last_active":"2024-03-26T03:19:19.408642+0000","last_peered":"2024-03-26T03:19:19.408642+0000","last_clean":"2024-03-26T03:19:19.408642+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:19:19.408642+0000","last_undegraded":"2024-03-26T03:19:19.408642+0000","last_fullsized":"2024-03-26T03:19:19.408642+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:32.537761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.537761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"446'57","reported_seq":1003,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408243+0000","last_change":"2024-03-26T03:18:02.671958+0000","last_active":"2024-03-26T03:19:19.408243+0000","last_peered":"2024-03-26T03:19:19.408243+0000","last_clean":"2024-03-26T03:19:19.408243+0000","last_became_active":"2024-03-26T03:14:37.266473+0000","last_became_peered":"2024-03-26T03:14:37.266473+0000","last_unstale":"2024-03-26T03:19:19.408243+0000","last_undegraded":"2024-03-26T03:19:19.408243+0000","last_fullsized":"2024-03-26T03:19:19.408243+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:18:02.671821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:02.671821+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:20.660317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":709,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415465+0000","last_change":"2024-03-26T03:17:54.517355+0000","last_active":"2024-03-26T03:19:19.415465+0000","last_peered":"2024-03-26T03:19:19.415465+0000","last_clean":"2024-03-26T03:19:19.415465+0000","last_became_active":"2024-03-26T03:16:50.208923+0000","last_became_peered":"2024-03-26T03:16:50.208923+0000","last_unstale":"2024-03-26T03:19:19.415465+0000","last_undegraded":"2024-03-26T03:19:19.415465+0000","last_fullsized":"2024-03-26T03:19:19.415465+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:54.517230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.517230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":880,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.402164+0000","last_change":"2024-03-26T03:18:05.489195+0000","last_active":"2024-03-26T03:19:19.402164+0000","last_peered":"2024-03-26T03:19:19.402164+0000","last_clean":"2024-03-26T03:19:19.402164+0000","last_became_active":"2024-03-26T03:14:37.242217+0000","last_became_peered":"2024-03-26T03:14:37.242217+0000","last_unstale":"2024-03-26T03:19:19.402164+0000","last_undegraded":"2024-03-26T03:19:19.402164+0000","last_fullsized":"2024-03-26T03:19:19.402164+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:05.489069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.227664+0000","last_clean_scrub_stamp":"2024-03-26T03:18:05.489069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:21.866500+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":800,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412445+0000","last_change":"2024-03-26T03:17:40.190354+0000","last_active":"2024-03-26T03:19:19.412445+0000","last_peered":"2024-03-26T03:19:19.412445+0000","last_clean":"2024-03-26T03:19:19.412445+0000","last_became_active":"2024-03-26T03:14:37.270129+0000","last_became_peered":"2024-03-26T03:14:37.270129+0000","last_unstale":"2024-03-26T03:19:19.412445+0000","last_undegraded":"2024-03-26T03:19:19.412445+0000","last_fullsized":"2024-03-26T03:19:19.412445+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.189873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.189873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"452'11637","reported_seq":11746,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.589961+0000","last_change":"2024-03-26T03:16:56.905137+0000","last_active":"2024-03-26T03:19:19.589961+0000","last_peered":"2024-03-26T03:19:19.589961+0000","last_clean":"2024-03-26T03:19:19.589961+0000","last_became_active":"2024-03-26T03:16:56.904798+0000","last_became_peered":"2024-03-26T03:16:56.904798+0000","last_unstale":"2024-03-26T03:19:19.589961+0000","last_undegraded":"2024-03-26T03:19:19.589961+0000","last_fullsized":"2024-03-26T03:19:19.589961+0000","mapping_epoch":403,"log_start":"440'8400","ondisk_log_start":"440'8400","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":2487,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.511632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"446'53","reported_seq":98,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405406+0000","last_change":"2024-03-26T03:17:56.585807+0000","last_active":"2024-03-26T03:19:19.405406+0000","last_peered":"2024-03-26T03:19:19.405406+0000","last_clean":"2024-03-26T03:19:19.405406+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:19:19.405406+0000","last_undegraded":"2024-03-26T03:19:19.405406+0000","last_fullsized":"2024-03-26T03:19:19.405406+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:17:36.443102+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:17:36.443102+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":59,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405244+0000","last_change":"2024-03-26T03:17:56.568076+0000","last_active":"2024-03-26T03:19:19.405244+0000","last_peered":"2024-03-26T03:19:19.405244+0000","last_clean":"2024-03-26T03:19:19.405244+0000","last_became_active":"2024-03-26T03:17:56.567051+0000","last_became_peered":"2024-03-26T03:17:56.567051+0000","last_unstale":"2024-03-26T03:19:19.405244+0000","last_undegraded":"2024-03-26T03:19:19.405244+0000","last_fullsized":"2024-03-26T03:19:19.405244+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.464363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.464363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":850,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417159+0000","last_change":"2024-03-26T03:16:46.947202+0000","last_active":"2024-03-26T03:19:19.417159+0000","last_peered":"2024-03-26T03:19:19.417159+0000","last_clean":"2024-03-26T03:19:19.417159+0000","last_became_active":"2024-03-26T03:13:35.277164+0000","last_became_peered":"2024-03-26T03:13:35.277164+0000","last_unstale":"2024-03-26T03:19:19.417159+0000","last_undegraded":"2024-03-26T03:19:19.417159+0000","last_fullsized":"2024-03-26T03:19:19.417159+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:46.947118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.947118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.052743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":264,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412093+0000","last_change":"2024-03-26T03:18:50.863004+0000","last_active":"2024-03-26T03:19:19.412093+0000","last_peered":"2024-03-26T03:19:19.412093+0000","last_clean":"2024-03-26T03:19:19.412093+0000","last_became_active":"2024-03-26T03:14:21.586859+0000","last_became_peered":"2024-03-26T03:14:21.586859+0000","last_unstale":"2024-03-26T03:19:19.412093+0000","last_undegraded":"2024-03-26T03:19:19.412093+0000","last_fullsized":"2024-03-26T03:19:19.412093+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:50.862872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:50.862872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:59.623780+0000","scrub_duration":310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":558,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412069+0000","last_change":"2024-03-26T03:16:28.866577+0000","last_active":"2024-03-26T03:19:19.412069+0000","last_peered":"2024-03-26T03:19:19.412069+0000","last_clean":"2024-03-26T03:19:19.412069+0000","last_became_active":"2024-03-26T03:14:21.590629+0000","last_became_peered":"2024-03-26T03:14:21.590629+0000","last_unstale":"2024-03-26T03:19:19.412069+0000","last_undegraded":"2024-03-26T03:19:19.412069+0000","last_fullsized":"2024-03-26T03:19:19.412069+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:28.866477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.866477+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"452'11407","reported_seq":11516,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.756649+0000","last_change":"2024-03-26T03:16:56.903756+0000","last_active":"2024-03-26T03:19:19.756649+0000","last_peered":"2024-03-26T03:19:19.756649+0000","last_clean":"2024-03-26T03:19:19.756649+0000","last_became_active":"2024-03-26T03:16:56.903414+0000","last_became_peered":"2024-03-26T03:16:56.903414+0000","last_unstale":"2024-03-26T03:19:19.756649+0000","last_undegraded":"2024-03-26T03:19:19.756649+0000","last_fullsized":"2024-03-26T03:19:19.756649+0000","mapping_epoch":403,"log_start":"440'7900","ondisk_log_start":"440'7900","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3507,"log_dups_size":0,"ondisk_log_size":3507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325517312,"num_objects":4967,"num_object_clones":0,"num_object_copies":14901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11407,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":725,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418392+0000","last_change":"2024-03-26T03:16:30.821771+0000","last_active":"2024-03-26T03:19:19.418392+0000","last_peered":"2024-03-26T03:19:19.418392+0000","last_clean":"2024-03-26T03:19:19.418392+0000","last_became_active":"2024-03-26T03:14:40.169332+0000","last_became_peered":"2024-03-26T03:14:40.169332+0000","last_unstale":"2024-03-26T03:19:19.418392+0000","last_undegraded":"2024-03-26T03:19:19.418392+0000","last_fullsized":"2024-03-26T03:19:19.418392+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.821670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.821670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"446'30","reported_seq":889,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405556+0000","last_change":"2024-03-26T03:16:39.220608+0000","last_active":"2024-03-26T03:19:19.405556+0000","last_peered":"2024-03-26T03:19:19.405556+0000","last_clean":"2024-03-26T03:19:19.405556+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:19:19.405556+0000","last_undegraded":"2024-03-26T03:19:19.405556+0000","last_fullsized":"2024-03-26T03:19:19.405556+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'20","last_scrub_stamp":"2024-03-26T03:16:39.220501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.220501+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":835,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416056+0000","last_change":"2024-03-26T03:15:09.812769+0000","last_active":"2024-03-26T03:19:19.416056+0000","last_peered":"2024-03-26T03:19:19.416056+0000","last_clean":"2024-03-26T03:19:19.416056+0000","last_became_active":"2024-03-26T03:14:37.251633+0000","last_became_peered":"2024-03-26T03:14:37.251633+0000","last_unstale":"2024-03-26T03:19:19.416056+0000","last_undegraded":"2024-03-26T03:19:19.416056+0000","last_fullsized":"2024-03-26T03:19:19.416056+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:09.812652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:15:09.812652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"401'11","reported_seq":976,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412492+0000","last_change":"2024-03-26T03:19:15.476137+0000","last_active":"2024-03-26T03:19:19.412492+0000","last_peered":"2024-03-26T03:19:19.412492+0000","last_clean":"2024-03-26T03:19:19.412492+0000","last_became_active":"2024-03-26T03:16:50.206797+0000","last_became_peered":"2024-03-26T03:16:50.206797+0000","last_unstale":"2024-03-26T03:19:19.412492+0000","last_undegraded":"2024-03-26T03:19:19.412492+0000","last_fullsized":"2024-03-26T03:19:19.412492+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'11","last_scrub_stamp":"2024-03-26T03:19:15.476016+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:19:15.476016+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:32.165218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":59,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406416+0000","last_change":"2024-03-26T03:17:56.574248+0000","last_active":"2024-03-26T03:19:19.406416+0000","last_peered":"2024-03-26T03:19:19.406416+0000","last_clean":"2024-03-26T03:19:19.406416+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:19:19.406416+0000","last_undegraded":"2024-03-26T03:19:19.406416+0000","last_fullsized":"2024-03-26T03:19:19.406416+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:32.853807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.853807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"452'11425","reported_seq":11534,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:20.631516+0000","last_change":"2024-03-26T03:16:56.909076+0000","last_active":"2024-03-26T03:19:20.631516+0000","last_peered":"2024-03-26T03:19:20.631516+0000","last_clean":"2024-03-26T03:19:20.631516+0000","last_became_active":"2024-03-26T03:16:56.908486+0000","last_became_peered":"2024-03-26T03:16:56.908486+0000","last_unstale":"2024-03-26T03:19:20.631516+0000","last_undegraded":"2024-03-26T03:19:20.631516+0000","last_fullsized":"2024-03-26T03:19:20.631516+0000","mapping_epoch":403,"log_start":"440'8100","ondisk_log_start":"440'8100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.059626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11425,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":423,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408359+0000","last_change":"2024-03-26T03:19:17.349314+0000","last_active":"2024-03-26T03:19:19.408359+0000","last_peered":"2024-03-26T03:19:19.408359+0000","last_clean":"2024-03-26T03:19:19.408359+0000","last_became_active":"2024-03-26T03:16:50.205474+0000","last_became_peered":"2024-03-26T03:16:50.205474+0000","last_unstale":"2024-03-26T03:19:19.408359+0000","last_undegraded":"2024-03-26T03:19:19.408359+0000","last_fullsized":"2024-03-26T03:19:19.408359+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:17.349212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:17.349212+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.072644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"446'39","reported_seq":917,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416088+0000","last_change":"2024-03-26T03:16:28.788850+0000","last_active":"2024-03-26T03:19:19.416088+0000","last_peered":"2024-03-26T03:19:19.416088+0000","last_clean":"2024-03-26T03:19:19.416088+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:19:19.416088+0000","last_undegraded":"2024-03-26T03:19:19.416088+0000","last_fullsized":"2024-03-26T03:19:19.416088+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'27","last_scrub_stamp":"2024-03-26T03:16:28.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.788749+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":835,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415525+0000","last_change":"2024-03-26T03:14:39.896972+0000","last_active":"2024-03-26T03:19:19.415525+0000","last_peered":"2024-03-26T03:19:19.415525+0000","last_clean":"2024-03-26T03:19:19.415525+0000","last_became_active":"2024-03-26T03:14:37.249790+0000","last_became_peered":"2024-03-26T03:14:37.249790+0000","last_unstale":"2024-03-26T03:19:19.415525+0000","last_undegraded":"2024-03-26T03:19:19.415525+0000","last_fullsized":"2024-03-26T03:19:19.415525+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:39.896823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:14:39.896823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"428'12","reported_seq":81,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406494+0000","last_change":"2024-03-26T03:17:56.575479+0000","last_active":"2024-03-26T03:19:19.406494+0000","last_peered":"2024-03-26T03:19:19.406494+0000","last_clean":"2024-03-26T03:19:19.406494+0000","last_became_active":"2024-03-26T03:17:56.566007+0000","last_became_peered":"2024-03-26T03:17:56.566007+0000","last_unstale":"2024-03-26T03:19:19.406494+0000","last_undegraded":"2024-03-26T03:19:19.406494+0000","last_fullsized":"2024-03-26T03:19:19.406494+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:13:30.083400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:30.083400+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":658,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.420714+0000","last_change":"2024-03-26T03:19:19.190221+0000","last_active":"2024-03-26T03:19:19.420714+0000","last_peered":"2024-03-26T03:19:19.420714+0000","last_clean":"2024-03-26T03:19:19.420714+0000","last_became_active":"2024-03-26T03:13:35.275396+0000","last_became_peered":"2024-03-26T03:13:35.275396+0000","last_unstale":"2024-03-26T03:19:19.420714+0000","last_undegraded":"2024-03-26T03:19:19.420714+0000","last_fullsized":"2024-03-26T03:19:19.420714+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:19.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:19:19.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.391231+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"452'11567","reported_seq":11723,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.764305+0000","last_change":"2024-03-26T03:19:19.448772+0000","last_active":"2024-03-26T03:19:19.764305+0000","last_peered":"2024-03-26T03:19:19.764305+0000","last_clean":"2024-03-26T03:19:19.764305+0000","last_became_active":"2024-03-26T03:16:56.905606+0000","last_became_peered":"2024-03-26T03:16:56.905606+0000","last_unstale":"2024-03-26T03:19:19.764305+0000","last_undegraded":"2024-03-26T03:19:19.764305+0000","last_fullsized":"2024-03-26T03:19:19.764305+0000","mapping_epoch":403,"log_start":"440'8100","ondisk_log_start":"440'8100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":4399,"log_size":3467,"log_dups_size":0,"ondisk_log_size":3467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.411379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323944448,"num_objects":4943,"num_object_clones":0,"num_object_copies":14829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":975,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418355+0000","last_change":"2024-03-26T03:17:39.489433+0000","last_active":"2024-03-26T03:19:19.418355+0000","last_peered":"2024-03-26T03:19:19.418355+0000","last_clean":"2024-03-26T03:19:19.418355+0000","last_became_active":"2024-03-26T03:13:38.314531+0000","last_became_peered":"2024-03-26T03:13:38.314531+0000","last_unstale":"2024-03-26T03:19:19.418355+0000","last_undegraded":"2024-03-26T03:19:19.418355+0000","last_fullsized":"2024-03-26T03:19:19.418355+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:39.489210+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:17:39.489210+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"446'39","reported_seq":915,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406956+0000","last_change":"2024-03-26T03:17:42.061990+0000","last_active":"2024-03-26T03:19:19.406956+0000","last_peered":"2024-03-26T03:19:19.406956+0000","last_clean":"2024-03-26T03:19:19.406956+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:19:19.406956+0000","last_undegraded":"2024-03-26T03:19:19.406956+0000","last_fullsized":"2024-03-26T03:19:19.406956+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:17:42.061780+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.061780+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":804,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415287+0000","last_change":"2024-03-26T03:14:37.253086+0000","last_active":"2024-03-26T03:19:19.415287+0000","last_peered":"2024-03-26T03:19:19.415287+0000","last_clean":"2024-03-26T03:19:19.415287+0000","last_became_active":"2024-03-26T03:14:37.252745+0000","last_became_peered":"2024-03-26T03:14:37.252745+0000","last_unstale":"2024-03-26T03:19:19.415287+0000","last_undegraded":"2024-03-26T03:19:19.415287+0000","last_fullsized":"2024-03-26T03:19:19.415287+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:31.980008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.980008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":655,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406076+0000","last_change":"2024-03-26T03:16:55.201337+0000","last_active":"2024-03-26T03:19:19.406076+0000","last_peered":"2024-03-26T03:19:19.406076+0000","last_clean":"2024-03-26T03:19:19.406076+0000","last_became_active":"2024-03-26T03:14:37.249741+0000","last_became_peered":"2024-03-26T03:14:37.249741+0000","last_unstale":"2024-03-26T03:19:19.406076+0000","last_undegraded":"2024-03-26T03:19:19.406076+0000","last_fullsized":"2024-03-26T03:19:19.406076+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.201190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.201190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":477,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412993+0000","last_change":"2024-03-26T03:17:44.249680+0000","last_active":"2024-03-26T03:19:19.412993+0000","last_peered":"2024-03-26T03:19:19.412993+0000","last_clean":"2024-03-26T03:19:19.412993+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:19:19.412993+0000","last_undegraded":"2024-03-26T03:19:19.412993+0000","last_fullsized":"2024-03-26T03:19:19.412993+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.249561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.249561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"452'11375","reported_seq":25377,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.734607+0000","last_change":"2024-03-26T03:18:56.612971+0000","last_active":"2024-03-26T03:19:19.734607+0000","last_peered":"2024-03-26T03:19:19.734607+0000","last_clean":"2024-03-26T03:19:19.734607+0000","last_became_active":"2024-03-26T03:18:56.603507+0000","last_became_peered":"2024-03-26T03:18:56.603507+0000","last_unstale":"2024-03-26T03:19:19.734607+0000","last_undegraded":"2024-03-26T03:19:19.734607+0000","last_fullsized":"2024-03-26T03:19:19.734607+0000","mapping_epoch":444,"log_start":"440'7900","ondisk_log_start":"440'7900","created":403,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3475,"log_dups_size":0,"ondisk_log_size":3475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318439424,"num_objects":4859,"num_object_clones":0,"num_object_copies":14577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11375,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6940,"num_bytes_recovered":454819840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":663,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.411965+0000","last_change":"2024-03-26T03:17:55.204987+0000","last_active":"2024-03-26T03:19:19.411965+0000","last_peered":"2024-03-26T03:19:19.411965+0000","last_clean":"2024-03-26T03:19:19.411965+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:19:19.411965+0000","last_undegraded":"2024-03-26T03:19:19.411965+0000","last_fullsized":"2024-03-26T03:19:19.411965+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:17:55.204836+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.204836+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"446'48","reported_seq":929,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.402064+0000","last_change":"2024-03-26T03:16:56.199180+0000","last_active":"2024-03-26T03:19:19.402064+0000","last_peered":"2024-03-26T03:19:19.402064+0000","last_clean":"2024-03-26T03:19:19.402064+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:19:19.402064+0000","last_undegraded":"2024-03-26T03:19:19.402064+0000","last_fullsized":"2024-03-26T03:19:19.402064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:16:56.199057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.199057+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":866,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408872+0000","last_change":"2024-03-26T03:17:41.533036+0000","last_active":"2024-03-26T03:19:19.408872+0000","last_peered":"2024-03-26T03:19:19.408872+0000","last_clean":"2024-03-26T03:19:19.408872+0000","last_became_active":"2024-03-26T03:14:37.276008+0000","last_became_peered":"2024-03-26T03:14:37.276008+0000","last_unstale":"2024-03-26T03:19:19.408872+0000","last_undegraded":"2024-03-26T03:19:19.408872+0000","last_fullsized":"2024-03-26T03:19:19.408872+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.532899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.532899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":242,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.411283+0000","last_change":"2024-03-26T03:16:40.872720+0000","last_active":"2024-03-26T03:19:19.411283+0000","last_peered":"2024-03-26T03:19:19.411283+0000","last_clean":"2024-03-26T03:19:19.411283+0000","last_became_active":"2024-03-26T03:14:37.248955+0000","last_became_peered":"2024-03-26T03:14:37.248955+0000","last_unstale":"2024-03-26T03:19:19.411283+0000","last_undegraded":"2024-03-26T03:19:19.411283+0000","last_fullsized":"2024-03-26T03:19:19.411283+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:40.872619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.872619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":647,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413395+0000","last_change":"2024-03-26T03:16:29.852173+0000","last_active":"2024-03-26T03:19:19.413395+0000","last_peered":"2024-03-26T03:19:19.413395+0000","last_clean":"2024-03-26T03:19:19.413395+0000","last_became_active":"2024-03-26T03:14:40.166583+0000","last_became_peered":"2024-03-26T03:14:40.166583+0000","last_unstale":"2024-03-26T03:19:19.413395+0000","last_undegraded":"2024-03-26T03:19:19.413395+0000","last_fullsized":"2024-03-26T03:19:19.413395+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.852073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.852073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"452'11745","reported_seq":11854,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:20.605456+0000","last_change":"2024-03-26T03:16:56.900055+0000","last_active":"2024-03-26T03:19:20.605456+0000","last_peered":"2024-03-26T03:19:20.605456+0000","last_clean":"2024-03-26T03:19:20.605456+0000","last_became_active":"2024-03-26T03:16:56.899679+0000","last_became_peered":"2024-03-26T03:16:56.899679+0000","last_unstale":"2024-03-26T03:19:20.605456+0000","last_undegraded":"2024-03-26T03:19:20.605456+0000","last_fullsized":"2024-03-26T03:19:20.605456+0000","mapping_epoch":403,"log_start":"440'8500","ondisk_log_start":"440'8500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.062074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327614464,"num_objects":4999,"num_object_clones":0,"num_object_copies":14997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":535808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":825,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.404007+0000","last_change":"2024-03-26T03:16:36.206735+0000","last_active":"2024-03-26T03:19:19.404007+0000","last_peered":"2024-03-26T03:19:19.404007+0000","last_clean":"2024-03-26T03:19:19.404007+0000","last_became_active":"2024-03-26T03:14:37.257004+0000","last_became_peered":"2024-03-26T03:14:37.257004+0000","last_unstale":"2024-03-26T03:19:19.404007+0000","last_undegraded":"2024-03-26T03:19:19.404007+0000","last_fullsized":"2024-03-26T03:19:19.404007+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:36.206634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.206634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"446'66","reported_seq":659,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417531+0000","last_change":"2024-03-26T03:16:38.931105+0000","last_active":"2024-03-26T03:19:19.417531+0000","last_peered":"2024-03-26T03:19:19.417531+0000","last_clean":"2024-03-26T03:19:19.417531+0000","last_became_active":"2024-03-26T03:14:37.278227+0000","last_became_peered":"2024-03-26T03:14:37.278227+0000","last_unstale":"2024-03-26T03:19:19.417531+0000","last_undegraded":"2024-03-26T03:19:19.417531+0000","last_fullsized":"2024-03-26T03:19:19.417531+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'54","last_scrub_stamp":"2024-03-26T03:16:38.931005+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.931005+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":854,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416320+0000","last_change":"2024-03-26T03:13:29.030426+0000","last_active":"2024-03-26T03:19:19.416320+0000","last_peered":"2024-03-26T03:19:19.416320+0000","last_clean":"2024-03-26T03:19:19.416320+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:19:19.416320+0000","last_undegraded":"2024-03-26T03:19:19.416320+0000","last_fullsized":"2024-03-26T03:19:19.416320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:13:29.030298+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.030298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":59,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.402586+0000","last_change":"2024-03-26T03:17:56.567975+0000","last_active":"2024-03-26T03:19:19.402586+0000","last_peered":"2024-03-26T03:19:19.402586+0000","last_clean":"2024-03-26T03:19:19.402586+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:19:19.402586+0000","last_undegraded":"2024-03-26T03:19:19.402586+0000","last_fullsized":"2024-03-26T03:19:19.402586+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.179410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.179410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":959,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405725+0000","last_change":"2024-03-26T03:17:54.095189+0000","last_active":"2024-03-26T03:19:19.405725+0000","last_peered":"2024-03-26T03:19:19.405725+0000","last_clean":"2024-03-26T03:19:19.405725+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:19:19.405725+0000","last_undegraded":"2024-03-26T03:19:19.405725+0000","last_fullsized":"2024-03-26T03:19:19.405725+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:54.095058+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.095058+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"452'11374","reported_seq":25317,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:20.626707+0000","last_change":"2024-03-26T03:19:16.501274+0000","last_active":"2024-03-26T03:19:20.626707+0000","last_peered":"2024-03-26T03:19:20.626707+0000","last_clean":"2024-03-26T03:19:20.626707+0000","last_became_active":"2024-03-26T03:19:16.441598+0000","last_became_peered":"2024-03-26T03:19:16.441598+0000","last_unstale":"2024-03-26T03:19:20.626707+0000","last_undegraded":"2024-03-26T03:19:20.626707+0000","last_fullsized":"2024-03-26T03:19:20.626707+0000","mapping_epoch":450,"log_start":"440'8100","ondisk_log_start":"440'8100","created":403,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.056919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11374,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6905,"num_bytes_recovered":452395008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":858,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408430+0000","last_change":"2024-03-26T03:17:34.573811+0000","last_active":"2024-03-26T03:19:19.408430+0000","last_peered":"2024-03-26T03:19:19.408430+0000","last_clean":"2024-03-26T03:19:19.408430+0000","last_became_active":"2024-03-26T03:13:35.269721+0000","last_became_peered":"2024-03-26T03:13:35.269721+0000","last_unstale":"2024-03-26T03:19:19.408430+0000","last_undegraded":"2024-03-26T03:19:19.408430+0000","last_fullsized":"2024-03-26T03:19:19.408430+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.573719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.573719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"446'45","reported_seq":813,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415256+0000","last_change":"2024-03-26T03:18:58.829566+0000","last_active":"2024-03-26T03:19:19.415256+0000","last_peered":"2024-03-26T03:19:19.415256+0000","last_clean":"2024-03-26T03:19:19.415256+0000","last_became_active":"2024-03-26T03:14:37.258677+0000","last_became_peered":"2024-03-26T03:14:37.258677+0000","last_unstale":"2024-03-26T03:19:19.415256+0000","last_undegraded":"2024-03-26T03:19:19.415256+0000","last_fullsized":"2024-03-26T03:19:19.415256+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:18:58.829458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:58.829458+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:27.144987+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":77,"num_read_kb":62,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":789,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405877+0000","last_change":"2024-03-26T03:14:37.257762+0000","last_active":"2024-03-26T03:19:19.405877+0000","last_peered":"2024-03-26T03:19:19.405877+0000","last_clean":"2024-03-26T03:19:19.405877+0000","last_became_active":"2024-03-26T03:14:37.257422+0000","last_became_peered":"2024-03-26T03:14:37.257422+0000","last_unstale":"2024-03-26T03:19:19.405877+0000","last_undegraded":"2024-03-26T03:19:19.405877+0000","last_fullsized":"2024-03-26T03:19:19.405877+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:35.954172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:35.954172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":843,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415651+0000","last_change":"2024-03-26T03:13:24.055056+0000","last_active":"2024-03-26T03:19:19.415651+0000","last_peered":"2024-03-26T03:19:19.415651+0000","last_clean":"2024-03-26T03:19:19.415651+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:19:19.415651+0000","last_undegraded":"2024-03-26T03:19:19.415651+0000","last_fullsized":"2024-03-26T03:19:19.415651+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:24.054721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:13:24.054721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":936,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408206+0000","last_change":"2024-03-26T03:18:53.735082+0000","last_active":"2024-03-26T03:19:19.408206+0000","last_peered":"2024-03-26T03:19:19.408206+0000","last_clean":"2024-03-26T03:19:19.408206+0000","last_became_active":"2024-03-26T03:14:22.569851+0000","last_became_peered":"2024-03-26T03:14:22.569851+0000","last_unstale":"2024-03-26T03:19:19.408206+0000","last_undegraded":"2024-03-26T03:19:19.408206+0000","last_fullsized":"2024-03-26T03:19:19.408206+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:53.734968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:18:53.734968+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:54.266150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"452'11500","reported_seq":11609,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.593952+0000","last_change":"2024-03-26T03:16:56.907700+0000","last_active":"2024-03-26T03:19:19.593952+0000","last_peered":"2024-03-26T03:19:19.593952+0000","last_clean":"2024-03-26T03:19:19.593952+0000","last_became_active":"2024-03-26T03:16:56.907183+0000","last_became_peered":"2024-03-26T03:16:56.907183+0000","last_unstale":"2024-03-26T03:19:19.593952+0000","last_undegraded":"2024-03-26T03:19:19.593952+0000","last_fullsized":"2024-03-26T03:19:19.593952+0000","mapping_epoch":403,"log_start":"440'8300","ondisk_log_start":"440'8300","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3200,"log_dups_size":2387,"ondisk_log_size":3200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.512055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325582848,"num_objects":4968,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":526976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":881,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.414512+0000","last_change":"2024-03-26T03:19:09.827441+0000","last_active":"2024-03-26T03:19:19.414512+0000","last_peered":"2024-03-26T03:19:19.414512+0000","last_clean":"2024-03-26T03:19:19.414512+0000","last_became_active":"2024-03-26T03:13:35.271824+0000","last_became_peered":"2024-03-26T03:13:35.271824+0000","last_unstale":"2024-03-26T03:19:19.414512+0000","last_undegraded":"2024-03-26T03:19:19.414512+0000","last_fullsized":"2024-03-26T03:19:19.414512+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:09.827318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:09.827318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:15.407835+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"446'42","reported_seq":765,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416606+0000","last_change":"2024-03-26T03:14:37.277935+0000","last_active":"2024-03-26T03:19:19.416606+0000","last_peered":"2024-03-26T03:19:19.416606+0000","last_clean":"2024-03-26T03:19:19.416606+0000","last_became_active":"2024-03-26T03:14:37.246803+0000","last_became_peered":"2024-03-26T03:14:37.246803+0000","last_unstale":"2024-03-26T03:19:19.416606+0000","last_undegraded":"2024-03-26T03:19:19.416606+0000","last_fullsized":"2024-03-26T03:19:19.416606+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:13:15.467975+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.467975+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":778,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415399+0000","last_change":"2024-03-26T03:16:56.019977+0000","last_active":"2024-03-26T03:19:19.415399+0000","last_peered":"2024-03-26T03:19:19.415399+0000","last_clean":"2024-03-26T03:19:19.415399+0000","last_became_active":"2024-03-26T03:14:37.265520+0000","last_became_peered":"2024-03-26T03:14:37.265520+0000","last_unstale":"2024-03-26T03:19:19.415399+0000","last_undegraded":"2024-03-26T03:19:19.415399+0000","last_fullsized":"2024-03-26T03:19:19.415399+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:56.019845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.019845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":59,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.401792+0000","last_change":"2024-03-26T03:17:56.568202+0000","last_active":"2024-03-26T03:19:19.401792+0000","last_peered":"2024-03-26T03:19:19.401792+0000","last_clean":"2024-03-26T03:19:19.401792+0000","last_became_active":"2024-03-26T03:17:56.567590+0000","last_became_peered":"2024-03-26T03:17:56.567590+0000","last_unstale":"2024-03-26T03:19:19.401792+0000","last_undegraded":"2024-03-26T03:19:19.401792+0000","last_fullsized":"2024-03-26T03:19:19.401792+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":825,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417207+0000","last_change":"2024-03-26T03:15:17.827310+0000","last_active":"2024-03-26T03:19:19.417207+0000","last_peered":"2024-03-26T03:19:19.417207+0000","last_clean":"2024-03-26T03:19:19.417207+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:19:19.417207+0000","last_undegraded":"2024-03-26T03:19:19.417207+0000","last_fullsized":"2024-03-26T03:19:19.417207+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:17.827186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:15:17.827186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":965,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418494+0000","last_change":"2024-03-26T03:17:59.603321+0000","last_active":"2024-03-26T03:19:19.418494+0000","last_peered":"2024-03-26T03:19:19.418494+0000","last_clean":"2024-03-26T03:19:19.418494+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:19:19.418494+0000","last_undegraded":"2024-03-26T03:19:19.418494+0000","last_fullsized":"2024-03-26T03:19:19.418494+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:43.863351+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:43.863351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.057624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"452'11491","reported_seq":11600,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.578587+0000","last_change":"2024-03-26T03:16:56.907179+0000","last_active":"2024-03-26T03:19:19.578587+0000","last_peered":"2024-03-26T03:19:19.578587+0000","last_clean":"2024-03-26T03:19:19.578587+0000","last_became_active":"2024-03-26T03:16:56.906797+0000","last_became_peered":"2024-03-26T03:16:56.906797+0000","last_unstale":"2024-03-26T03:19:19.578587+0000","last_undegraded":"2024-03-26T03:19:19.578587+0000","last_fullsized":"2024-03-26T03:19:19.578587+0000","mapping_epoch":403,"log_start":"440'8300","ondisk_log_start":"440'8300","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":2387,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.511929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323813376,"num_objects":4941,"num_object_clones":0,"num_object_copies":14823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":117,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415341+0000","last_change":"2024-03-26T03:16:50.209616+0000","last_active":"2024-03-26T03:19:19.415341+0000","last_peered":"2024-03-26T03:19:19.415341+0000","last_clean":"2024-03-26T03:19:19.415341+0000","last_became_active":"2024-03-26T03:16:50.208883+0000","last_became_peered":"2024-03-26T03:16:50.208883+0000","last_unstale":"2024-03-26T03:19:19.415341+0000","last_undegraded":"2024-03-26T03:19:19.415341+0000","last_fullsized":"2024-03-26T03:19:19.415341+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.231346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.231346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"446'41","reported_seq":1191,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408273+0000","last_change":"2024-03-26T03:19:08.477738+0000","last_active":"2024-03-26T03:19:19.408273+0000","last_peered":"2024-03-26T03:19:19.408273+0000","last_clean":"2024-03-26T03:19:19.408273+0000","last_became_active":"2024-03-26T03:14:37.265178+0000","last_became_peered":"2024-03-26T03:14:37.265178+0000","last_unstale":"2024-03-26T03:19:19.408273+0000","last_undegraded":"2024-03-26T03:19:19.408273+0000","last_fullsized":"2024-03-26T03:19:19.408273+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'41","last_scrub_stamp":"2024-03-26T03:19:08.477608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:08.477608+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:36.652940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":545,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407361+0000","last_change":"2024-03-26T03:19:05.588431+0000","last_active":"2024-03-26T03:19:19.407361+0000","last_peered":"2024-03-26T03:19:19.407361+0000","last_clean":"2024-03-26T03:19:19.407361+0000","last_became_active":"2024-03-26T03:14:40.180729+0000","last_became_peered":"2024-03-26T03:14:40.180729+0000","last_unstale":"2024-03-26T03:19:19.407361+0000","last_undegraded":"2024-03-26T03:19:19.407361+0000","last_fullsized":"2024-03-26T03:19:19.407361+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:05.588322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:05.588322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:22.612472+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":895,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415188+0000","last_change":"2024-03-26T03:17:52.570959+0000","last_active":"2024-03-26T03:19:19.415188+0000","last_peered":"2024-03-26T03:19:19.415188+0000","last_clean":"2024-03-26T03:19:19.415188+0000","last_became_active":"2024-03-26T03:14:37.274568+0000","last_became_peered":"2024-03-26T03:14:37.274568+0000","last_unstale":"2024-03-26T03:19:19.415188+0000","last_undegraded":"2024-03-26T03:19:19.415188+0000","last_fullsized":"2024-03-26T03:19:19.415188+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.570835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.570835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":539,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417132+0000","last_change":"2024-03-26T03:19:03.628055+0000","last_active":"2024-03-26T03:19:19.417132+0000","last_peered":"2024-03-26T03:19:19.417132+0000","last_clean":"2024-03-26T03:19:19.417132+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:19:19.417132+0000","last_undegraded":"2024-03-26T03:19:19.417132+0000","last_fullsized":"2024-03-26T03:19:19.417132+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:19:03.627938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:03.627938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:12.334666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":832,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.414538+0000","last_change":"2024-03-26T03:17:41.822116+0000","last_active":"2024-03-26T03:19:19.414538+0000","last_peered":"2024-03-26T03:19:19.414538+0000","last_clean":"2024-03-26T03:19:19.414538+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:19:19.414538+0000","last_undegraded":"2024-03-26T03:19:19.414538+0000","last_fullsized":"2024-03-26T03:19:19.414538+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.822032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.822032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"452'11688","reported_seq":11797,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.583301+0000","last_change":"2024-03-26T03:16:56.908716+0000","last_active":"2024-03-26T03:19:19.583301+0000","last_peered":"2024-03-26T03:19:19.583301+0000","last_clean":"2024-03-26T03:19:19.583301+0000","last_became_active":"2024-03-26T03:16:56.908005+0000","last_became_peered":"2024-03-26T03:16:56.908005+0000","last_unstale":"2024-03-26T03:19:19.583301+0000","last_undegraded":"2024-03-26T03:19:19.583301+0000","last_fullsized":"2024-03-26T03:19:19.583301+0000","mapping_epoch":403,"log_start":"440'8500","ondisk_log_start":"440'8500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3188,"log_dups_size":2487,"ondisk_log_size":3188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.511842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326631424,"num_objects":4984,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11688,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":758,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.414158+0000","last_change":"2024-03-26T03:18:54.811073+0000","last_active":"2024-03-26T03:19:19.414158+0000","last_peered":"2024-03-26T03:19:19.414158+0000","last_clean":"2024-03-26T03:19:19.414158+0000","last_became_active":"2024-03-26T03:16:50.206619+0000","last_became_peered":"2024-03-26T03:16:50.206619+0000","last_unstale":"2024-03-26T03:19:19.414158+0000","last_undegraded":"2024-03-26T03:19:19.414158+0000","last_fullsized":"2024-03-26T03:19:19.414158+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:54.810939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:18:54.810939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:57.976578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"446'27","reported_seq":939,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416236+0000","last_change":"2024-03-26T03:14:37.294831+0000","last_active":"2024-03-26T03:19:19.416236+0000","last_peered":"2024-03-26T03:19:19.416236+0000","last_clean":"2024-03-26T03:19:19.416236+0000","last_became_active":"2024-03-26T03:14:37.272426+0000","last_became_peered":"2024-03-26T03:14:37.272426+0000","last_unstale":"2024-03-26T03:19:19.416236+0000","last_undegraded":"2024-03-26T03:19:19.416236+0000","last_fullsized":"2024-03-26T03:19:19.416236+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:13:23.013434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:23.013434+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":699,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418567+0000","last_change":"2024-03-26T03:16:50.211076+0000","last_active":"2024-03-26T03:19:19.418567+0000","last_peered":"2024-03-26T03:19:19.418567+0000","last_clean":"2024-03-26T03:19:19.418567+0000","last_became_active":"2024-03-26T03:16:50.210676+0000","last_became_peered":"2024-03-26T03:16:50.210676+0000","last_unstale":"2024-03-26T03:19:19.418567+0000","last_undegraded":"2024-03-26T03:19:19.418567+0000","last_fullsized":"2024-03-26T03:19:19.418567+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:41.390700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:41.390700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2083,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416667+0000","last_change":"2024-03-26T03:13:31.075605+0000","last_active":"2024-03-26T03:19:19.416667+0000","last_peered":"2024-03-26T03:19:19.416667+0000","last_clean":"2024-03-26T03:19:19.416667+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:19:19.416667+0000","last_undegraded":"2024-03-26T03:19:19.416667+0000","last_fullsized":"2024-03-26T03:19:19.416667+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:13:31.074419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.074419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.052207+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1212,"num_read_kb":1207,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":816,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408103+0000","last_change":"2024-03-26T03:18:55.818417+0000","last_active":"2024-03-26T03:19:19.408103+0000","last_peered":"2024-03-26T03:19:19.408103+0000","last_clean":"2024-03-26T03:19:19.408103+0000","last_became_active":"2024-03-26T03:14:37.373684+0000","last_became_peered":"2024-03-26T03:14:37.373684+0000","last_unstale":"2024-03-26T03:19:19.408103+0000","last_undegraded":"2024-03-26T03:19:19.408103+0000","last_fullsized":"2024-03-26T03:19:19.408103+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:55.818297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:55.818297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:21.616424+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":900,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.403738+0000","last_change":"2024-03-26T03:17:59.594483+0000","last_active":"2024-03-26T03:19:19.403738+0000","last_peered":"2024-03-26T03:19:19.403738+0000","last_clean":"2024-03-26T03:19:19.403738+0000","last_became_active":"2024-03-26T03:17:59.594123+0000","last_became_peered":"2024-03-26T03:17:59.594123+0000","last_unstale":"2024-03-26T03:19:19.403738+0000","last_undegraded":"2024-03-26T03:19:19.403738+0000","last_fullsized":"2024-03-26T03:19:19.403738+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:50.701574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.701574+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.b","version":"452'11460","reported_seq":25189,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.576431+0000","last_change":"2024-03-26T03:18:49.658787+0000","last_active":"2024-03-26T03:19:19.576431+0000","last_peered":"2024-03-26T03:19:19.576431+0000","last_clean":"2024-03-26T03:19:19.576431+0000","last_became_active":"2024-03-26T03:18:44.514105+0000","last_became_peered":"2024-03-26T03:18:44.514105+0000","last_unstale":"2024-03-26T03:19:19.576431+0000","last_undegraded":"2024-03-26T03:19:19.576431+0000","last_fullsized":"2024-03-26T03:19:19.576431+0000","mapping_epoch":441,"log_start":"440'8300","ondisk_log_start":"440'8300","created":403,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'8981","last_scrub_stamp":"2024-03-26T03:18:49.658668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.658668+0000","objects_scrubbed":7499,"log_size":3160,"log_dups_size":2387,"ondisk_log_size":3160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:04.967991+0000","scrub_duration":3070,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322174976,"num_objects":4916,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":443023360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":302,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.420933+0000","last_change":"2024-03-26T03:17:56.570342+0000","last_active":"2024-03-26T03:19:19.420933+0000","last_peered":"2024-03-26T03:19:19.420933+0000","last_clean":"2024-03-26T03:19:19.420933+0000","last_became_active":"2024-03-26T03:17:56.570006+0000","last_became_peered":"2024-03-26T03:17:56.570006+0000","last_unstale":"2024-03-26T03:19:19.420933+0000","last_undegraded":"2024-03-26T03:19:19.420933+0000","last_fullsized":"2024-03-26T03:19:19.420933+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:17.442291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.442291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"446'71","reported_seq":1201,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415702+0000","last_change":"2024-03-26T03:15:12.789448+0000","last_active":"2024-03-26T03:19:19.415702+0000","last_peered":"2024-03-26T03:19:19.415702+0000","last_clean":"2024-03-26T03:19:19.415702+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:19:19.415702+0000","last_undegraded":"2024-03-26T03:19:19.415702+0000","last_fullsized":"2024-03-26T03:19:19.415702+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'59","last_scrub_stamp":"2024-03-26T03:15:12.789148+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:15:12.789148+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":653,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415921+0000","last_change":"2024-03-26T03:16:50.962056+0000","last_active":"2024-03-26T03:19:19.415921+0000","last_peered":"2024-03-26T03:19:19.415921+0000","last_clean":"2024-03-26T03:19:19.415921+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:19:19.415921+0000","last_undegraded":"2024-03-26T03:19:19.415921+0000","last_fullsized":"2024-03-26T03:19:19.415921+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:50.961975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.961975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":551,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417013+0000","last_change":"2024-03-26T03:19:12.537772+0000","last_active":"2024-03-26T03:19:19.417013+0000","last_peered":"2024-03-26T03:19:19.417013+0000","last_clean":"2024-03-26T03:19:19.417013+0000","last_became_active":"2024-03-26T03:14:37.274135+0000","last_became_peered":"2024-03-26T03:14:37.274135+0000","last_unstale":"2024-03-26T03:19:19.417013+0000","last_undegraded":"2024-03-26T03:19:19.417013+0000","last_fullsized":"2024-03-26T03:19:19.417013+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_clean_scrub_stamp":"2024-03-26T03:19:12.537519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:40.732976+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":577,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.404722+0000","last_change":"2024-03-26T03:16:30.089444+0000","last_active":"2024-03-26T03:19:19.404722+0000","last_peered":"2024-03-26T03:19:19.404722+0000","last_clean":"2024-03-26T03:19:19.404722+0000","last_became_active":"2024-03-26T03:13:56.449984+0000","last_became_peered":"2024-03-26T03:13:56.449984+0000","last_unstale":"2024-03-26T03:19:19.404722+0000","last_undegraded":"2024-03-26T03:19:19.404722+0000","last_fullsized":"2024-03-26T03:19:19.404722+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.089341+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.089341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.d","version":"452'11603","reported_seq":11710,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.742083+0000","last_change":"2024-03-26T03:16:56.901084+0000","last_active":"2024-03-26T03:19:19.742083+0000","last_peered":"2024-03-26T03:19:19.742083+0000","last_clean":"2024-03-26T03:19:19.742083+0000","last_became_active":"2024-03-26T03:16:56.900700+0000","last_became_peered":"2024-03-26T03:16:56.900700+0000","last_unstale":"2024-03-26T03:19:19.742083+0000","last_undegraded":"2024-03-26T03:19:19.742083+0000","last_fullsized":"2024-03-26T03:19:19.742083+0000","mapping_epoch":403,"log_start":"440'8191","ondisk_log_start":"440'8191","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335675416,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11603,"num_write_kb":535169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"333'12","reported_seq":405,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.411180+0000","last_change":"2024-03-26T03:17:42.576855+0000","last_active":"2024-03-26T03:19:19.411180+0000","last_peered":"2024-03-26T03:19:19.411180+0000","last_clean":"2024-03-26T03:19:19.411180+0000","last_became_active":"2024-03-26T03:14:22.574243+0000","last_became_peered":"2024-03-26T03:14:22.574243+0000","last_unstale":"2024-03-26T03:19:19.411180+0000","last_undegraded":"2024-03-26T03:19:19.411180+0000","last_fullsized":"2024-03-26T03:19:19.411180+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'12","last_scrub_stamp":"2024-03-26T03:17:42.576771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.576771+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"446'42","reported_seq":954,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416274+0000","last_change":"2024-03-26T03:17:56.581987+0000","last_active":"2024-03-26T03:19:19.416274+0000","last_peered":"2024-03-26T03:19:19.416274+0000","last_clean":"2024-03-26T03:19:19.416274+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:19:19.416274+0000","last_undegraded":"2024-03-26T03:19:19.416274+0000","last_fullsized":"2024-03-26T03:19:19.416274+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:16:40.828476+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.828476+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":864,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416663+0000","last_change":"2024-03-26T03:14:37.252086+0000","last_active":"2024-03-26T03:19:19.416663+0000","last_peered":"2024-03-26T03:19:19.416663+0000","last_clean":"2024-03-26T03:19:19.416663+0000","last_became_active":"2024-03-26T03:14:37.251746+0000","last_became_peered":"2024-03-26T03:14:37.251746+0000","last_unstale":"2024-03-26T03:19:19.416663+0000","last_undegraded":"2024-03-26T03:19:19.416663+0000","last_fullsized":"2024-03-26T03:19:19.416663+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:19.981413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.981413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":262,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412164+0000","last_change":"2024-03-26T03:18:49.512742+0000","last_active":"2024-03-26T03:19:19.412164+0000","last_peered":"2024-03-26T03:19:19.412164+0000","last_clean":"2024-03-26T03:19:19.412164+0000","last_became_active":"2024-03-26T03:14:22.566823+0000","last_became_peered":"2024-03-26T03:14:22.566823+0000","last_unstale":"2024-03-26T03:19:19.412164+0000","last_undegraded":"2024-03-26T03:19:19.412164+0000","last_fullsized":"2024-03-26T03:19:19.412164+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:49.512502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.512502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:52.569595+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":937,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408749+0000","last_change":"2024-03-26T03:19:06.555275+0000","last_active":"2024-03-26T03:19:19.408749+0000","last_peered":"2024-03-26T03:19:19.408749+0000","last_clean":"2024-03-26T03:19:19.408749+0000","last_became_active":"2024-03-26T03:14:40.178828+0000","last_became_peered":"2024-03-26T03:14:40.178828+0000","last_unstale":"2024-03-26T03:19:19.408749+0000","last_undegraded":"2024-03-26T03:19:19.408749+0000","last_fullsized":"2024-03-26T03:19:19.408749+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:06.555052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:06.555052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:23.198309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"452'11842","reported_seq":11951,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:20.631695+0000","last_change":"2024-03-26T03:16:56.899075+0000","last_active":"2024-03-26T03:19:20.631695+0000","last_peered":"2024-03-26T03:19:20.631695+0000","last_clean":"2024-03-26T03:19:20.631695+0000","last_became_active":"2024-03-26T03:16:56.898139+0000","last_became_peered":"2024-03-26T03:16:56.898139+0000","last_unstale":"2024-03-26T03:19:20.631695+0000","last_undegraded":"2024-03-26T03:19:20.631695+0000","last_fullsized":"2024-03-26T03:19:20.631695+0000","mapping_epoch":403,"log_start":"440'8600","ondisk_log_start":"440'8600","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3242,"log_dups_size":0,"ondisk_log_size":3242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.057546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329383936,"num_objects":5026,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11842,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":654,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412733+0000","last_change":"2024-03-26T03:17:59.595756+0000","last_active":"2024-03-26T03:19:19.412733+0000","last_peered":"2024-03-26T03:19:19.412733+0000","last_clean":"2024-03-26T03:19:19.412733+0000","last_became_active":"2024-03-26T03:17:59.594586+0000","last_became_peered":"2024-03-26T03:17:59.594586+0000","last_unstale":"2024-03-26T03:19:19.412733+0000","last_undegraded":"2024-03-26T03:19:19.412733+0000","last_fullsized":"2024-03-26T03:19:19.412733+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.474559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.474559+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":701,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415107+0000","last_change":"2024-03-26T03:17:53.556662+0000","last_active":"2024-03-26T03:19:19.415107+0000","last_peered":"2024-03-26T03:19:19.415107+0000","last_clean":"2024-03-26T03:19:19.415107+0000","last_became_active":"2024-03-26T03:16:50.208194+0000","last_became_peered":"2024-03-26T03:16:50.208194+0000","last_unstale":"2024-03-26T03:19:19.415107+0000","last_undegraded":"2024-03-26T03:19:19.415107+0000","last_fullsized":"2024-03-26T03:19:19.415107+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:53.556429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:53.556429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"446'36","reported_seq":919,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.403903+0000","last_change":"2024-03-26T03:14:37.290458+0000","last_active":"2024-03-26T03:19:19.403903+0000","last_peered":"2024-03-26T03:19:19.403903+0000","last_clean":"2024-03-26T03:19:19.403903+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:19:19.403903+0000","last_undegraded":"2024-03-26T03:19:19.403903+0000","last_fullsized":"2024-03-26T03:19:19.403903+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:14:37.290313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:14:37.290313+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":918,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406018+0000","last_change":"2024-03-26T03:17:37.107175+0000","last_active":"2024-03-26T03:19:19.406018+0000","last_peered":"2024-03-26T03:19:19.406018+0000","last_clean":"2024-03-26T03:19:19.406018+0000","last_became_active":"2024-03-26T03:14:37.249021+0000","last_became_peered":"2024-03-26T03:14:37.249021+0000","last_unstale":"2024-03-26T03:19:19.406018+0000","last_undegraded":"2024-03-26T03:19:19.406018+0000","last_fullsized":"2024-03-26T03:19:19.406018+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:17:37.106951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:37.106951+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":523,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412660+0000","last_change":"2024-03-26T03:19:00.815058+0000","last_active":"2024-03-26T03:19:19.412660+0000","last_peered":"2024-03-26T03:19:19.412660+0000","last_clean":"2024-03-26T03:19:19.412660+0000","last_became_active":"2024-03-26T03:14:37.265041+0000","last_became_peered":"2024-03-26T03:14:37.265041+0000","last_unstale":"2024-03-26T03:19:19.412660+0000","last_undegraded":"2024-03-26T03:19:19.412660+0000","last_fullsized":"2024-03-26T03:19:19.412660+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:00.814950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:00.814950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:01.474562+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":976,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418252+0000","last_change":"2024-03-26T03:17:59.607666+0000","last_active":"2024-03-26T03:19:19.418252+0000","last_peered":"2024-03-26T03:19:19.418252+0000","last_clean":"2024-03-26T03:19:19.418252+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:19:19.418252+0000","last_undegraded":"2024-03-26T03:19:19.418252+0000","last_fullsized":"2024-03-26T03:19:19.418252+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:44.879010+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.879010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"446'33","reported_seq":357,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413701+0000","last_change":"2024-03-26T03:19:07.549173+0000","last_active":"2024-03-26T03:19:19.413701+0000","last_peered":"2024-03-26T03:19:19.413701+0000","last_clean":"2024-03-26T03:19:19.413701+0000","last_became_active":"2024-03-26T03:14:37.269153+0000","last_became_peered":"2024-03-26T03:14:37.269153+0000","last_unstale":"2024-03-26T03:19:19.413701+0000","last_undegraded":"2024-03-26T03:19:19.413701+0000","last_fullsized":"2024-03-26T03:19:19.413701+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'33","last_scrub_stamp":"2024-03-26T03:19:07.549012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:07.549012+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:13.033562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":133,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415556+0000","last_change":"2024-03-26T03:17:56.578376+0000","last_active":"2024-03-26T03:19:19.415556+0000","last_peered":"2024-03-26T03:19:19.415556+0000","last_clean":"2024-03-26T03:19:19.415556+0000","last_became_active":"2024-03-26T03:17:56.571085+0000","last_became_peered":"2024-03-26T03:17:56.571085+0000","last_unstale":"2024-03-26T03:19:19.415556+0000","last_undegraded":"2024-03-26T03:19:19.415556+0000","last_fullsized":"2024-03-26T03:19:19.415556+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:16:49.187353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.187353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":543,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412332+0000","last_change":"2024-03-26T03:19:01.660770+0000","last_active":"2024-03-26T03:19:19.412332+0000","last_peered":"2024-03-26T03:19:19.412332+0000","last_clean":"2024-03-26T03:19:19.412332+0000","last_became_active":"2024-03-26T03:14:37.273658+0000","last_became_peered":"2024-03-26T03:14:37.273658+0000","last_unstale":"2024-03-26T03:19:19.412332+0000","last_undegraded":"2024-03-26T03:19:19.412332+0000","last_fullsized":"2024-03-26T03:19:19.412332+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:01.660326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:01.660326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:14.456341+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":745,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405363+0000","last_change":"2024-03-26T03:17:55.099141+0000","last_active":"2024-03-26T03:19:19.405363+0000","last_peered":"2024-03-26T03:19:19.405363+0000","last_clean":"2024-03-26T03:19:19.405363+0000","last_became_active":"2024-03-26T03:16:50.205905+0000","last_became_peered":"2024-03-26T03:16:50.205905+0000","last_unstale":"2024-03-26T03:19:19.405363+0000","last_undegraded":"2024-03-26T03:19:19.405363+0000","last_fullsized":"2024-03-26T03:19:19.405363+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:55.099060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.099060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":371,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.403639+0000","last_change":"2024-03-26T03:17:45.079338+0000","last_active":"2024-03-26T03:19:19.403639+0000","last_peered":"2024-03-26T03:19:19.403639+0000","last_clean":"2024-03-26T03:19:19.403639+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:19:19.403639+0000","last_undegraded":"2024-03-26T03:19:19.403639+0000","last_fullsized":"2024-03-26T03:19:19.403639+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:45.079209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.079209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"446'24","reported_seq":617,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.411906+0000","last_change":"2024-03-26T03:16:40.927377+0000","last_active":"2024-03-26T03:19:19.411906+0000","last_peered":"2024-03-26T03:19:19.411906+0000","last_clean":"2024-03-26T03:19:19.411906+0000","last_became_active":"2024-03-26T03:14:37.269408+0000","last_became_peered":"2024-03-26T03:14:37.269408+0000","last_unstale":"2024-03-26T03:19:19.411906+0000","last_undegraded":"2024-03-26T03:19:19.411906+0000","last_fullsized":"2024-03-26T03:19:19.411906+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'18","last_scrub_stamp":"2024-03-26T03:16:40.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.927260+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":871,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408702+0000","last_change":"2024-03-26T03:17:56.570470+0000","last_active":"2024-03-26T03:19:19.408702+0000","last_peered":"2024-03-26T03:19:19.408702+0000","last_clean":"2024-03-26T03:19:19.408702+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:19:19.408702+0000","last_undegraded":"2024-03-26T03:19:19.408702+0000","last_fullsized":"2024-03-26T03:19:19.408702+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:59.707108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.707108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":843,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413866+0000","last_change":"2024-03-26T03:19:05.801891+0000","last_active":"2024-03-26T03:19:19.413866+0000","last_peered":"2024-03-26T03:19:19.413866+0000","last_clean":"2024-03-26T03:19:19.413866+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:19:19.413866+0000","last_undegraded":"2024-03-26T03:19:19.413866+0000","last_fullsized":"2024-03-26T03:19:19.413866+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:05.801789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:05.801789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:07.979149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":249,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408191+0000","last_change":"2024-03-26T03:19:00.728008+0000","last_active":"2024-03-26T03:19:19.408191+0000","last_peered":"2024-03-26T03:19:19.408191+0000","last_clean":"2024-03-26T03:19:19.408191+0000","last_became_active":"2024-03-26T03:14:37.333430+0000","last_became_peered":"2024-03-26T03:14:37.333430+0000","last_unstale":"2024-03-26T03:19:19.408191+0000","last_undegraded":"2024-03-26T03:19:19.408191+0000","last_fullsized":"2024-03-26T03:19:19.408191+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:00.727659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:00.727659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:09.168697+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":620,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412352+0000","last_change":"2024-03-26T03:19:16.735157+0000","last_active":"2024-03-26T03:19:19.412352+0000","last_peered":"2024-03-26T03:19:19.412352+0000","last_clean":"2024-03-26T03:19:19.412352+0000","last_became_active":"2024-03-26T03:14:24.567937+0000","last_became_peered":"2024-03-26T03:14:24.567937+0000","last_unstale":"2024-03-26T03:19:19.412352+0000","last_undegraded":"2024-03-26T03:19:19.412352+0000","last_fullsized":"2024-03-26T03:19:19.412352+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:16.735031+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:19:16.735031+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:46.708237+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"446'49","reported_seq":640,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412902+0000","last_change":"2024-03-26T03:16:49.904794+0000","last_active":"2024-03-26T03:19:19.412902+0000","last_peered":"2024-03-26T03:19:19.412902+0000","last_clean":"2024-03-26T03:19:19.412902+0000","last_became_active":"2024-03-26T03:14:37.265466+0000","last_became_peered":"2024-03-26T03:14:37.265466+0000","last_unstale":"2024-03-26T03:19:19.412902+0000","last_undegraded":"2024-03-26T03:19:19.412902+0000","last_fullsized":"2024-03-26T03:19:19.412902+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.904637+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":877,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413749+0000","last_change":"2024-03-26T03:19:11.775706+0000","last_active":"2024-03-26T03:19:19.413749+0000","last_peered":"2024-03-26T03:19:19.413749+0000","last_clean":"2024-03-26T03:19:19.413749+0000","last_became_active":"2024-03-26T03:17:56.569537+0000","last_became_peered":"2024-03-26T03:17:56.569537+0000","last_unstale":"2024-03-26T03:19:19.413749+0000","last_undegraded":"2024-03-26T03:19:19.413749+0000","last_fullsized":"2024-03-26T03:19:19.413749+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:11.775591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:19:11.775591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:33.687560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"333'13","reported_seq":1136,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413987+0000","last_change":"2024-03-26T03:18:09.760752+0000","last_active":"2024-03-26T03:19:19.413987+0000","last_peered":"2024-03-26T03:19:19.413987+0000","last_clean":"2024-03-26T03:19:19.413987+0000","last_became_active":"2024-03-26T03:14:22.577229+0000","last_became_peered":"2024-03-26T03:14:22.577229+0000","last_unstale":"2024-03-26T03:19:19.413987+0000","last_undegraded":"2024-03-26T03:19:19.413987+0000","last_fullsized":"2024-03-26T03:19:19.413987+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'13","last_scrub_stamp":"2024-03-26T03:18:09.760609+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:18:09.760609+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:30.051655+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":579,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416916+0000","last_change":"2024-03-26T03:18:59.591864+0000","last_active":"2024-03-26T03:19:19.416916+0000","last_peered":"2024-03-26T03:19:19.416916+0000","last_clean":"2024-03-26T03:19:19.416916+0000","last_became_active":"2024-03-26T03:14:37.273988+0000","last_became_peered":"2024-03-26T03:14:37.273988+0000","last_unstale":"2024-03-26T03:19:19.416916+0000","last_undegraded":"2024-03-26T03:19:19.416916+0000","last_fullsized":"2024-03-26T03:19:19.416916+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:59.591751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:59.591751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:15.424958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":545,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407298+0000","last_change":"2024-03-26T03:18:54.739935+0000","last_active":"2024-03-26T03:19:19.407298+0000","last_peered":"2024-03-26T03:19:19.407298+0000","last_clean":"2024-03-26T03:19:19.407298+0000","last_became_active":"2024-03-26T03:14:40.173243+0000","last_became_peered":"2024-03-26T03:14:40.173243+0000","last_unstale":"2024-03-26T03:19:19.407298+0000","last_undegraded":"2024-03-26T03:19:19.407298+0000","last_fullsized":"2024-03-26T03:19:19.407298+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:54.739688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:18:54.739688+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:23.231065+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"446'27","reported_seq":681,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412333+0000","last_change":"2024-03-26T03:16:59.931429+0000","last_active":"2024-03-26T03:19:19.412333+0000","last_peered":"2024-03-26T03:19:19.412333+0000","last_clean":"2024-03-26T03:19:19.412333+0000","last_became_active":"2024-03-26T03:14:37.333426+0000","last_became_peered":"2024-03-26T03:14:37.333426+0000","last_unstale":"2024-03-26T03:19:19.412333+0000","last_undegraded":"2024-03-26T03:19:19.412333+0000","last_fullsized":"2024-03-26T03:19:19.412333+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'21","last_scrub_stamp":"2024-03-26T03:16:59.931283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.931283+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":605,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405570+0000","last_change":"2024-03-26T03:16:29.133272+0000","last_active":"2024-03-26T03:19:19.405570+0000","last_peered":"2024-03-26T03:19:19.405570+0000","last_clean":"2024-03-26T03:19:19.405570+0000","last_became_active":"2024-03-26T03:14:37.262766+0000","last_became_peered":"2024-03-26T03:14:37.262766+0000","last_unstale":"2024-03-26T03:19:19.405570+0000","last_undegraded":"2024-03-26T03:19:19.405570+0000","last_fullsized":"2024-03-26T03:19:19.405570+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:29.133174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.133174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"342'11","reported_seq":643,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413134+0000","last_change":"2024-03-26T03:17:50.265634+0000","last_active":"2024-03-26T03:19:19.413134+0000","last_peered":"2024-03-26T03:19:19.413134+0000","last_clean":"2024-03-26T03:19:19.413134+0000","last_became_active":"2024-03-26T03:14:37.264759+0000","last_became_peered":"2024-03-26T03:14:37.264759+0000","last_unstale":"2024-03-26T03:19:19.413134+0000","last_undegraded":"2024-03-26T03:19:19.413134+0000","last_fullsized":"2024-03-26T03:19:19.413134+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'11","last_scrub_stamp":"2024-03-26T03:17:50.265404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:50.265404+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":838,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.406163+0000","last_change":"2024-03-26T03:17:48.227168+0000","last_active":"2024-03-26T03:19:19.406163+0000","last_peered":"2024-03-26T03:19:19.406163+0000","last_clean":"2024-03-26T03:19:19.406163+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:19:19.406163+0000","last_undegraded":"2024-03-26T03:19:19.406163+0000","last_fullsized":"2024-03-26T03:19:19.406163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:17:48.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.226947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":949,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.414212+0000","last_change":"2024-03-26T03:17:55.577343+0000","last_active":"2024-03-26T03:19:19.414212+0000","last_peered":"2024-03-26T03:19:19.414212+0000","last_clean":"2024-03-26T03:19:19.414212+0000","last_became_active":"2024-03-26T03:14:40.170537+0000","last_became_peered":"2024-03-26T03:14:40.170537+0000","last_unstale":"2024-03-26T03:19:19.414212+0000","last_undegraded":"2024-03-26T03:19:19.414212+0000","last_fullsized":"2024-03-26T03:19:19.414212+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:55.577263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.577263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:21.650938+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"446'25","reported_seq":885,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416171+0000","last_change":"2024-03-26T03:13:33.092136+0000","last_active":"2024-03-26T03:19:19.416171+0000","last_peered":"2024-03-26T03:19:19.416171+0000","last_clean":"2024-03-26T03:19:19.416171+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:19:19.416171+0000","last_undegraded":"2024-03-26T03:19:19.416171+0000","last_fullsized":"2024-03-26T03:19:19.416171+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:13:33.091975+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:13:33.091975+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":836,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407513+0000","last_change":"2024-03-26T03:16:33.748393+0000","last_active":"2024-03-26T03:19:19.407513+0000","last_peered":"2024-03-26T03:19:19.407513+0000","last_clean":"2024-03-26T03:19:19.407513+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:19:19.407513+0000","last_undegraded":"2024-03-26T03:19:19.407513+0000","last_fullsized":"2024-03-26T03:19:19.407513+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:33.748295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:33.748295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":526,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412476+0000","last_change":"2024-03-26T03:16:44.909663+0000","last_active":"2024-03-26T03:19:19.412476+0000","last_peered":"2024-03-26T03:19:19.412476+0000","last_clean":"2024-03-26T03:19:19.412476+0000","last_became_active":"2024-03-26T03:14:22.573232+0000","last_became_peered":"2024-03-26T03:14:22.573232+0000","last_unstale":"2024-03-26T03:19:19.412476+0000","last_undegraded":"2024-03-26T03:19:19.412476+0000","last_fullsized":"2024-03-26T03:19:19.412476+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.909523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.909523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":299,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415174+0000","last_change":"2024-03-26T03:17:35.405461+0000","last_active":"2024-03-26T03:19:19.415174+0000","last_peered":"2024-03-26T03:19:19.415174+0000","last_clean":"2024-03-26T03:19:19.415174+0000","last_became_active":"2024-03-26T03:13:35.278675+0000","last_became_peered":"2024-03-26T03:13:35.278675+0000","last_unstale":"2024-03-26T03:19:19.415174+0000","last_undegraded":"2024-03-26T03:19:19.415174+0000","last_fullsized":"2024-03-26T03:19:19.415174+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.405139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.405139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":617,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416202+0000","last_change":"2024-03-26T03:14:22.578069+0000","last_active":"2024-03-26T03:19:19.416202+0000","last_peered":"2024-03-26T03:19:19.416202+0000","last_clean":"2024-03-26T03:19:19.416202+0000","last_became_active":"2024-03-26T03:14:22.577614+0000","last_became_peered":"2024-03-26T03:14:22.577614+0000","last_unstale":"2024-03-26T03:19:19.416202+0000","last_undegraded":"2024-03-26T03:19:19.416202+0000","last_fullsized":"2024-03-26T03:19:19.416202+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.940687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.940687+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"446'27","reported_seq":888,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405426+0000","last_change":"2024-03-26T03:16:53.170995+0000","last_active":"2024-03-26T03:19:19.405426+0000","last_peered":"2024-03-26T03:19:19.405426+0000","last_clean":"2024-03-26T03:19:19.405426+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:19:19.405426+0000","last_undegraded":"2024-03-26T03:19:19.405426+0000","last_fullsized":"2024-03-26T03:19:19.405426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:16:53.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:53.170849+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":873,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.421096+0000","last_change":"2024-03-26T03:17:34.417755+0000","last_active":"2024-03-26T03:19:19.421096+0000","last_peered":"2024-03-26T03:19:19.421096+0000","last_clean":"2024-03-26T03:19:19.421096+0000","last_became_active":"2024-03-26T03:13:35.274830+0000","last_became_peered":"2024-03-26T03:13:35.274830+0000","last_unstale":"2024-03-26T03:19:19.421096+0000","last_undegraded":"2024-03-26T03:19:19.421096+0000","last_fullsized":"2024-03-26T03:19:19.421096+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.417610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.417610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":820,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416384+0000","last_change":"2024-03-26T03:15:14.862264+0000","last_active":"2024-03-26T03:19:19.416384+0000","last_peered":"2024-03-26T03:19:19.416384+0000","last_clean":"2024-03-26T03:19:19.416384+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:19:19.416384+0000","last_undegraded":"2024-03-26T03:19:19.416384+0000","last_fullsized":"2024-03-26T03:19:19.416384+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:14.862148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:14.862148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":530,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418665+0000","last_change":"2024-03-26T03:14:22.577571+0000","last_active":"2024-03-26T03:19:19.418665+0000","last_peered":"2024-03-26T03:19:19.418665+0000","last_clean":"2024-03-26T03:19:19.418665+0000","last_became_active":"2024-03-26T03:14:22.577265+0000","last_became_peered":"2024-03-26T03:14:22.577265+0000","last_unstale":"2024-03-26T03:19:19.418665+0000","last_undegraded":"2024-03-26T03:19:19.418665+0000","last_fullsized":"2024-03-26T03:19:19.418665+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:19.903253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:19.903253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":540,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.420311+0000","last_change":"2024-03-26T03:14:40.950126+0000","last_active":"2024-03-26T03:19:19.420311+0000","last_peered":"2024-03-26T03:19:19.420311+0000","last_clean":"2024-03-26T03:19:19.420311+0000","last_became_active":"2024-03-26T03:14:40.166745+0000","last_became_peered":"2024-03-26T03:14:40.166745+0000","last_unstale":"2024-03-26T03:19:19.420311+0000","last_undegraded":"2024-03-26T03:19:19.420311+0000","last_fullsized":"2024-03-26T03:19:19.420311+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:40.950003+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:14:40.950003+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"446'12","reported_seq":553,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412350+0000","last_change":"2024-03-26T03:16:34.843257+0000","last_active":"2024-03-26T03:19:19.412350+0000","last_peered":"2024-03-26T03:19:19.412350+0000","last_clean":"2024-03-26T03:19:19.412350+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:19:19.412350+0000","last_undegraded":"2024-03-26T03:19:19.412350+0000","last_fullsized":"2024-03-26T03:19:19.412350+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'8","last_scrub_stamp":"2024-03-26T03:16:34.843159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:34.843159+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":882,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.414072+0000","last_change":"2024-03-26T03:19:18.786789+0000","last_active":"2024-03-26T03:19:19.414072+0000","last_peered":"2024-03-26T03:19:19.414072+0000","last_clean":"2024-03-26T03:19:19.414072+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:19:19.414072+0000","last_undegraded":"2024-03-26T03:19:19.414072+0000","last_fullsized":"2024-03-26T03:19:19.414072+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:18.786671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:19:18.786671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.252575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":877,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408530+0000","last_change":"2024-03-26T03:17:57.629803+0000","last_active":"2024-03-26T03:19:19.408530+0000","last_peered":"2024-03-26T03:19:19.408530+0000","last_clean":"2024-03-26T03:19:19.408530+0000","last_became_active":"2024-03-26T03:17:56.569576+0000","last_became_peered":"2024-03-26T03:17:56.569576+0000","last_unstale":"2024-03-26T03:19:19.408530+0000","last_undegraded":"2024-03-26T03:19:19.408530+0000","last_fullsized":"2024-03-26T03:19:19.408530+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.629637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.629637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:22.066988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":242,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.404439+0000","last_change":"2024-03-26T03:16:45.148252+0000","last_active":"2024-03-26T03:19:19.404439+0000","last_peered":"2024-03-26T03:19:19.404439+0000","last_clean":"2024-03-26T03:19:19.404439+0000","last_became_active":"2024-03-26T03:14:37.260920+0000","last_became_peered":"2024-03-26T03:14:37.260920+0000","last_unstale":"2024-03-26T03:19:19.404439+0000","last_undegraded":"2024-03-26T03:19:19.404439+0000","last_fullsized":"2024-03-26T03:19:19.404439+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.148129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.148129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":613,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405720+0000","last_change":"2024-03-26T03:17:59.621631+0000","last_active":"2024-03-26T03:19:19.405720+0000","last_peered":"2024-03-26T03:19:19.405720+0000","last_clean":"2024-03-26T03:19:19.405720+0000","last_became_active":"2024-03-26T03:17:59.621344+0000","last_became_peered":"2024-03-26T03:17:59.621344+0000","last_unstale":"2024-03-26T03:19:19.405720+0000","last_undegraded":"2024-03-26T03:19:19.405720+0000","last_fullsized":"2024-03-26T03:19:19.405720+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:45.232258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.232258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"446'39","reported_seq":622,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.418854+0000","last_change":"2024-03-26T03:16:56.991143+0000","last_active":"2024-03-26T03:19:19.418854+0000","last_peered":"2024-03-26T03:19:19.418854+0000","last_clean":"2024-03-26T03:19:19.418854+0000","last_became_active":"2024-03-26T03:14:37.273380+0000","last_became_peered":"2024-03-26T03:14:37.273380+0000","last_unstale":"2024-03-26T03:19:19.418854+0000","last_undegraded":"2024-03-26T03:19:19.418854+0000","last_fullsized":"2024-03-26T03:19:19.418854+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'31","last_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.991031+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":859,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408741+0000","last_change":"2024-03-26T03:17:58.647438+0000","last_active":"2024-03-26T03:19:19.408741+0000","last_peered":"2024-03-26T03:19:19.408741+0000","last_clean":"2024-03-26T03:19:19.408741+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:19:19.408741+0000","last_undegraded":"2024-03-26T03:19:19.408741+0000","last_fullsized":"2024-03-26T03:19:19.408741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:58.647326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:58.647326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":245,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.401797+0000","last_change":"2024-03-26T03:17:32.002736+0000","last_active":"2024-03-26T03:19:19.401797+0000","last_peered":"2024-03-26T03:19:19.401797+0000","last_clean":"2024-03-26T03:19:19.401797+0000","last_became_active":"2024-03-26T03:14:37.249377+0000","last_became_peered":"2024-03-26T03:14:37.249377+0000","last_unstale":"2024-03-26T03:19:19.401797+0000","last_undegraded":"2024-03-26T03:19:19.401797+0000","last_fullsized":"2024-03-26T03:19:19.401797+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.002635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.002635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":524,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.410493+0000","last_change":"2024-03-26T03:19:04.682708+0000","last_active":"2024-03-26T03:19:19.410493+0000","last_peered":"2024-03-26T03:19:19.410493+0000","last_clean":"2024-03-26T03:19:19.410493+0000","last_became_active":"2024-03-26T03:14:22.603198+0000","last_became_peered":"2024-03-26T03:14:22.603198+0000","last_unstale":"2024-03-26T03:19:19.410493+0000","last_undegraded":"2024-03-26T03:19:19.410493+0000","last_fullsized":"2024-03-26T03:19:19.410493+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:04.682583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:04.682583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.009193+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":860,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415040+0000","last_change":"2024-03-26T03:18:52.795890+0000","last_active":"2024-03-26T03:19:19.415040+0000","last_peered":"2024-03-26T03:19:19.415040+0000","last_clean":"2024-03-26T03:19:19.415040+0000","last_became_active":"2024-03-26T03:13:38.309830+0000","last_became_peered":"2024-03-26T03:13:38.309830+0000","last_unstale":"2024-03-26T03:19:19.415040+0000","last_undegraded":"2024-03-26T03:19:19.415040+0000","last_fullsized":"2024-03-26T03:19:19.415040+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:52.795807+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:18:52.795807+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:21.039399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"446'36","reported_seq":744,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413662+0000","last_change":"2024-03-26T03:19:11.558002+0000","last_active":"2024-03-26T03:19:19.413662+0000","last_peered":"2024-03-26T03:19:19.413662+0000","last_clean":"2024-03-26T03:19:19.413662+0000","last_became_active":"2024-03-26T03:14:37.274931+0000","last_became_peered":"2024-03-26T03:14:37.274931+0000","last_unstale":"2024-03-26T03:19:19.413662+0000","last_undegraded":"2024-03-26T03:19:19.413662+0000","last_fullsized":"2024-03-26T03:19:19.413662+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:19:11.557859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:11.557859+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:35.928237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":888,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412591+0000","last_change":"2024-03-26T03:19:04.803767+0000","last_active":"2024-03-26T03:19:19.412591+0000","last_peered":"2024-03-26T03:19:19.412591+0000","last_clean":"2024-03-26T03:19:19.412591+0000","last_became_active":"2024-03-26T03:14:37.266519+0000","last_became_peered":"2024-03-26T03:14:37.266519+0000","last_unstale":"2024-03-26T03:19:19.412591+0000","last_undegraded":"2024-03-26T03:19:19.412591+0000","last_fullsized":"2024-03-26T03:19:19.412591+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:04.803650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:19:04.803650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:13.283931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"428'11","reported_seq":1139,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408903+0000","last_change":"2024-03-26T03:17:56.576319+0000","last_active":"2024-03-26T03:19:19.408903+0000","last_peered":"2024-03-26T03:19:19.408903+0000","last_clean":"2024-03-26T03:19:19.408903+0000","last_became_active":"2024-03-26T03:17:56.569871+0000","last_became_peered":"2024-03-26T03:17:56.569871+0000","last_unstale":"2024-03-26T03:19:19.408903+0000","last_undegraded":"2024-03-26T03:19:19.408903+0000","last_fullsized":"2024-03-26T03:19:19.408903+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'10","last_scrub_stamp":"2024-03-26T03:16:56.648648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.648648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":632,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.410467+0000","last_change":"2024-03-26T03:19:17.489071+0000","last_active":"2024-03-26T03:19:19.410467+0000","last_peered":"2024-03-26T03:19:19.410467+0000","last_clean":"2024-03-26T03:19:19.410467+0000","last_became_active":"2024-03-26T03:16:50.206818+0000","last_became_peered":"2024-03-26T03:16:50.206818+0000","last_unstale":"2024-03-26T03:19:19.410467+0000","last_undegraded":"2024-03-26T03:19:19.410467+0000","last_fullsized":"2024-03-26T03:19:19.410467+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:19:17.488941+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:19:17.488941+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:27.176056+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":838,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.403539+0000","last_change":"2024-03-26T03:17:59.611282+0000","last_active":"2024-03-26T03:19:19.403539+0000","last_peered":"2024-03-26T03:19:19.403539+0000","last_clean":"2024-03-26T03:19:19.403539+0000","last_became_active":"2024-03-26T03:17:59.610887+0000","last_became_peered":"2024-03-26T03:17:59.610887+0000","last_unstale":"2024-03-26T03:19:19.403539+0000","last_undegraded":"2024-03-26T03:19:19.403539+0000","last_fullsized":"2024-03-26T03:19:19.403539+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.803243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.803243+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"446'33","reported_seq":911,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.409024+0000","last_change":"2024-03-26T03:18:49.800384+0000","last_active":"2024-03-26T03:19:19.409024+0000","last_peered":"2024-03-26T03:19:19.409024+0000","last_clean":"2024-03-26T03:19:19.409024+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:19:19.409024+0000","last_undegraded":"2024-03-26T03:19:19.409024+0000","last_fullsized":"2024-03-26T03:19:19.409024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:18:49.800256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.800256+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:06.949716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":497,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.419351+0000","last_change":"2024-03-26T03:17:31.999279+0000","last_active":"2024-03-26T03:19:19.419351+0000","last_peered":"2024-03-26T03:19:19.419351+0000","last_clean":"2024-03-26T03:19:19.419351+0000","last_became_active":"2024-03-26T03:14:22.576816+0000","last_became_peered":"2024-03-26T03:14:22.576816+0000","last_unstale":"2024-03-26T03:19:19.419351+0000","last_undegraded":"2024-03-26T03:19:19.419351+0000","last_fullsized":"2024-03-26T03:19:19.419351+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:17:31.999158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:31.999158+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":237,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.416437+0000","last_change":"2024-03-26T03:14:37.255762+0000","last_active":"2024-03-26T03:19:19.416437+0000","last_peered":"2024-03-26T03:19:19.416437+0000","last_clean":"2024-03-26T03:19:19.416437+0000","last_became_active":"2024-03-26T03:14:37.255400+0000","last_became_peered":"2024-03-26T03:14:37.255400+0000","last_unstale":"2024-03-26T03:19:19.416437+0000","last_undegraded":"2024-03-26T03:19:19.416437+0000","last_fullsized":"2024-03-26T03:19:19.416437+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:25.571608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:13:25.571608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":851,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.420817+0000","last_change":"2024-03-26T03:17:56.570706+0000","last_active":"2024-03-26T03:19:19.420817+0000","last_peered":"2024-03-26T03:19:19.420817+0000","last_clean":"2024-03-26T03:19:19.420817+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:19:19.420817+0000","last_undegraded":"2024-03-26T03:19:19.420817+0000","last_fullsized":"2024-03-26T03:19:19.420817+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.953482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.953482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":742,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.419470+0000","last_change":"2024-03-26T03:14:40.162652+0000","last_active":"2024-03-26T03:19:19.419470+0000","last_peered":"2024-03-26T03:19:19.419470+0000","last_clean":"2024-03-26T03:19:19.419470+0000","last_became_active":"2024-03-26T03:14:40.162258+0000","last_became_peered":"2024-03-26T03:14:40.162258+0000","last_unstale":"2024-03-26T03:19:19.419470+0000","last_undegraded":"2024-03-26T03:19:19.419470+0000","last_fullsized":"2024-03-26T03:19:19.419470+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.546156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.546156+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"446'39","reported_seq":544,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417011+0000","last_change":"2024-03-26T03:16:50.211157+0000","last_active":"2024-03-26T03:19:19.417011+0000","last_peered":"2024-03-26T03:19:19.417011+0000","last_clean":"2024-03-26T03:19:19.417011+0000","last_became_active":"2024-03-26T03:16:50.209508+0000","last_became_peered":"2024-03-26T03:16:50.209508+0000","last_unstale":"2024-03-26T03:19:19.417011+0000","last_undegraded":"2024-03-26T03:19:19.417011+0000","last_fullsized":"2024-03-26T03:19:19.417011+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'27","last_scrub_stamp":"2024-03-26T03:16:36.265181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.265181+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:25.052933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":873,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408344+0000","last_change":"2024-03-26T03:17:47.637363+0000","last_active":"2024-03-26T03:19:19.408344+0000","last_peered":"2024-03-26T03:19:19.408344+0000","last_clean":"2024-03-26T03:19:19.408344+0000","last_became_active":"2024-03-26T03:14:37.249066+0000","last_became_peered":"2024-03-26T03:14:37.249066+0000","last_unstale":"2024-03-26T03:19:19.408344+0000","last_undegraded":"2024-03-26T03:19:19.408344+0000","last_fullsized":"2024-03-26T03:19:19.408344+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:47.637246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:47.637246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":138,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407736+0000","last_change":"2024-03-26T03:17:56.573995+0000","last_active":"2024-03-26T03:19:19.407736+0000","last_peered":"2024-03-26T03:19:19.407736+0000","last_clean":"2024-03-26T03:19:19.407736+0000","last_became_active":"2024-03-26T03:17:56.571840+0000","last_became_peered":"2024-03-26T03:17:56.571840+0000","last_unstale":"2024-03-26T03:19:19.407736+0000","last_undegraded":"2024-03-26T03:19:19.407736+0000","last_fullsized":"2024-03-26T03:19:19.407736+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.507151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.507151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":839,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407347+0000","last_change":"2024-03-26T03:13:29.068724+0000","last_active":"2024-03-26T03:19:19.407347+0000","last_peered":"2024-03-26T03:19:19.407347+0000","last_clean":"2024-03-26T03:19:19.407347+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:19:19.407347+0000","last_undegraded":"2024-03-26T03:19:19.407347+0000","last_fullsized":"2024-03-26T03:19:19.407347+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.068502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.068502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":645,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412703+0000","last_change":"2024-03-26T03:18:48.870805+0000","last_active":"2024-03-26T03:19:19.412703+0000","last_peered":"2024-03-26T03:19:19.412703+0000","last_clean":"2024-03-26T03:19:19.412703+0000","last_became_active":"2024-03-26T03:14:40.162583+0000","last_became_peered":"2024-03-26T03:14:40.162583+0000","last_unstale":"2024-03-26T03:19:19.412703+0000","last_undegraded":"2024-03-26T03:19:19.412703+0000","last_fullsized":"2024-03-26T03:19:19.412703+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:18:48.870709+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:18:48.870709+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:01.732773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"446'36","reported_seq":597,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412644+0000","last_change":"2024-03-26T03:16:48.957169+0000","last_active":"2024-03-26T03:19:19.412644+0000","last_peered":"2024-03-26T03:19:19.412644+0000","last_clean":"2024-03-26T03:19:19.412644+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:19:19.412644+0000","last_undegraded":"2024-03-26T03:19:19.412644+0000","last_fullsized":"2024-03-26T03:19:19.412644+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'30","last_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.957049+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.511804+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":555,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412606+0000","last_change":"2024-03-26T03:16:39.889851+0000","last_active":"2024-03-26T03:19:19.412606+0000","last_peered":"2024-03-26T03:19:19.412606+0000","last_clean":"2024-03-26T03:19:19.412606+0000","last_became_active":"2024-03-26T03:14:37.275278+0000","last_became_peered":"2024-03-26T03:14:37.275278+0000","last_unstale":"2024-03-26T03:19:19.412606+0000","last_undegraded":"2024-03-26T03:19:19.412606+0000","last_fullsized":"2024-03-26T03:19:19.412606+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:39.889770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.889770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:24.510732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":818,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405461+0000","last_change":"2024-03-26T03:14:36.271942+0000","last_active":"2024-03-26T03:19:19.405461+0000","last_peered":"2024-03-26T03:19:19.405461+0000","last_clean":"2024-03-26T03:19:19.405461+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:19:19.405461+0000","last_undegraded":"2024-03-26T03:19:19.405461+0000","last_fullsized":"2024-03-26T03:19:19.405461+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.271810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.271810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":518,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412186+0000","last_change":"2024-03-26T03:18:57.829715+0000","last_active":"2024-03-26T03:19:19.412186+0000","last_peered":"2024-03-26T03:19:19.412186+0000","last_clean":"2024-03-26T03:19:19.412186+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:19:19.412186+0000","last_undegraded":"2024-03-26T03:19:19.412186+0000","last_fullsized":"2024-03-26T03:19:19.412186+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:57.829604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:57.829604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:15.026242+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"446'39","reported_seq":90,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.403436+0000","last_change":"2024-03-26T03:17:57.160567+0000","last_active":"2024-03-26T03:19:19.403436+0000","last_peered":"2024-03-26T03:19:19.403436+0000","last_clean":"2024-03-26T03:19:19.403436+0000","last_became_active":"2024-03-26T03:17:56.567557+0000","last_became_peered":"2024-03-26T03:17:56.567557+0000","last_unstale":"2024-03-26T03:19:19.403436+0000","last_undegraded":"2024-03-26T03:19:19.403436+0000","last_fullsized":"2024-03-26T03:19:19.403436+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'33","last_scrub_stamp":"2024-03-26T03:17:57.160442+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.160442+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":248,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407665+0000","last_change":"2024-03-26T03:19:14.394760+0000","last_active":"2024-03-26T03:19:19.407665+0000","last_peered":"2024-03-26T03:19:19.407665+0000","last_clean":"2024-03-26T03:19:19.407665+0000","last_became_active":"2024-03-26T03:14:37.243678+0000","last_became_peered":"2024-03-26T03:14:37.243678+0000","last_unstale":"2024-03-26T03:19:19.407665+0000","last_undegraded":"2024-03-26T03:19:19.407665+0000","last_fullsized":"2024-03-26T03:19:19.407665+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:14.394668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:19:14.394668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:33.267097+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":890,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415158+0000","last_change":"2024-03-26T03:18:01.638296+0000","last_active":"2024-03-26T03:19:19.415158+0000","last_peered":"2024-03-26T03:19:19.415158+0000","last_clean":"2024-03-26T03:19:19.415158+0000","last_became_active":"2024-03-26T03:17:56.570631+0000","last_became_peered":"2024-03-26T03:17:56.570631+0000","last_unstale":"2024-03-26T03:19:19.415158+0000","last_undegraded":"2024-03-26T03:19:19.415158+0000","last_fullsized":"2024-03-26T03:19:19.415158+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:01.638157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:01.638157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:28.331978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":309,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415061+0000","last_change":"2024-03-26T03:17:56.571755+0000","last_active":"2024-03-26T03:19:19.415061+0000","last_peered":"2024-03-26T03:19:19.415061+0000","last_clean":"2024-03-26T03:19:19.415061+0000","last_became_active":"2024-03-26T03:17:56.571178+0000","last_became_peered":"2024-03-26T03:17:56.571178+0000","last_unstale":"2024-03-26T03:19:19.415061+0000","last_undegraded":"2024-03-26T03:19:19.415061+0000","last_fullsized":"2024-03-26T03:19:19.415061+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.660527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.660527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":584,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.413439+0000","last_change":"2024-03-26T03:17:59.595137+0000","last_active":"2024-03-26T03:19:19.413439+0000","last_peered":"2024-03-26T03:19:19.413439+0000","last_clean":"2024-03-26T03:19:19.413439+0000","last_became_active":"2024-03-26T03:17:59.594759+0000","last_became_peered":"2024-03-26T03:17:59.594759+0000","last_unstale":"2024-03-26T03:19:19.413439+0000","last_undegraded":"2024-03-26T03:19:19.413439+0000","last_fullsized":"2024-03-26T03:19:19.413439+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:32.825864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.825864+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"446'15","reported_seq":70,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.404550+0000","last_change":"2024-03-26T03:17:56.580380+0000","last_active":"2024-03-26T03:19:19.404550+0000","last_peered":"2024-03-26T03:19:19.404550+0000","last_clean":"2024-03-26T03:19:19.404550+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:19:19.404550+0000","last_undegraded":"2024-03-26T03:19:19.404550+0000","last_fullsized":"2024-03-26T03:19:19.404550+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'13","last_scrub_stamp":"2024-03-26T03:17:45.281375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.281375+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":845,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.408022+0000","last_change":"2024-03-26T03:16:38.824092+0000","last_active":"2024-03-26T03:19:19.408022+0000","last_peered":"2024-03-26T03:19:19.408022+0000","last_clean":"2024-03-26T03:19:19.408022+0000","last_became_active":"2024-03-26T03:14:37.373600+0000","last_became_peered":"2024-03-26T03:14:37.373600+0000","last_unstale":"2024-03-26T03:19:19.408022+0000","last_undegraded":"2024-03-26T03:19:19.408022+0000","last_fullsized":"2024-03-26T03:19:19.408022+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:38.823985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.823985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":241,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.404645+0000","last_change":"2024-03-26T03:16:51.182163+0000","last_active":"2024-03-26T03:19:19.404645+0000","last_peered":"2024-03-26T03:19:19.404645+0000","last_clean":"2024-03-26T03:19:19.404645+0000","last_became_active":"2024-03-26T03:14:37.253707+0000","last_became_peered":"2024-03-26T03:14:37.253707+0000","last_unstale":"2024-03-26T03:19:19.404645+0000","last_undegraded":"2024-03-26T03:19:19.404645+0000","last_fullsized":"2024-03-26T03:19:19.404645+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.182029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":839,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.412538+0000","last_change":"2024-03-26T03:19:13.804053+0000","last_active":"2024-03-26T03:19:19.412538+0000","last_peered":"2024-03-26T03:19:19.412538+0000","last_clean":"2024-03-26T03:19:19.412538+0000","last_became_active":"2024-03-26T03:14:21.596806+0000","last_became_peered":"2024-03-26T03:14:21.596806+0000","last_unstale":"2024-03-26T03:19:19.412538+0000","last_undegraded":"2024-03-26T03:19:19.412538+0000","last_fullsized":"2024-03-26T03:19:19.412538+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:13.803940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:13.803940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:36.864830+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":644,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.420600+0000","last_change":"2024-03-26T03:17:59.610214+0000","last_active":"2024-03-26T03:19:19.420600+0000","last_peered":"2024-03-26T03:19:19.420600+0000","last_clean":"2024-03-26T03:19:19.420600+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:19:19.420600+0000","last_undegraded":"2024-03-26T03:19:19.420600+0000","last_fullsized":"2024-03-26T03:19:19.420600+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:51.516127+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:17:51.516127+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"446'27","reported_seq":891,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.401654+0000","last_change":"2024-03-26T03:16:46.178208+0000","last_active":"2024-03-26T03:19:19.401654+0000","last_peered":"2024-03-26T03:19:19.401654+0000","last_clean":"2024-03-26T03:19:19.401654+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:19:19.401654+0000","last_undegraded":"2024-03-26T03:19:19.401654+0000","last_fullsized":"2024-03-26T03:19:19.401654+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:16:46.178099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.178099+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":527,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.417178+0000","last_change":"2024-03-26T03:19:06.601810+0000","last_active":"2024-03-26T03:19:19.417178+0000","last_peered":"2024-03-26T03:19:19.417178+0000","last_clean":"2024-03-26T03:19:19.417178+0000","last_became_active":"2024-03-26T03:14:22.571857+0000","last_became_peered":"2024-03-26T03:14:22.571857+0000","last_unstale":"2024-03-26T03:19:19.417178+0000","last_undegraded":"2024-03-26T03:19:19.417178+0000","last_fullsized":"2024-03-26T03:19:19.417178+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:06.601677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:06.601677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:30.452453+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":849,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.415644+0000","last_change":"2024-03-26T03:15:20.809430+0000","last_active":"2024-03-26T03:19:19.415644+0000","last_peered":"2024-03-26T03:19:19.415644+0000","last_clean":"2024-03-26T03:19:19.415644+0000","last_became_active":"2024-03-26T03:14:37.242540+0000","last_became_peered":"2024-03-26T03:14:37.242540+0000","last_unstale":"2024-03-26T03:19:19.415644+0000","last_undegraded":"2024-03-26T03:19:19.415644+0000","last_fullsized":"2024-03-26T03:19:19.415644+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:20.809176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:20.809176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":845,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.404492+0000","last_change":"2024-03-26T03:17:43.076516+0000","last_active":"2024-03-26T03:19:19.404492+0000","last_peered":"2024-03-26T03:19:19.404492+0000","last_clean":"2024-03-26T03:19:19.404492+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:19:19.404492+0000","last_undegraded":"2024-03-26T03:19:19.404492+0000","last_fullsized":"2024-03-26T03:19:19.404492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.076334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.076334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":548,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.407553+0000","last_change":"2024-03-26T03:19:01.651338+0000","last_active":"2024-03-26T03:19:19.407553+0000","last_peered":"2024-03-26T03:19:19.407553+0000","last_clean":"2024-03-26T03:19:19.407553+0000","last_became_active":"2024-03-26T03:14:40.174111+0000","last_became_peered":"2024-03-26T03:14:40.174111+0000","last_unstale":"2024-03-26T03:19:19.407553+0000","last_undegraded":"2024-03-26T03:19:19.407553+0000","last_fullsized":"2024-03-26T03:19:19.407553+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:01.651232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:01.651232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.733163+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"446'64","reported_seq":634,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.419257+0000","last_change":"2024-03-26T03:19:09.592737+0000","last_active":"2024-03-26T03:19:19.419257+0000","last_peered":"2024-03-26T03:19:19.419257+0000","last_clean":"2024-03-26T03:19:19.419257+0000","last_became_active":"2024-03-26T03:14:22.576815+0000","last_became_peered":"2024-03-26T03:14:22.576815+0000","last_unstale":"2024-03-26T03:19:19.419257+0000","last_undegraded":"2024-03-26T03:19:19.419257+0000","last_fullsized":"2024-03-26T03:19:19.419257+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'64","last_scrub_stamp":"2024-03-26T03:19:09.592406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:09.592406+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:20.478394+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":245,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.405903+0000","last_change":"2024-03-26T03:16:48.135457+0000","last_active":"2024-03-26T03:19:19.405903+0000","last_peered":"2024-03-26T03:19:19.405903+0000","last_clean":"2024-03-26T03:19:19.405903+0000","last_became_active":"2024-03-26T03:14:37.248588+0000","last_became_peered":"2024-03-26T03:14:37.248588+0000","last_unstale":"2024-03-26T03:19:19.405903+0000","last_undegraded":"2024-03-26T03:19:19.405903+0000","last_fullsized":"2024-03-26T03:19:19.405903+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:48.135354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.135354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":59,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.401992+0000","last_change":"2024-03-26T03:17:56.572431+0000","last_active":"2024-03-26T03:19:19.401992+0000","last_peered":"2024-03-26T03:19:19.401992+0000","last_clean":"2024-03-26T03:19:19.401992+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:19:19.401992+0000","last_undegraded":"2024-03-26T03:19:19.401992+0000","last_fullsized":"2024-03-26T03:19:19.401992+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.615644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.615644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":519,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-26T03:19:19.410291+0000","last_change":"2024-03-26T03:18:58.798963+0000","last_active":"2024-03-26T03:19:19.410291+0000","last_peered":"2024-03-26T03:19:19.410291+0000","last_clean":"2024-03-26T03:19:19.410291+0000","last_became_active":"2024-03-26T03:14:37.265472+0000","last_became_peered":"2024-03-26T03:14:37.265472+0000","last_unstale":"2024-03-26T03:19:19.410291+0000","last_undegraded":"2024-03-26T03:19:19.410291+0000","last_fullsized":"2024-03-26T03:19:19.410291+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:58.798827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:58.798827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.573112+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5196546072,"num_objects":79294,"num_object_clones":0,"num_object_copies":237882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79294,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184498,"num_write_kb":8441281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20605,"num_bytes_recovered":1350238208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15963402240,"data_stored":15963390024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53107,"ondisk_log_size":53107,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1212,"num_read_kb":1207,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1828,"num_read_kb":1553,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":3014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1244,"ondisk_log_size":1244,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430461,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4092308,"kb_used_data":3276748,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":89652844,"statfs":{"total":95995035648,"available":91804512256,"internally_reserved":0,"allocated":3355389952,"data_stored":3354963948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":835123654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575126,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4498760,"kb_used_data":3478968,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":89246392,"statfs":{"total":95995035648,"available":91388305408,"internally_reserved":0,"allocated":3562463232,"data_stored":3562057287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1044117957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411600,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3136556,"kb_used_data":2261084,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":90608596,"statfs":{"total":95995035648,"available":92783202304,"internally_reserved":0,"allocated":2315350016,"data_stored":2314972050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175037,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3298880,"kb_used_data":2862960,"kb_used_omap":4,"kb_used_meta":435771,"kb_avail":90446272,"statfs":{"total":95995035648,"available":92616982528,"internally_reserved":0,"allocated":2931671040,"data_stored":2931271766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":446230179},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986439,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3142812,"kb_used_data":2259220,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90602340,"statfs":{"total":95995035648,"available":92776796160,"internally_reserved":0,"allocated":2313441280,"data_stored":2313007884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692099,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2609444,"kb_used_data":1613340,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":91135708,"statfs":{"total":95995035648,"available":93322964992,"internally_reserved":0,"allocated":1652060160,"data_stored":1651618700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3518627840,"data_stored":3518627840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2276069376,"data_stored":2276065304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2924806144,"data_stored":2924806144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2295795712,"data_stored":2295791640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1622605824,"data_stored":1622605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3325497344,"data_stored":3325493272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:19:29.283 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.331 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:29.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:29.432 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:29.473 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:29.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:29.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:29.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:29.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:29.787 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:29 smithi066 ceph-mon[29666]: pgmap v971: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 339 B/s rd, 0 B/s wr, 1.50k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:29.787 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:29 smithi066 ceph-mon[37920]: pgmap v971: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 339 B/s rd, 0 B/s wr, 1.50k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:29.843 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:29.886 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:29.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:29.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:29 smithi192 ceph-mon[33110]: pgmap v971: 178 pgs: 178 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 339 B/s rd, 0 B/s wr, 1.50k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:29.989 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:30.037 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:30.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:30.167 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:19:30.185 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:30.225 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:19:30.268 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:19:30.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:30.311 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:19:30.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:30.352 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:19:30.390 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:19:30.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:30.431 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:19:30.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:30.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:19:30.472 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:19:30.477 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:19:30.477 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:19:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:19:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:19:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:19:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:19:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:19:30.478 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:19:30.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:19:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:19:30.550 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:30.602 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:30.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:30.837 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:30.844 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:30 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2217963631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:30.844 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:30 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2217963631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:30.885 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:30.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:30.987 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:30.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:30 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2217963631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:31.039 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:31.141 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:31.185 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:31.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:31.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:31.419 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:31.465 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:31.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:31.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:31.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:31.799 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:31.830 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:31.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:31.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:31 smithi066 ceph-mon[29666]: pgmap v972: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:31.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:31.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:31 smithi066 ceph-mon[37920]: pgmap v972: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:31.889 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:31.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:31.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:31 smithi192 ceph-mon[33110]: pgmap v972: 178 pgs: 178 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:19:31.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:32.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:32.127 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:32.175 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:32.248 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:19:32.277 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.348 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:32.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:32.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:32.498 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:32.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:32.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:32.729 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:32.771 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:32.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:32.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:32.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:33.004 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:33.060 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:33.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:33.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:33.243 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:19:33.296 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.334 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:33.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:33.435 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.480 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:33.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:33.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:33.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:33.718 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.766 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:33.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:33.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:33 smithi066 ceph-mon[29666]: pgmap v973: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T03:19:33.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:33 smithi066 ceph-mon[37920]: pgmap v973: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T03:19:33.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:33.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:33.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:33.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:33 smithi192 ceph-mon[33110]: pgmap v973: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T03:19:34.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:34.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:19:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:19:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:19:34.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:34.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:34.268 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.303 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:34.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:34.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:34.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:34.539 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:34.586 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:34.688 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:34.738 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:34.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:34.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:34.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:34.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:35.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:35.106 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:35.152 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:35.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:35.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:35.387 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.433 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:35.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:35.534 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:35.565 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:35 smithi066 ceph-mon[29666]: pgmap v974: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:19:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:35 smithi066 ceph-mon[37920]: pgmap v974: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:19:35.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:35.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:35.793 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:35.836 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:35.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:35.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:35.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:35.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:35 smithi192 ceph-mon[33110]: pgmap v974: 178 pgs: 178 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:19:36.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:36.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:36.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:36.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:36.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:36.385 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:36.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:36.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:36.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:36.671 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:36.773 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.825 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:36.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:36.926 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:36.970 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:36.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:37.071 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:37.084 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:19:37.085 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:19:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.118 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:37.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:37.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:37.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:37.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:37.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:37.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:37.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:37.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:37.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:37.699 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:37.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:37.806 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:37 smithi066 ceph-mon[29666]: pgmap v975: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-03-26T03:19:37.807 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:37 smithi066 ceph-mon[37920]: pgmap v975: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-03-26T03:19:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.807 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:37.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:37.908 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:37.955 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:37.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:37 smithi192 ceph-mon[33110]: pgmap v975: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-03-26T03:19:38.056 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.099 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:38.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:38.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:38.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:38.331 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:19:38.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:38.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:38.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:38.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:38.601 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:38.655 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:38.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.802 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:38.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:38.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:38.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:38.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:39.036 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:39.081 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:39.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:39.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:39.313 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.355 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:39.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:39.456 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:39.499 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:39.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:39.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:39 smithi066 ceph-mon[29666]: pgmap v976: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:39 smithi066 ceph-mon[37920]: pgmap v976: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:19:39.612 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:39.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:39.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:39.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:39.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:39.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:39.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:39.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:39 smithi192 ceph-mon[33110]: pgmap v976: 178 pgs: 178 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:19:40.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.061 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:40.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:40.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.211 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:40.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:40.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:40.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:40.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:40.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:40.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:19:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:19:40.572 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:40.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:40.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:40.619 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:40.721 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.768 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:40.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:40.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[29666]: osdmap e454: 8 total, 8 up, 6 in 2024-03-26T03:19:40.924 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:40.925 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:40.926 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:40.926 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:40.926 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[37920]: osdmap e454: 8 total, 8 up, 6 in 2024-03-26T03:19:40.926 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:40.983 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:40.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:40.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:40 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:40.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:40 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:40.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:40 smithi192 ceph-mon[33110]: osdmap e454: 8 total, 8 up, 6 in 2024-03-26T03:19:40.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:41.020 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:41.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:41.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:41.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:41.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:41.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:41.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:41.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:41.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:41.625 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is set 2024-03-26T03:19:41.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:41.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:41.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:41.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:41.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[29666]: pgmap v978: 178 pgs: 178 active+clean; 3.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2339520251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[29666]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[37920]: pgmap v978: 178 pgs: 178 active+clean; 3.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2339520251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:41.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:41 smithi066 ceph-mon[37920]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:41.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:41.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:41 smithi192 ceph-mon[33110]: pgmap v978: 178 pgs: 178 active+clean; 3.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-26T03:19:41.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2339520251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:41.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:41 smithi192 ceph-mon[33110]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:19:41.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:41.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:41.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:42.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:42.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:42.164 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:42.206 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:42.308 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:42.349 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:42.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:42.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:42.588 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:42.631 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:42.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:42.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:42.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:42 smithi066 ceph-mon[29666]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:42.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:42 smithi066 ceph-mon[29666]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:19:42.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:42 smithi066 ceph-mon[29666]: osdmap e455: 8 total, 8 up, 6 in 2024-03-26T03:19:42.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:42 smithi066 ceph-mon[37920]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:42.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:42 smithi066 ceph-mon[37920]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:19:42.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:42 smithi066 ceph-mon[37920]: osdmap e455: 8 total, 8 up, 6 in 2024-03-26T03:19:42.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:42.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:42.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:42 smithi192 ceph-mon[33110]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:42 smithi192 ceph-mon[33110]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:19:42.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:42 smithi192 ceph-mon[33110]: osdmap e455: 8 total, 8 up, 6 in 2024-03-26T03:19:43.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:43.045 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:43.147 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:43.195 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:43.297 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:43.339 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:43.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:43.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:43.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.623 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:43.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:43.724 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:43.770 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:43.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:43 smithi066 ceph-mon[29666]: pgmap v980: 178 pgs: 178 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s 2024-03-26T03:19:43.774 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:43 smithi066 ceph-mon[37920]: pgmap v980: 178 pgs: 178 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s 2024-03-26T03:19:43.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:43.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:43.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:43.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:43 smithi192 ceph-mon[33110]: pgmap v980: 178 pgs: 178 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.90k op/s 2024-03-26T03:19:44.002 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:44.050 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:44.094 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:19:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:19:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:19:44.153 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.204 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:44.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:44.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:44.349 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:44.451 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.495 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:44.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:44.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:44.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:44.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:44.747 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:19:44.756 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:19:44.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:44.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:19:44.859 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:44.923 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:44.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:45.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:45.074 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:45.177 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:45.220 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:45.322 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:45.366 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:45.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:45.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:45.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:45 smithi066 ceph-mon[29666]: pgmap v981: 178 pgs: 178 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 954 op/s 2024-03-26T03:19:45.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:45 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:45.566 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:45 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:45.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:45 smithi066 ceph-mon[37920]: pgmap v981: 178 pgs: 178 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 954 op/s 2024-03-26T03:19:45.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:45 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:45.566 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:45 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:45.568 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:19:45.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:45.611 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:19:45.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:45.654 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:19:45.697 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:19:45.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:45 smithi192 ceph-mon[33110]: pgmap v981: 178 pgs: 178 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 954 op/s 2024-03-26T03:19:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:45 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:45 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:45.739 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:19:45.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:45.778 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:19:45.822 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:19:45.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:45.870 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:19:45.877 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:19:45.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:45.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:45.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:46.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:46.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:46.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:46.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:46.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:46.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.446 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:46.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:46.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:46.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:46.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:46 smithi066 ceph-mon[29666]: from='client.16128 v1:172.21.15.66:0/366857118' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:19:46.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:46 smithi066 ceph-mon[29666]: osdmap e456: 8 total, 8 up, 6 in 2024-03-26T03:19:46.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2261562155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:46.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:46 smithi066 ceph-mon[37920]: from='client.16128 v1:172.21.15.66:0/366857118' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:19:46.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:46 smithi066 ceph-mon[37920]: osdmap e456: 8 total, 8 up, 6 in 2024-03-26T03:19:46.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2261562155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:46.688 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:46.736 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:46 smithi192 ceph-mon[33110]: from='client.16128 v1:172.21.15.66:0/366857118' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:19:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:46 smithi192 ceph-mon[33110]: osdmap e456: 8 total, 8 up, 6 in 2024-03-26T03:19:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2261562155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:46.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:46.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:46.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:46.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:47.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:47.108 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":981,"stamp":"2024-03-26T03:19:44.461638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3232831936,"num_objects":49551,"num_object_clones":0,"num_object_copies":148653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49551,"num_whiteouts":0,"num_read":3078,"num_read_kb":2798,"num_write":215671,"num_write_kb":8441326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20849,"num_bytes_recovered":1351161765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58501,"ondisk_log_size":58501,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":535,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":15010936,"kb_used_data":9932996,"kb_used_omap":12,"kb_used_meta":5077235,"kb_avail":547459976,"statfs":{"total":575970213888,"available":560599015424,"internally_reserved":0,"allocated":10171387904,"data_stored":10169020583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12419,"internal_metadata":5199089533},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-500367360,"num_objects":-7635,"num_object_clones":0,"num_object_copies":-22905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7635,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7635,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003325"},"pg_stats":[{"pgid":"11.7","version":"454'13308","reported_seq":13502,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.792090+0000","last_change":"2024-03-26T03:18:52.598875+0000","last_active":"2024-03-26T03:19:39.792090+0000","last_peered":"2024-03-26T03:19:39.792090+0000","last_clean":"2024-03-26T03:19:39.792090+0000","last_became_active":"2024-03-26T03:16:56.900984+0000","last_became_peered":"2024-03-26T03:16:56.900984+0000","last_unstale":"2024-03-26T03:19:39.792090+0000","last_undegraded":"2024-03-26T03:19:39.792090+0000","last_fullsized":"2024-03-26T03:19:39.792090+0000","mapping_epoch":403,"log_start":"446'10000","ondisk_log_start":"446'10000","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'9185","last_scrub_stamp":"2024-03-26T03:18:52.598640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:18:52.598640+0000","objects_scrubbed":7305,"log_size":3308,"log_dups_size":928,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.644456+0000","scrub_duration":1807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203423744,"num_objects":3104,"num_object_clones":0,"num_object_copies":9312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13308,"num_write_kb":525184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"82'128","reported_seq":764,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594740+0000","last_change":"2024-03-26T03:16:31.121425+0000","last_active":"2024-03-26T03:19:39.594740+0000","last_peered":"2024-03-26T03:19:39.594740+0000","last_clean":"2024-03-26T03:19:39.594740+0000","last_became_active":"2024-03-26T03:14:40.174462+0000","last_became_peered":"2024-03-26T03:14:40.174462+0000","last_unstale":"2024-03-26T03:19:39.594740+0000","last_undegraded":"2024-03-26T03:19:39.594740+0000","last_fullsized":"2024-03-26T03:19:39.594740+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.121331+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.121331+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"342'10","reported_seq":465,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597190+0000","last_change":"2024-03-26T03:17:48.646060+0000","last_active":"2024-03-26T03:19:39.597190+0000","last_peered":"2024-03-26T03:19:39.597190+0000","last_clean":"2024-03-26T03:19:39.597190+0000","last_became_active":"2024-03-26T03:14:37.273000+0000","last_became_peered":"2024-03-26T03:14:37.273000+0000","last_unstale":"2024-03-26T03:19:39.597190+0000","last_undegraded":"2024-03-26T03:19:39.597190+0000","last_fullsized":"2024-03-26T03:19:39.597190+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'10","last_scrub_stamp":"2024-03-26T03:17:48.645977+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.645977+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"446'51","reported_seq":994,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599218+0000","last_change":"2024-03-26T03:18:51.773314+0000","last_active":"2024-03-26T03:19:39.599218+0000","last_peered":"2024-03-26T03:19:39.599218+0000","last_clean":"2024-03-26T03:19:39.599218+0000","last_became_active":"2024-03-26T03:14:37.267205+0000","last_became_peered":"2024-03-26T03:14:37.267205+0000","last_unstale":"2024-03-26T03:19:39.599218+0000","last_undegraded":"2024-03-26T03:19:39.599218+0000","last_fullsized":"2024-03-26T03:19:39.599218+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'39","last_scrub_stamp":"2024-03-26T03:18:51.773171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:51.773171+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:52.240058+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":274,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596957+0000","last_change":"2024-03-26T03:17:57.225859+0000","last_active":"2024-03-26T03:19:39.596957+0000","last_peered":"2024-03-26T03:19:39.596957+0000","last_clean":"2024-03-26T03:19:39.596957+0000","last_became_active":"2024-03-26T03:14:37.264771+0000","last_became_peered":"2024-03-26T03:14:37.264771+0000","last_unstale":"2024-03-26T03:19:39.596957+0000","last_undegraded":"2024-03-26T03:19:39.596957+0000","last_fullsized":"2024-03-26T03:19:39.596957+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":679,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595838+0000","last_change":"2024-03-26T03:18:07.660598+0000","last_active":"2024-03-26T03:19:39.595838+0000","last_peered":"2024-03-26T03:19:39.595838+0000","last_clean":"2024-03-26T03:19:39.595838+0000","last_became_active":"2024-03-26T03:13:35.269469+0000","last_became_peered":"2024-03-26T03:13:35.269469+0000","last_unstale":"2024-03-26T03:19:39.595838+0000","last_undegraded":"2024-03-26T03:19:39.595838+0000","last_fullsized":"2024-03-26T03:19:39.595838+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_clean_scrub_stamp":"2024-03-26T03:18:07.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"454'13203","reported_seq":13378,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.773185+0000","last_change":"2024-03-26T03:19:14.031564+0000","last_active":"2024-03-26T03:19:39.773185+0000","last_peered":"2024-03-26T03:19:39.773185+0000","last_clean":"2024-03-26T03:19:39.773185+0000","last_became_active":"2024-03-26T03:16:56.903390+0000","last_became_peered":"2024-03-26T03:16:56.903390+0000","last_unstale":"2024-03-26T03:19:39.773185+0000","last_undegraded":"2024-03-26T03:19:39.773185+0000","last_fullsized":"2024-03-26T03:19:39.773185+0000","mapping_epoch":403,"log_start":"445'9700","ondisk_log_start":"445'9700","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'10985","last_scrub_stamp":"2024-03-26T03:19:14.031321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:19:14.031321+0000","objects_scrubbed":5362,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:19.550278+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200736768,"num_objects":3063,"num_object_clones":0,"num_object_copies":9189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13203,"num_write_kb":520512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":368,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597550+0000","last_change":"2024-03-26T03:19:16.437899+0000","last_active":"2024-03-26T03:19:39.597550+0000","last_peered":"2024-03-26T03:19:39.597550+0000","last_clean":"2024-03-26T03:19:39.597550+0000","last_became_active":"2024-03-26T03:17:59.592113+0000","last_became_peered":"2024-03-26T03:17:59.592113+0000","last_unstale":"2024-03-26T03:19:39.597550+0000","last_undegraded":"2024-03-26T03:19:39.597550+0000","last_fullsized":"2024-03-26T03:19:39.597550+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_clean_scrub_stamp":"2024-03-26T03:19:16.437810+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:44.087975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"446'32","reported_seq":752,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.607386+0000","last_change":"2024-03-26T03:17:38.464630+0000","last_active":"2024-03-26T03:19:39.607386+0000","last_peered":"2024-03-26T03:19:39.607386+0000","last_clean":"2024-03-26T03:19:39.607386+0000","last_became_active":"2024-03-26T03:14:22.579044+0000","last_became_peered":"2024-03-26T03:14:22.579044+0000","last_unstale":"2024-03-26T03:19:39.607386+0000","last_undegraded":"2024-03-26T03:19:39.607386+0000","last_fullsized":"2024-03-26T03:19:39.607386+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:17:38.464358+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:17:38.464358+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105301+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":853,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.600045+0000","last_change":"2024-03-26T03:17:56.571530+0000","last_active":"2024-03-26T03:19:39.600045+0000","last_peered":"2024-03-26T03:19:39.600045+0000","last_clean":"2024-03-26T03:19:39.600045+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:19:39.600045+0000","last_undegraded":"2024-03-26T03:19:39.600045+0000","last_fullsized":"2024-03-26T03:19:39.600045+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:16.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:15:16.933002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094267+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":63,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593170+0000","last_change":"2024-03-26T03:17:56.568649+0000","last_active":"2024-03-26T03:19:39.593170+0000","last_peered":"2024-03-26T03:19:39.593170+0000","last_clean":"2024-03-26T03:19:39.593170+0000","last_became_active":"2024-03-26T03:17:56.568322+0000","last_became_peered":"2024-03-26T03:17:56.568322+0000","last_unstale":"2024-03-26T03:19:39.593170+0000","last_undegraded":"2024-03-26T03:19:39.593170+0000","last_fullsized":"2024-03-26T03:19:39.593170+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.222175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.222175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":167,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596333+0000","last_change":"2024-03-26T03:16:50.206535+0000","last_active":"2024-03-26T03:19:39.596333+0000","last_peered":"2024-03-26T03:19:39.596333+0000","last_clean":"2024-03-26T03:19:39.596333+0000","last_became_active":"2024-03-26T03:16:50.206224+0000","last_became_peered":"2024-03-26T03:16:50.206224+0000","last_unstale":"2024-03-26T03:19:39.596333+0000","last_undegraded":"2024-03-26T03:19:39.596333+0000","last_fullsized":"2024-03-26T03:19:39.596333+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:26.722546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:13:26.722546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.587831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"454'13390","reported_seq":13581,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.809182+0000","last_change":"2024-03-26T03:18:55.362846+0000","last_active":"2024-03-26T03:19:39.809182+0000","last_peered":"2024-03-26T03:19:39.809182+0000","last_clean":"2024-03-26T03:19:39.809182+0000","last_became_active":"2024-03-26T03:16:56.908452+0000","last_became_peered":"2024-03-26T03:16:56.908452+0000","last_unstale":"2024-03-26T03:19:39.809182+0000","last_undegraded":"2024-03-26T03:19:39.809182+0000","last_fullsized":"2024-03-26T03:19:39.809182+0000","mapping_epoch":403,"log_start":"446'10100","ondisk_log_start":"446'10100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'9482","last_scrub_stamp":"2024-03-26T03:18:55.362409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:18:55.362409+0000","objects_scrubbed":7039,"log_size":3290,"log_dups_size":986,"ondisk_log_size":3290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:20.391650+0000","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201457664,"num_objects":3074,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13390,"num_write_kb":526848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":940,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597464+0000","last_change":"2024-03-26T03:17:32.537859+0000","last_active":"2024-03-26T03:19:39.597464+0000","last_peered":"2024-03-26T03:19:39.597464+0000","last_clean":"2024-03-26T03:19:39.597464+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:19:39.597464+0000","last_undegraded":"2024-03-26T03:19:39.597464+0000","last_fullsized":"2024-03-26T03:19:39.597464+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:32.537761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.537761+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"446'57","reported_seq":1007,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596056+0000","last_change":"2024-03-26T03:18:02.671958+0000","last_active":"2024-03-26T03:19:39.596056+0000","last_peered":"2024-03-26T03:19:39.596056+0000","last_clean":"2024-03-26T03:19:39.596056+0000","last_became_active":"2024-03-26T03:14:37.266473+0000","last_became_peered":"2024-03-26T03:14:37.266473+0000","last_unstale":"2024-03-26T03:19:39.596056+0000","last_undegraded":"2024-03-26T03:19:39.596056+0000","last_fullsized":"2024-03-26T03:19:39.596056+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:18:02.671821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:02.671821+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":713,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.610362+0000","last_change":"2024-03-26T03:17:54.517355+0000","last_active":"2024-03-26T03:19:39.610362+0000","last_peered":"2024-03-26T03:19:39.610362+0000","last_clean":"2024-03-26T03:19:39.610362+0000","last_became_active":"2024-03-26T03:16:50.208923+0000","last_became_peered":"2024-03-26T03:16:50.208923+0000","last_unstale":"2024-03-26T03:19:39.610362+0000","last_undegraded":"2024-03-26T03:19:39.610362+0000","last_fullsized":"2024-03-26T03:19:39.610362+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:54.517230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.517230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105460+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":884,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.590504+0000","last_change":"2024-03-26T03:18:05.489195+0000","last_active":"2024-03-26T03:19:39.590504+0000","last_peered":"2024-03-26T03:19:39.590504+0000","last_clean":"2024-03-26T03:19:39.590504+0000","last_became_active":"2024-03-26T03:14:37.242217+0000","last_became_peered":"2024-03-26T03:14:37.242217+0000","last_unstale":"2024-03-26T03:19:39.590504+0000","last_undegraded":"2024-03-26T03:19:39.590504+0000","last_fullsized":"2024-03-26T03:19:39.590504+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:05.489069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.227664+0000","last_clean_scrub_stamp":"2024-03-26T03:18:05.489069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:39.697037+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":804,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595364+0000","last_change":"2024-03-26T03:17:40.190354+0000","last_active":"2024-03-26T03:19:39.595364+0000","last_peered":"2024-03-26T03:19:39.595364+0000","last_clean":"2024-03-26T03:19:39.595364+0000","last_became_active":"2024-03-26T03:14:37.270129+0000","last_became_peered":"2024-03-26T03:14:37.270129+0000","last_unstale":"2024-03-26T03:19:39.595364+0000","last_undegraded":"2024-03-26T03:19:39.595364+0000","last_fullsized":"2024-03-26T03:19:39.595364+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.189873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.189873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592302+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"454'13548","reported_seq":13661,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.622759+0000","last_change":"2024-03-26T03:16:56.905137+0000","last_active":"2024-03-26T03:19:39.622759+0000","last_peered":"2024-03-26T03:19:39.622759+0000","last_clean":"2024-03-26T03:19:39.622759+0000","last_became_active":"2024-03-26T03:16:56.904798+0000","last_became_peered":"2024-03-26T03:16:56.904798+0000","last_unstale":"2024-03-26T03:19:39.622759+0000","last_undegraded":"2024-03-26T03:19:39.622759+0000","last_fullsized":"2024-03-26T03:19:39.622759+0000","mapping_epoch":403,"log_start":"446'10300","ondisk_log_start":"446'10300","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3248,"log_dups_size":2487,"ondisk_log_size":3248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206438400,"num_objects":3150,"num_object_clones":0,"num_object_copies":9450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13548,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"446'53","reported_seq":102,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594536+0000","last_change":"2024-03-26T03:17:56.585807+0000","last_active":"2024-03-26T03:19:39.594536+0000","last_peered":"2024-03-26T03:19:39.594536+0000","last_clean":"2024-03-26T03:19:39.594536+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:19:39.594536+0000","last_undegraded":"2024-03-26T03:19:39.594536+0000","last_fullsized":"2024-03-26T03:19:39.594536+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:17:36.443102+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:17:36.443102+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307511+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":63,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595996+0000","last_change":"2024-03-26T03:17:56.568076+0000","last_active":"2024-03-26T03:19:39.595996+0000","last_peered":"2024-03-26T03:19:39.595996+0000","last_clean":"2024-03-26T03:19:39.595996+0000","last_became_active":"2024-03-26T03:17:56.567051+0000","last_became_peered":"2024-03-26T03:17:56.567051+0000","last_unstale":"2024-03-26T03:19:39.595996+0000","last_undegraded":"2024-03-26T03:19:39.595996+0000","last_fullsized":"2024-03-26T03:19:39.595996+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.464363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.464363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":854,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.615137+0000","last_change":"2024-03-26T03:16:46.947202+0000","last_active":"2024-03-26T03:19:39.615137+0000","last_peered":"2024-03-26T03:19:39.615137+0000","last_clean":"2024-03-26T03:19:39.615137+0000","last_became_active":"2024-03-26T03:13:35.277164+0000","last_became_peered":"2024-03-26T03:13:35.277164+0000","last_unstale":"2024-03-26T03:19:39.615137+0000","last_undegraded":"2024-03-26T03:19:39.615137+0000","last_fullsized":"2024-03-26T03:19:39.615137+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:46.947118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.947118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":268,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.603558+0000","last_change":"2024-03-26T03:18:50.863004+0000","last_active":"2024-03-26T03:19:39.603558+0000","last_peered":"2024-03-26T03:19:39.603558+0000","last_clean":"2024-03-26T03:19:39.603558+0000","last_became_active":"2024-03-26T03:14:21.586859+0000","last_became_peered":"2024-03-26T03:14:21.586859+0000","last_unstale":"2024-03-26T03:19:39.603558+0000","last_undegraded":"2024-03-26T03:19:39.603558+0000","last_fullsized":"2024-03-26T03:19:39.603558+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:50.862872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:50.862872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:59.623780+0000","scrub_duration":310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":562,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595101+0000","last_change":"2024-03-26T03:16:28.866577+0000","last_active":"2024-03-26T03:19:39.595101+0000","last_peered":"2024-03-26T03:19:39.595101+0000","last_clean":"2024-03-26T03:19:39.595101+0000","last_became_active":"2024-03-26T03:14:21.590629+0000","last_became_peered":"2024-03-26T03:14:21.590629+0000","last_unstale":"2024-03-26T03:19:39.595101+0000","last_undegraded":"2024-03-26T03:19:39.595101+0000","last_fullsized":"2024-03-26T03:19:39.595101+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:28.866477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.866477+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"454'13277","reported_seq":13390,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.777656+0000","last_change":"2024-03-26T03:16:56.903756+0000","last_active":"2024-03-26T03:19:39.777656+0000","last_peered":"2024-03-26T03:19:39.777656+0000","last_clean":"2024-03-26T03:19:39.777656+0000","last_became_active":"2024-03-26T03:16:56.903414+0000","last_became_peered":"2024-03-26T03:16:56.903414+0000","last_unstale":"2024-03-26T03:19:39.777656+0000","last_undegraded":"2024-03-26T03:19:39.777656+0000","last_fullsized":"2024-03-26T03:19:39.777656+0000","mapping_epoch":403,"log_start":"445'9800","ondisk_log_start":"445'9800","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3477,"log_dups_size":0,"ondisk_log_size":3477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202964992,"num_objects":3097,"num_object_clones":0,"num_object_copies":9291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13277,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":729,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595083+0000","last_change":"2024-03-26T03:16:30.821771+0000","last_active":"2024-03-26T03:19:39.595083+0000","last_peered":"2024-03-26T03:19:39.595083+0000","last_clean":"2024-03-26T03:19:39.595083+0000","last_became_active":"2024-03-26T03:14:40.169332+0000","last_became_peered":"2024-03-26T03:14:40.169332+0000","last_unstale":"2024-03-26T03:19:39.595083+0000","last_undegraded":"2024-03-26T03:19:39.595083+0000","last_fullsized":"2024-03-26T03:19:39.595083+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.821670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.821670+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"446'30","reported_seq":893,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594729+0000","last_change":"2024-03-26T03:16:39.220608+0000","last_active":"2024-03-26T03:19:39.594729+0000","last_peered":"2024-03-26T03:19:39.594729+0000","last_clean":"2024-03-26T03:19:39.594729+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:19:39.594729+0000","last_undegraded":"2024-03-26T03:19:39.594729+0000","last_fullsized":"2024-03-26T03:19:39.594729+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'20","last_scrub_stamp":"2024-03-26T03:16:39.220501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.220501+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305101+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":839,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.607999+0000","last_change":"2024-03-26T03:15:09.812769+0000","last_active":"2024-03-26T03:19:39.607999+0000","last_peered":"2024-03-26T03:19:39.607999+0000","last_clean":"2024-03-26T03:19:39.607999+0000","last_became_active":"2024-03-26T03:14:37.251633+0000","last_became_peered":"2024-03-26T03:14:37.251633+0000","last_unstale":"2024-03-26T03:19:39.607999+0000","last_undegraded":"2024-03-26T03:19:39.607999+0000","last_fullsized":"2024-03-26T03:19:39.607999+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:09.812652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:15:09.812652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"401'11","reported_seq":984,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599074+0000","last_change":"2024-03-26T03:19:15.476137+0000","last_active":"2024-03-26T03:19:39.599074+0000","last_peered":"2024-03-26T03:19:39.599074+0000","last_clean":"2024-03-26T03:19:39.599074+0000","last_became_active":"2024-03-26T03:16:50.206797+0000","last_became_peered":"2024-03-26T03:16:50.206797+0000","last_unstale":"2024-03-26T03:19:39.599074+0000","last_undegraded":"2024-03-26T03:19:39.599074+0000","last_fullsized":"2024-03-26T03:19:39.599074+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'11","last_scrub_stamp":"2024-03-26T03:19:15.476016+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:19:15.476016+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:32.165218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":63,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597682+0000","last_change":"2024-03-26T03:17:56.574248+0000","last_active":"2024-03-26T03:19:39.597682+0000","last_peered":"2024-03-26T03:19:39.597682+0000","last_clean":"2024-03-26T03:19:39.597682+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:19:39.597682+0000","last_undegraded":"2024-03-26T03:19:39.597682+0000","last_fullsized":"2024-03-26T03:19:39.597682+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:32.853807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.853807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"454'13352","reported_seq":13465,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:40.695375+0000","last_change":"2024-03-26T03:16:56.909076+0000","last_active":"2024-03-26T03:19:40.695375+0000","last_peered":"2024-03-26T03:19:40.695375+0000","last_clean":"2024-03-26T03:19:40.695375+0000","last_became_active":"2024-03-26T03:16:56.908486+0000","last_became_peered":"2024-03-26T03:16:56.908486+0000","last_unstale":"2024-03-26T03:19:40.695375+0000","last_undegraded":"2024-03-26T03:19:40.695375+0000","last_fullsized":"2024-03-26T03:19:40.695375+0000","mapping_epoch":403,"log_start":"446'10100","ondisk_log_start":"446'10100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3252,"log_dups_size":0,"ondisk_log_size":3252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.103611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13352,"num_write_kb":521536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":427,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595711+0000","last_change":"2024-03-26T03:19:17.349314+0000","last_active":"2024-03-26T03:19:39.595711+0000","last_peered":"2024-03-26T03:19:39.595711+0000","last_clean":"2024-03-26T03:19:39.595711+0000","last_became_active":"2024-03-26T03:16:50.205474+0000","last_became_peered":"2024-03-26T03:16:50.205474+0000","last_unstale":"2024-03-26T03:19:39.595711+0000","last_undegraded":"2024-03-26T03:19:39.595711+0000","last_fullsized":"2024-03-26T03:19:39.595711+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:17.349212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:17.349212+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.072644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"446'39","reported_seq":921,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.602516+0000","last_change":"2024-03-26T03:16:28.788850+0000","last_active":"2024-03-26T03:19:39.602516+0000","last_peered":"2024-03-26T03:19:39.602516+0000","last_clean":"2024-03-26T03:19:39.602516+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:19:39.602516+0000","last_undegraded":"2024-03-26T03:19:39.602516+0000","last_fullsized":"2024-03-26T03:19:39.602516+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'27","last_scrub_stamp":"2024-03-26T03:16:28.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.788749+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":839,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597700+0000","last_change":"2024-03-26T03:14:39.896972+0000","last_active":"2024-03-26T03:19:39.597700+0000","last_peered":"2024-03-26T03:19:39.597700+0000","last_clean":"2024-03-26T03:19:39.597700+0000","last_became_active":"2024-03-26T03:14:37.249790+0000","last_became_peered":"2024-03-26T03:14:37.249790+0000","last_unstale":"2024-03-26T03:19:39.597700+0000","last_undegraded":"2024-03-26T03:19:39.597700+0000","last_fullsized":"2024-03-26T03:19:39.597700+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:39.896823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:14:39.896823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"428'12","reported_seq":89,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597775+0000","last_change":"2024-03-26T03:17:56.575479+0000","last_active":"2024-03-26T03:19:39.597775+0000","last_peered":"2024-03-26T03:19:39.597775+0000","last_clean":"2024-03-26T03:19:39.597775+0000","last_became_active":"2024-03-26T03:17:56.566007+0000","last_became_peered":"2024-03-26T03:17:56.566007+0000","last_unstale":"2024-03-26T03:19:39.597775+0000","last_undegraded":"2024-03-26T03:19:39.597775+0000","last_fullsized":"2024-03-26T03:19:39.597775+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'8","last_scrub_stamp":"2024-03-26T03:13:30.083400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:30.083400+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":662,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599562+0000","last_change":"2024-03-26T03:19:19.190221+0000","last_active":"2024-03-26T03:19:39.599562+0000","last_peered":"2024-03-26T03:19:39.599562+0000","last_clean":"2024-03-26T03:19:39.599562+0000","last_became_active":"2024-03-26T03:13:35.275396+0000","last_became_peered":"2024-03-26T03:13:35.275396+0000","last_unstale":"2024-03-26T03:19:39.599562+0000","last_undegraded":"2024-03-26T03:19:39.599562+0000","last_fullsized":"2024-03-26T03:19:39.599562+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:19.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:19:19.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.391231+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"454'13515","reported_seq":13675,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.751499+0000","last_change":"2024-03-26T03:19:19.448772+0000","last_active":"2024-03-26T03:19:39.751499+0000","last_peered":"2024-03-26T03:19:39.751499+0000","last_clean":"2024-03-26T03:19:39.751499+0000","last_became_active":"2024-03-26T03:16:56.905606+0000","last_became_peered":"2024-03-26T03:16:56.905606+0000","last_unstale":"2024-03-26T03:19:39.751499+0000","last_undegraded":"2024-03-26T03:19:39.751499+0000","last_fullsized":"2024-03-26T03:19:39.751499+0000","mapping_epoch":403,"log_start":"445'10000","ondisk_log_start":"445'10000","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":4399,"log_size":3515,"log_dups_size":0,"ondisk_log_size":3515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196280320,"num_objects":2995,"num_object_clones":0,"num_object_copies":8985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13515,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":980,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.602594+0000","last_change":"2024-03-26T03:17:39.489433+0000","last_active":"2024-03-26T03:19:39.602594+0000","last_peered":"2024-03-26T03:19:39.602594+0000","last_clean":"2024-03-26T03:19:39.602594+0000","last_became_active":"2024-03-26T03:13:38.314531+0000","last_became_peered":"2024-03-26T03:13:38.314531+0000","last_unstale":"2024-03-26T03:19:39.602594+0000","last_undegraded":"2024-03-26T03:19:39.602594+0000","last_fullsized":"2024-03-26T03:19:39.602594+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:39.489210+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:17:39.489210+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105395+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"446'39","reported_seq":919,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598214+0000","last_change":"2024-03-26T03:17:42.061990+0000","last_active":"2024-03-26T03:19:39.598214+0000","last_peered":"2024-03-26T03:19:39.598214+0000","last_clean":"2024-03-26T03:19:39.598214+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:19:39.598214+0000","last_undegraded":"2024-03-26T03:19:39.598214+0000","last_fullsized":"2024-03-26T03:19:39.598214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:17:42.061780+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.061780+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307479+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":808,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601162+0000","last_change":"2024-03-26T03:14:37.253086+0000","last_active":"2024-03-26T03:19:39.601162+0000","last_peered":"2024-03-26T03:19:39.601162+0000","last_clean":"2024-03-26T03:19:39.601162+0000","last_became_active":"2024-03-26T03:14:37.252745+0000","last_became_peered":"2024-03-26T03:14:37.252745+0000","last_unstale":"2024-03-26T03:19:39.601162+0000","last_undegraded":"2024-03-26T03:19:39.601162+0000","last_fullsized":"2024-03-26T03:19:39.601162+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:31.980008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.980008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094108+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":659,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598679+0000","last_change":"2024-03-26T03:16:55.201337+0000","last_active":"2024-03-26T03:19:39.598679+0000","last_peered":"2024-03-26T03:19:39.598679+0000","last_clean":"2024-03-26T03:19:39.598679+0000","last_became_active":"2024-03-26T03:14:37.249741+0000","last_became_peered":"2024-03-26T03:14:37.249741+0000","last_unstale":"2024-03-26T03:19:39.598679+0000","last_undegraded":"2024-03-26T03:19:39.598679+0000","last_fullsized":"2024-03-26T03:19:39.598679+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.201190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.201190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":481,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595035+0000","last_change":"2024-03-26T03:17:44.249680+0000","last_active":"2024-03-26T03:19:39.595035+0000","last_peered":"2024-03-26T03:19:39.595035+0000","last_clean":"2024-03-26T03:19:39.595035+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:19:39.595035+0000","last_undegraded":"2024-03-26T03:19:39.595035+0000","last_fullsized":"2024-03-26T03:19:39.595035+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.249561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.249561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"454'13205","reported_seq":27211,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.730603+0000","last_change":"2024-03-26T03:18:56.612971+0000","last_active":"2024-03-26T03:19:39.730603+0000","last_peered":"2024-03-26T03:19:39.730603+0000","last_clean":"2024-03-26T03:19:39.730603+0000","last_became_active":"2024-03-26T03:18:56.603507+0000","last_became_peered":"2024-03-26T03:18:56.603507+0000","last_unstale":"2024-03-26T03:19:39.730603+0000","last_undegraded":"2024-03-26T03:19:39.730603+0000","last_fullsized":"2024-03-26T03:19:39.730603+0000","mapping_epoch":444,"log_start":"445'9700","ondisk_log_start":"445'9700","created":403,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3505,"log_dups_size":0,"ondisk_log_size":3505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198508544,"num_objects":3029,"num_object_clones":0,"num_object_copies":9087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6940,"num_bytes_recovered":454819840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":671,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594991+0000","last_change":"2024-03-26T03:17:55.204987+0000","last_active":"2024-03-26T03:19:39.594991+0000","last_peered":"2024-03-26T03:19:39.594991+0000","last_clean":"2024-03-26T03:19:39.594991+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:19:39.594991+0000","last_undegraded":"2024-03-26T03:19:39.594991+0000","last_fullsized":"2024-03-26T03:19:39.594991+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:17:55.204836+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.204836+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592368+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"446'48","reported_seq":933,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592806+0000","last_change":"2024-03-26T03:16:56.199180+0000","last_active":"2024-03-26T03:19:39.592806+0000","last_peered":"2024-03-26T03:19:39.592806+0000","last_clean":"2024-03-26T03:19:39.592806+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:19:39.592806+0000","last_undegraded":"2024-03-26T03:19:39.592806+0000","last_fullsized":"2024-03-26T03:19:39.592806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:16:56.199057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.199057+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":870,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598395+0000","last_change":"2024-03-26T03:17:41.533036+0000","last_active":"2024-03-26T03:19:39.598395+0000","last_peered":"2024-03-26T03:19:39.598395+0000","last_clean":"2024-03-26T03:19:39.598395+0000","last_became_active":"2024-03-26T03:14:37.276008+0000","last_became_peered":"2024-03-26T03:14:37.276008+0000","last_unstale":"2024-03-26T03:19:39.598395+0000","last_undegraded":"2024-03-26T03:19:39.598395+0000","last_fullsized":"2024-03-26T03:19:39.598395+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.532899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:29.683310+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.532899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":246,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597313+0000","last_change":"2024-03-26T03:16:40.872720+0000","last_active":"2024-03-26T03:19:39.597313+0000","last_peered":"2024-03-26T03:19:39.597313+0000","last_clean":"2024-03-26T03:19:39.597313+0000","last_became_active":"2024-03-26T03:14:37.248955+0000","last_became_peered":"2024-03-26T03:14:37.248955+0000","last_unstale":"2024-03-26T03:19:39.597313+0000","last_undegraded":"2024-03-26T03:19:39.597313+0000","last_fullsized":"2024-03-26T03:19:39.597313+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:40.872619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.872619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":651,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595548+0000","last_change":"2024-03-26T03:16:29.852173+0000","last_active":"2024-03-26T03:19:39.595548+0000","last_peered":"2024-03-26T03:19:39.595548+0000","last_clean":"2024-03-26T03:19:39.595548+0000","last_became_active":"2024-03-26T03:14:40.166583+0000","last_became_peered":"2024-03-26T03:14:40.166583+0000","last_unstale":"2024-03-26T03:19:39.595548+0000","last_undegraded":"2024-03-26T03:19:39.595548+0000","last_fullsized":"2024-03-26T03:19:39.595548+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.852073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.852073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"454'13672","reported_seq":13785,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:40.651428+0000","last_change":"2024-03-26T03:16:56.900055+0000","last_active":"2024-03-26T03:19:40.651428+0000","last_peered":"2024-03-26T03:19:40.651428+0000","last_clean":"2024-03-26T03:19:40.651428+0000","last_became_active":"2024-03-26T03:16:56.899679+0000","last_became_peered":"2024-03-26T03:16:56.899679+0000","last_unstale":"2024-03-26T03:19:40.651428+0000","last_undegraded":"2024-03-26T03:19:40.651428+0000","last_fullsized":"2024-03-26T03:19:40.651428+0000","mapping_epoch":403,"log_start":"446'10400","ondisk_log_start":"446'10400","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":3072,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13672,"num_write_kb":535808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":829,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592984+0000","last_change":"2024-03-26T03:16:36.206735+0000","last_active":"2024-03-26T03:19:39.592984+0000","last_peered":"2024-03-26T03:19:39.592984+0000","last_clean":"2024-03-26T03:19:39.592984+0000","last_became_active":"2024-03-26T03:14:37.257004+0000","last_became_peered":"2024-03-26T03:14:37.257004+0000","last_unstale":"2024-03-26T03:19:39.592984+0000","last_undegraded":"2024-03-26T03:19:39.592984+0000","last_fullsized":"2024-03-26T03:19:39.592984+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:36.206634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.206634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"446'66","reported_seq":663,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595746+0000","last_change":"2024-03-26T03:16:38.931105+0000","last_active":"2024-03-26T03:19:39.595746+0000","last_peered":"2024-03-26T03:19:39.595746+0000","last_clean":"2024-03-26T03:19:39.595746+0000","last_became_active":"2024-03-26T03:14:37.278227+0000","last_became_peered":"2024-03-26T03:14:37.278227+0000","last_unstale":"2024-03-26T03:19:39.595746+0000","last_undegraded":"2024-03-26T03:19:39.595746+0000","last_fullsized":"2024-03-26T03:19:39.595746+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'54","last_scrub_stamp":"2024-03-26T03:16:38.931005+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.931005+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590782+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":858,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.600965+0000","last_change":"2024-03-26T03:13:29.030426+0000","last_active":"2024-03-26T03:19:39.600965+0000","last_peered":"2024-03-26T03:19:39.600965+0000","last_clean":"2024-03-26T03:19:39.600965+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:19:39.600965+0000","last_undegraded":"2024-03-26T03:19:39.600965+0000","last_fullsized":"2024-03-26T03:19:39.600965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:13:29.030298+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.030298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093832+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":63,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593246+0000","last_change":"2024-03-26T03:17:56.567975+0000","last_active":"2024-03-26T03:19:39.593246+0000","last_peered":"2024-03-26T03:19:39.593246+0000","last_clean":"2024-03-26T03:19:39.593246+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:19:39.593246+0000","last_undegraded":"2024-03-26T03:19:39.593246+0000","last_fullsized":"2024-03-26T03:19:39.593246+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.179410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.179410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307686+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":963,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594838+0000","last_change":"2024-03-26T03:17:54.095189+0000","last_active":"2024-03-26T03:19:39.594838+0000","last_peered":"2024-03-26T03:19:39.594838+0000","last_clean":"2024-03-26T03:19:39.594838+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:19:39.594838+0000","last_undegraded":"2024-03-26T03:19:39.594838+0000","last_fullsized":"2024-03-26T03:19:39.594838+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:54.095058+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.095058+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307616+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"454'13249","reported_seq":27196,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:40.676331+0000","last_change":"2024-03-26T03:19:16.501274+0000","last_active":"2024-03-26T03:19:40.676331+0000","last_peered":"2024-03-26T03:19:40.676331+0000","last_clean":"2024-03-26T03:19:40.676331+0000","last_became_active":"2024-03-26T03:19:16.441598+0000","last_became_peered":"2024-03-26T03:19:16.441598+0000","last_unstale":"2024-03-26T03:19:40.676331+0000","last_undegraded":"2024-03-26T03:19:40.676331+0000","last_fullsized":"2024-03-26T03:19:40.676331+0000","mapping_epoch":450,"log_start":"446'10000","ondisk_log_start":"446'10000","created":403,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.103396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196673536,"num_objects":3001,"num_object_clones":0,"num_object_copies":9003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13249,"num_write_kb":520000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6905,"num_bytes_recovered":452395008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":862,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596152+0000","last_change":"2024-03-26T03:17:34.573811+0000","last_active":"2024-03-26T03:19:39.596152+0000","last_peered":"2024-03-26T03:19:39.596152+0000","last_clean":"2024-03-26T03:19:39.596152+0000","last_became_active":"2024-03-26T03:13:35.269721+0000","last_became_peered":"2024-03-26T03:13:35.269721+0000","last_unstale":"2024-03-26T03:19:39.596152+0000","last_undegraded":"2024-03-26T03:19:39.596152+0000","last_fullsized":"2024-03-26T03:19:39.596152+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.573719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.573719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"446'45","reported_seq":817,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598526+0000","last_change":"2024-03-26T03:18:58.829566+0000","last_active":"2024-03-26T03:19:39.598526+0000","last_peered":"2024-03-26T03:19:39.598526+0000","last_clean":"2024-03-26T03:19:39.598526+0000","last_became_active":"2024-03-26T03:14:37.258677+0000","last_became_peered":"2024-03-26T03:14:37.258677+0000","last_unstale":"2024-03-26T03:19:39.598526+0000","last_undegraded":"2024-03-26T03:19:39.598526+0000","last_fullsized":"2024-03-26T03:19:39.598526+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:18:58.829458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:58.829458+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:27.144987+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":77,"num_read_kb":62,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":793,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595053+0000","last_change":"2024-03-26T03:14:37.257762+0000","last_active":"2024-03-26T03:19:39.595053+0000","last_peered":"2024-03-26T03:19:39.595053+0000","last_clean":"2024-03-26T03:19:39.595053+0000","last_became_active":"2024-03-26T03:14:37.257422+0000","last_became_peered":"2024-03-26T03:14:37.257422+0000","last_unstale":"2024-03-26T03:19:39.595053+0000","last_undegraded":"2024-03-26T03:19:39.595053+0000","last_fullsized":"2024-03-26T03:19:39.595053+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:35.954172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:35.954172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304790+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":847,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597885+0000","last_change":"2024-03-26T03:13:24.055056+0000","last_active":"2024-03-26T03:19:39.597885+0000","last_peered":"2024-03-26T03:19:39.597885+0000","last_clean":"2024-03-26T03:19:39.597885+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:19:39.597885+0000","last_undegraded":"2024-03-26T03:19:39.597885+0000","last_fullsized":"2024-03-26T03:19:39.597885+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:24.054721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:13:24.054721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093802+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":940,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596716+0000","last_change":"2024-03-26T03:18:53.735082+0000","last_active":"2024-03-26T03:19:39.596716+0000","last_peered":"2024-03-26T03:19:39.596716+0000","last_clean":"2024-03-26T03:19:39.596716+0000","last_became_active":"2024-03-26T03:14:22.569851+0000","last_became_peered":"2024-03-26T03:14:22.569851+0000","last_unstale":"2024-03-26T03:19:39.596716+0000","last_undegraded":"2024-03-26T03:19:39.596716+0000","last_fullsized":"2024-03-26T03:19:39.596716+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:53.734968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:18:53.734968+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:54.266150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"453'13304","reported_seq":13417,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596568+0000","last_change":"2024-03-26T03:16:56.907700+0000","last_active":"2024-03-26T03:19:39.596568+0000","last_peered":"2024-03-26T03:19:39.596568+0000","last_clean":"2024-03-26T03:19:39.596568+0000","last_became_active":"2024-03-26T03:16:56.907183+0000","last_became_peered":"2024-03-26T03:16:56.907183+0000","last_unstale":"2024-03-26T03:19:39.596568+0000","last_undegraded":"2024-03-26T03:19:39.596568+0000","last_fullsized":"2024-03-26T03:19:39.596568+0000","mapping_epoch":403,"log_start":"446'10100","ondisk_log_start":"446'10100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3204,"log_dups_size":2387,"ondisk_log_size":3204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207355904,"num_objects":3164,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13304,"num_write_kb":526976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":885,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599762+0000","last_change":"2024-03-26T03:19:09.827441+0000","last_active":"2024-03-26T03:19:39.599762+0000","last_peered":"2024-03-26T03:19:39.599762+0000","last_clean":"2024-03-26T03:19:39.599762+0000","last_became_active":"2024-03-26T03:13:35.271824+0000","last_became_peered":"2024-03-26T03:13:35.271824+0000","last_unstale":"2024-03-26T03:19:39.599762+0000","last_undegraded":"2024-03-26T03:19:39.599762+0000","last_fullsized":"2024-03-26T03:19:39.599762+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:09.827318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:09.827318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:15.407835+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"446'42","reported_seq":769,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601235+0000","last_change":"2024-03-26T03:14:37.277935+0000","last_active":"2024-03-26T03:19:39.601235+0000","last_peered":"2024-03-26T03:19:39.601235+0000","last_clean":"2024-03-26T03:19:39.601235+0000","last_became_active":"2024-03-26T03:14:37.246803+0000","last_became_peered":"2024-03-26T03:14:37.246803+0000","last_unstale":"2024-03-26T03:19:39.601235+0000","last_undegraded":"2024-03-26T03:19:39.601235+0000","last_fullsized":"2024-03-26T03:19:39.601235+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:13:15.467975+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:13:15.467975+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093913+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":782,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601120+0000","last_change":"2024-03-26T03:16:56.019977+0000","last_active":"2024-03-26T03:19:39.601120+0000","last_peered":"2024-03-26T03:19:39.601120+0000","last_clean":"2024-03-26T03:19:39.601120+0000","last_became_active":"2024-03-26T03:14:37.265520+0000","last_became_peered":"2024-03-26T03:14:37.265520+0000","last_unstale":"2024-03-26T03:19:39.601120+0000","last_undegraded":"2024-03-26T03:19:39.601120+0000","last_fullsized":"2024-03-26T03:19:39.601120+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:56.019845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.019845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592529+0000","last_change":"2024-03-26T03:17:56.568202+0000","last_active":"2024-03-26T03:19:39.592529+0000","last_peered":"2024-03-26T03:19:39.592529+0000","last_clean":"2024-03-26T03:19:39.592529+0000","last_became_active":"2024-03-26T03:17:56.567590+0000","last_became_peered":"2024-03-26T03:17:56.567590+0000","last_unstale":"2024-03-26T03:19:39.592529+0000","last_undegraded":"2024-03-26T03:19:39.592529+0000","last_fullsized":"2024-03-26T03:19:39.592529+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:06:18.665375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:06:18.665375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304762+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":829,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.616246+0000","last_change":"2024-03-26T03:15:17.827310+0000","last_active":"2024-03-26T03:19:39.616246+0000","last_peered":"2024-03-26T03:19:39.616246+0000","last_clean":"2024-03-26T03:19:39.616246+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:19:39.616246+0000","last_undegraded":"2024-03-26T03:19:39.616246+0000","last_fullsized":"2024-03-26T03:19:39.616246+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:17.827186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:15:17.827186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":969,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.607098+0000","last_change":"2024-03-26T03:17:59.603321+0000","last_active":"2024-03-26T03:19:39.607098+0000","last_peered":"2024-03-26T03:19:39.607098+0000","last_clean":"2024-03-26T03:19:39.607098+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:19:39.607098+0000","last_undegraded":"2024-03-26T03:19:39.607098+0000","last_fullsized":"2024-03-26T03:19:39.607098+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:43.863351+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:43.863351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.103557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"454'13286","reported_seq":13399,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596806+0000","last_change":"2024-03-26T03:16:56.907179+0000","last_active":"2024-03-26T03:19:39.596806+0000","last_peered":"2024-03-26T03:19:39.596806+0000","last_clean":"2024-03-26T03:19:39.596806+0000","last_became_active":"2024-03-26T03:16:56.906797+0000","last_became_peered":"2024-03-26T03:16:56.906797+0000","last_unstale":"2024-03-26T03:19:39.596806+0000","last_undegraded":"2024-03-26T03:19:39.596806+0000","last_fullsized":"2024-03-26T03:19:39.596806+0000","mapping_epoch":403,"log_start":"446'10100","ondisk_log_start":"446'10100","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":2387,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206176256,"num_objects":3146,"num_object_clones":0,"num_object_copies":9438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13286,"num_write_kb":525824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":121,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.610234+0000","last_change":"2024-03-26T03:16:50.209616+0000","last_active":"2024-03-26T03:19:39.610234+0000","last_peered":"2024-03-26T03:19:39.610234+0000","last_clean":"2024-03-26T03:19:39.610234+0000","last_became_active":"2024-03-26T03:16:50.208883+0000","last_became_peered":"2024-03-26T03:16:50.208883+0000","last_unstale":"2024-03-26T03:19:39.610234+0000","last_undegraded":"2024-03-26T03:19:39.610234+0000","last_fullsized":"2024-03-26T03:19:39.610234+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.231346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.231346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"446'41","reported_seq":1199,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596901+0000","last_change":"2024-03-26T03:19:08.477738+0000","last_active":"2024-03-26T03:19:39.596901+0000","last_peered":"2024-03-26T03:19:39.596901+0000","last_clean":"2024-03-26T03:19:39.596901+0000","last_became_active":"2024-03-26T03:14:37.265178+0000","last_became_peered":"2024-03-26T03:14:37.265178+0000","last_unstale":"2024-03-26T03:19:39.596901+0000","last_undegraded":"2024-03-26T03:19:39.596901+0000","last_fullsized":"2024-03-26T03:19:39.596901+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'41","last_scrub_stamp":"2024-03-26T03:19:08.477608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:08.477608+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:36.652940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":549,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595309+0000","last_change":"2024-03-26T03:19:05.588431+0000","last_active":"2024-03-26T03:19:39.595309+0000","last_peered":"2024-03-26T03:19:39.595309+0000","last_clean":"2024-03-26T03:19:39.595309+0000","last_became_active":"2024-03-26T03:14:40.180729+0000","last_became_peered":"2024-03-26T03:14:40.180729+0000","last_unstale":"2024-03-26T03:19:39.595309+0000","last_undegraded":"2024-03-26T03:19:39.595309+0000","last_fullsized":"2024-03-26T03:19:39.595309+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:05.588322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:05.588322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:22.612472+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":899,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598434+0000","last_change":"2024-03-26T03:17:52.570959+0000","last_active":"2024-03-26T03:19:39.598434+0000","last_peered":"2024-03-26T03:19:39.598434+0000","last_clean":"2024-03-26T03:19:39.598434+0000","last_became_active":"2024-03-26T03:14:37.274568+0000","last_became_peered":"2024-03-26T03:14:37.274568+0000","last_unstale":"2024-03-26T03:19:39.598434+0000","last_undegraded":"2024-03-26T03:19:39.598434+0000","last_fullsized":"2024-03-26T03:19:39.598434+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.570835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.570835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":543,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596861+0000","last_change":"2024-03-26T03:19:03.628055+0000","last_active":"2024-03-26T03:19:39.596861+0000","last_peered":"2024-03-26T03:19:39.596861+0000","last_clean":"2024-03-26T03:19:39.596861+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:19:39.596861+0000","last_undegraded":"2024-03-26T03:19:39.596861+0000","last_fullsized":"2024-03-26T03:19:39.596861+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:19:03.627938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:03.627938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:12.334666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":836,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599793+0000","last_change":"2024-03-26T03:17:41.822116+0000","last_active":"2024-03-26T03:19:39.599793+0000","last_peered":"2024-03-26T03:19:39.599793+0000","last_clean":"2024-03-26T03:19:39.599793+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:19:39.599793+0000","last_undegraded":"2024-03-26T03:19:39.599793+0000","last_fullsized":"2024-03-26T03:19:39.599793+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:41.822032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:41.822032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"454'13574","reported_seq":13687,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595251+0000","last_change":"2024-03-26T03:16:56.908716+0000","last_active":"2024-03-26T03:19:39.595251+0000","last_peered":"2024-03-26T03:19:39.595251+0000","last_clean":"2024-03-26T03:19:39.595251+0000","last_became_active":"2024-03-26T03:16:56.908005+0000","last_became_peered":"2024-03-26T03:16:56.908005+0000","last_unstale":"2024-03-26T03:19:39.595251+0000","last_undegraded":"2024-03-26T03:19:39.595251+0000","last_fullsized":"2024-03-26T03:19:39.595251+0000","mapping_epoch":403,"log_start":"446'10400","ondisk_log_start":"446'10400","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3174,"log_dups_size":2487,"ondisk_log_size":3174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203030528,"num_objects":3098,"num_object_clones":0,"num_object_copies":9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13574,"num_write_kb":533504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":762,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599185+0000","last_change":"2024-03-26T03:18:54.811073+0000","last_active":"2024-03-26T03:19:39.599185+0000","last_peered":"2024-03-26T03:19:39.599185+0000","last_clean":"2024-03-26T03:19:39.599185+0000","last_became_active":"2024-03-26T03:16:50.206619+0000","last_became_peered":"2024-03-26T03:16:50.206619+0000","last_unstale":"2024-03-26T03:19:39.599185+0000","last_undegraded":"2024-03-26T03:19:39.599185+0000","last_fullsized":"2024-03-26T03:19:39.599185+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:54.810939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:18:54.810939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:57.976578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"446'27","reported_seq":943,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.607587+0000","last_change":"2024-03-26T03:14:37.294831+0000","last_active":"2024-03-26T03:19:39.607587+0000","last_peered":"2024-03-26T03:19:39.607587+0000","last_clean":"2024-03-26T03:19:39.607587+0000","last_became_active":"2024-03-26T03:14:37.272426+0000","last_became_peered":"2024-03-26T03:14:37.272426+0000","last_unstale":"2024-03-26T03:19:39.607587+0000","last_undegraded":"2024-03-26T03:19:39.607587+0000","last_fullsized":"2024-03-26T03:19:39.607587+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'23","last_scrub_stamp":"2024-03-26T03:13:23.013434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:13:23.013434+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094056+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":703,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595240+0000","last_change":"2024-03-26T03:16:50.211076+0000","last_active":"2024-03-26T03:19:39.595240+0000","last_peered":"2024-03-26T03:19:39.595240+0000","last_clean":"2024-03-26T03:19:39.595240+0000","last_became_active":"2024-03-26T03:16:50.210676+0000","last_became_peered":"2024-03-26T03:16:50.210676+0000","last_unstale":"2024-03-26T03:19:39.595240+0000","last_undegraded":"2024-03-26T03:19:39.595240+0000","last_fullsized":"2024-03-26T03:19:39.595240+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:41.390700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:41.390700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588368+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2107,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601532+0000","last_change":"2024-03-26T03:13:31.075605+0000","last_active":"2024-03-26T03:19:39.601532+0000","last_peered":"2024-03-26T03:19:39.601532+0000","last_clean":"2024-03-26T03:19:39.601532+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:19:39.601532+0000","last_undegraded":"2024-03-26T03:19:39.601532+0000","last_fullsized":"2024-03-26T03:19:39.601532+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:13:31.074419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:31.074419+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094027+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1232,"num_read_kb":1227,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":820,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596754+0000","last_change":"2024-03-26T03:18:55.818417+0000","last_active":"2024-03-26T03:19:39.596754+0000","last_peered":"2024-03-26T03:19:39.596754+0000","last_clean":"2024-03-26T03:19:39.596754+0000","last_became_active":"2024-03-26T03:14:37.373684+0000","last_became_peered":"2024-03-26T03:14:37.373684+0000","last_unstale":"2024-03-26T03:19:39.596754+0000","last_undegraded":"2024-03-26T03:19:39.596754+0000","last_fullsized":"2024-03-26T03:19:39.596754+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:55.818297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:55.818297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:21.616424+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":904,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593621+0000","last_change":"2024-03-26T03:17:59.594483+0000","last_active":"2024-03-26T03:19:39.593621+0000","last_peered":"2024-03-26T03:19:39.593621+0000","last_clean":"2024-03-26T03:19:39.593621+0000","last_became_active":"2024-03-26T03:17:59.594123+0000","last_became_peered":"2024-03-26T03:17:59.594123+0000","last_unstale":"2024-03-26T03:19:39.593621+0000","last_undegraded":"2024-03-26T03:19:39.593621+0000","last_fullsized":"2024-03-26T03:19:39.593621+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:50.701574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.701574+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307304+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.b","version":"454'13326","reported_seq":27058,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.620949+0000","last_change":"2024-03-26T03:18:49.658787+0000","last_active":"2024-03-26T03:19:39.620949+0000","last_peered":"2024-03-26T03:19:39.620949+0000","last_clean":"2024-03-26T03:19:39.620949+0000","last_became_active":"2024-03-26T03:18:44.514105+0000","last_became_peered":"2024-03-26T03:18:44.514105+0000","last_unstale":"2024-03-26T03:19:39.620949+0000","last_undegraded":"2024-03-26T03:19:39.620949+0000","last_fullsized":"2024-03-26T03:19:39.620949+0000","mapping_epoch":441,"log_start":"446'10100","ondisk_log_start":"446'10100","created":403,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'8981","last_scrub_stamp":"2024-03-26T03:18:49.658668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.658668+0000","objects_scrubbed":7499,"log_size":3226,"log_dups_size":2387,"ondisk_log_size":3226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:04.967991+0000","scrub_duration":3070,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199884800,"num_objects":3050,"num_object_clones":0,"num_object_copies":9150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13326,"num_write_kb":524032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":443023360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":306,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599747+0000","last_change":"2024-03-26T03:17:56.570342+0000","last_active":"2024-03-26T03:19:39.599747+0000","last_peered":"2024-03-26T03:19:39.599747+0000","last_clean":"2024-03-26T03:19:39.599747+0000","last_became_active":"2024-03-26T03:17:56.570006+0000","last_became_peered":"2024-03-26T03:17:56.570006+0000","last_unstale":"2024-03-26T03:19:39.599747+0000","last_undegraded":"2024-03-26T03:19:39.599747+0000","last_fullsized":"2024-03-26T03:19:39.599747+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:17.442291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:13:17.442291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"446'71","reported_seq":1209,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.613612+0000","last_change":"2024-03-26T03:15:12.789448+0000","last_active":"2024-03-26T03:19:39.613612+0000","last_peered":"2024-03-26T03:19:39.613612+0000","last_clean":"2024-03-26T03:19:39.613612+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:19:39.613612+0000","last_undegraded":"2024-03-26T03:19:39.613612+0000","last_fullsized":"2024-03-26T03:19:39.613612+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'59","last_scrub_stamp":"2024-03-26T03:15:12.789148+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:15:12.789148+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":657,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.607844+0000","last_change":"2024-03-26T03:16:50.962056+0000","last_active":"2024-03-26T03:19:39.607844+0000","last_peered":"2024-03-26T03:19:39.607844+0000","last_clean":"2024-03-26T03:19:39.607844+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:19:39.607844+0000","last_undegraded":"2024-03-26T03:19:39.607844+0000","last_fullsized":"2024-03-26T03:19:39.607844+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:50.961975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.961975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":555,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596653+0000","last_change":"2024-03-26T03:19:12.537772+0000","last_active":"2024-03-26T03:19:39.596653+0000","last_peered":"2024-03-26T03:19:39.596653+0000","last_clean":"2024-03-26T03:19:39.596653+0000","last_became_active":"2024-03-26T03:14:37.274135+0000","last_became_peered":"2024-03-26T03:14:37.274135+0000","last_unstale":"2024-03-26T03:19:39.596653+0000","last_undegraded":"2024-03-26T03:19:39.596653+0000","last_fullsized":"2024-03-26T03:19:39.596653+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_clean_scrub_stamp":"2024-03-26T03:19:12.537519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:40.732976+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":581,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592669+0000","last_change":"2024-03-26T03:16:30.089444+0000","last_active":"2024-03-26T03:19:39.592669+0000","last_peered":"2024-03-26T03:19:39.592669+0000","last_clean":"2024-03-26T03:19:39.592669+0000","last_became_active":"2024-03-26T03:13:56.449984+0000","last_became_peered":"2024-03-26T03:13:56.449984+0000","last_unstale":"2024-03-26T03:19:39.592669+0000","last_undegraded":"2024-03-26T03:19:39.592669+0000","last_fullsized":"2024-03-26T03:19:39.592669+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:30.089341+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:16:30.089341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.d","version":"454'13497","reported_seq":13611,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.738320+0000","last_change":"2024-03-26T03:16:56.901084+0000","last_active":"2024-03-26T03:19:39.738320+0000","last_peered":"2024-03-26T03:19:39.738320+0000","last_clean":"2024-03-26T03:19:39.738320+0000","last_became_active":"2024-03-26T03:16:56.900700+0000","last_became_peered":"2024-03-26T03:16:56.900700+0000","last_unstale":"2024-03-26T03:19:39.738320+0000","last_undegraded":"2024-03-26T03:19:39.738320+0000","last_fullsized":"2024-03-26T03:19:39.738320+0000","mapping_epoch":403,"log_start":"445'9991","ondisk_log_start":"445'9991","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211550232,"num_objects":3229,"num_object_clones":0,"num_object_copies":9687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13497,"num_write_kb":535169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"333'12","reported_seq":413,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597236+0000","last_change":"2024-03-26T03:17:42.576855+0000","last_active":"2024-03-26T03:19:39.597236+0000","last_peered":"2024-03-26T03:19:39.597236+0000","last_clean":"2024-03-26T03:19:39.597236+0000","last_became_active":"2024-03-26T03:14:22.574243+0000","last_became_peered":"2024-03-26T03:14:22.574243+0000","last_unstale":"2024-03-26T03:19:39.597236+0000","last_undegraded":"2024-03-26T03:19:39.597236+0000","last_fullsized":"2024-03-26T03:19:39.597236+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'12","last_scrub_stamp":"2024-03-26T03:17:42.576771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.576771+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"446'42","reported_seq":958,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608162+0000","last_change":"2024-03-26T03:17:56.581987+0000","last_active":"2024-03-26T03:19:39.608162+0000","last_peered":"2024-03-26T03:19:39.608162+0000","last_clean":"2024-03-26T03:19:39.608162+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:19:39.608162+0000","last_undegraded":"2024-03-26T03:19:39.608162+0000","last_fullsized":"2024-03-26T03:19:39.608162+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:16:40.828476+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.828476+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":868,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.613740+0000","last_change":"2024-03-26T03:14:37.252086+0000","last_active":"2024-03-26T03:19:39.613740+0000","last_peered":"2024-03-26T03:19:39.613740+0000","last_clean":"2024-03-26T03:19:39.613740+0000","last_became_active":"2024-03-26T03:14:37.251746+0000","last_became_peered":"2024-03-26T03:14:37.251746+0000","last_unstale":"2024-03-26T03:19:39.613740+0000","last_undegraded":"2024-03-26T03:19:39.613740+0000","last_fullsized":"2024-03-26T03:19:39.613740+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:19.981413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:13:19.981413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093857+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":266,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595142+0000","last_change":"2024-03-26T03:18:49.512742+0000","last_active":"2024-03-26T03:19:39.595142+0000","last_peered":"2024-03-26T03:19:39.595142+0000","last_clean":"2024-03-26T03:19:39.595142+0000","last_became_active":"2024-03-26T03:14:22.566823+0000","last_became_peered":"2024-03-26T03:14:22.566823+0000","last_unstale":"2024-03-26T03:19:39.595142+0000","last_undegraded":"2024-03-26T03:19:39.595142+0000","last_fullsized":"2024-03-26T03:19:39.595142+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:49.512502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.512502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:52.569595+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":941,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597520+0000","last_change":"2024-03-26T03:19:06.555275+0000","last_active":"2024-03-26T03:19:39.597520+0000","last_peered":"2024-03-26T03:19:39.597520+0000","last_clean":"2024-03-26T03:19:39.597520+0000","last_became_active":"2024-03-26T03:14:40.178828+0000","last_became_peered":"2024-03-26T03:14:40.178828+0000","last_unstale":"2024-03-26T03:19:39.597520+0000","last_undegraded":"2024-03-26T03:19:39.597520+0000","last_fullsized":"2024-03-26T03:19:39.597520+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:06.555052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:06.555052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:23.198309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"454'13765","reported_seq":13876,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:40.703614+0000","last_change":"2024-03-26T03:16:56.899075+0000","last_active":"2024-03-26T03:19:40.703614+0000","last_peered":"2024-03-26T03:19:40.703614+0000","last_clean":"2024-03-26T03:19:40.703614+0000","last_became_active":"2024-03-26T03:16:56.898139+0000","last_became_peered":"2024-03-26T03:16:56.898139+0000","last_unstale":"2024-03-26T03:19:40.703614+0000","last_undegraded":"2024-03-26T03:19:40.703614+0000","last_fullsized":"2024-03-26T03:19:40.703614+0000","mapping_epoch":403,"log_start":"446'10500","ondisk_log_start":"446'10500","created":403,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:55.883571+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.883571+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.103459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203358208,"num_objects":3103,"num_object_clones":0,"num_object_copies":9309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13765,"num_write_kb":539776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":658,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594723+0000","last_change":"2024-03-26T03:17:59.595756+0000","last_active":"2024-03-26T03:19:39.594723+0000","last_peered":"2024-03-26T03:19:39.594723+0000","last_clean":"2024-03-26T03:19:39.594723+0000","last_became_active":"2024-03-26T03:17:59.594586+0000","last_became_peered":"2024-03-26T03:17:59.594586+0000","last_unstale":"2024-03-26T03:19:39.594723+0000","last_undegraded":"2024-03-26T03:19:39.594723+0000","last_fullsized":"2024-03-26T03:19:39.594723+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:16.474559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:16.474559+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.587923+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":705,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.600753+0000","last_change":"2024-03-26T03:17:53.556662+0000","last_active":"2024-03-26T03:19:39.600753+0000","last_peered":"2024-03-26T03:19:39.600753+0000","last_clean":"2024-03-26T03:19:39.600753+0000","last_became_active":"2024-03-26T03:16:50.208194+0000","last_became_peered":"2024-03-26T03:16:50.208194+0000","last_unstale":"2024-03-26T03:19:39.600753+0000","last_undegraded":"2024-03-26T03:19:39.600753+0000","last_fullsized":"2024-03-26T03:19:39.600753+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:53.556429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:53.556429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105362+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"446'36","reported_seq":923,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592890+0000","last_change":"2024-03-26T03:14:37.290458+0000","last_active":"2024-03-26T03:19:39.592890+0000","last_peered":"2024-03-26T03:19:39.592890+0000","last_clean":"2024-03-26T03:19:39.592890+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:19:39.592890+0000","last_undegraded":"2024-03-26T03:19:39.592890+0000","last_fullsized":"2024-03-26T03:19:39.592890+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:14:37.290313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:14:37.290313+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304910+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":922,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598524+0000","last_change":"2024-03-26T03:17:37.107175+0000","last_active":"2024-03-26T03:19:39.598524+0000","last_peered":"2024-03-26T03:19:39.598524+0000","last_clean":"2024-03-26T03:19:39.598524+0000","last_became_active":"2024-03-26T03:14:37.249021+0000","last_became_peered":"2024-03-26T03:14:37.249021+0000","last_unstale":"2024-03-26T03:19:39.598524+0000","last_undegraded":"2024-03-26T03:19:39.598524+0000","last_fullsized":"2024-03-26T03:19:39.598524+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:17:37.106951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:37.106951+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":527,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608425+0000","last_change":"2024-03-26T03:19:00.815058+0000","last_active":"2024-03-26T03:19:39.608425+0000","last_peered":"2024-03-26T03:19:39.608425+0000","last_clean":"2024-03-26T03:19:39.608425+0000","last_became_active":"2024-03-26T03:14:37.265041+0000","last_became_peered":"2024-03-26T03:14:37.265041+0000","last_unstale":"2024-03-26T03:19:39.608425+0000","last_undegraded":"2024-03-26T03:19:39.608425+0000","last_fullsized":"2024-03-26T03:19:39.608425+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:00.814950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:00.814950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:01.474562+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":980,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.602510+0000","last_change":"2024-03-26T03:17:59.607666+0000","last_active":"2024-03-26T03:19:39.602510+0000","last_peered":"2024-03-26T03:19:39.602510+0000","last_clean":"2024-03-26T03:19:39.602510+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:19:39.602510+0000","last_undegraded":"2024-03-26T03:19:39.602510+0000","last_fullsized":"2024-03-26T03:19:39.602510+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:44.879010+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.879010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"446'33","reported_seq":361,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596465+0000","last_change":"2024-03-26T03:19:07.549173+0000","last_active":"2024-03-26T03:19:39.596465+0000","last_peered":"2024-03-26T03:19:39.596465+0000","last_clean":"2024-03-26T03:19:39.596465+0000","last_became_active":"2024-03-26T03:14:37.269153+0000","last_became_peered":"2024-03-26T03:14:37.269153+0000","last_unstale":"2024-03-26T03:19:39.596465+0000","last_undegraded":"2024-03-26T03:19:39.596465+0000","last_fullsized":"2024-03-26T03:19:39.596465+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'33","last_scrub_stamp":"2024-03-26T03:19:07.549012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:07.549012+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:13.033562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":137,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601360+0000","last_change":"2024-03-26T03:17:56.578376+0000","last_active":"2024-03-26T03:19:39.601360+0000","last_peered":"2024-03-26T03:19:39.601360+0000","last_clean":"2024-03-26T03:19:39.601360+0000","last_became_active":"2024-03-26T03:17:56.571085+0000","last_became_peered":"2024-03-26T03:17:56.571085+0000","last_unstale":"2024-03-26T03:19:39.601360+0000","last_undegraded":"2024-03-26T03:19:39.601360+0000","last_fullsized":"2024-03-26T03:19:39.601360+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:16:49.187353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.187353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.103508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":547,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.603941+0000","last_change":"2024-03-26T03:19:01.660770+0000","last_active":"2024-03-26T03:19:39.603941+0000","last_peered":"2024-03-26T03:19:39.603941+0000","last_clean":"2024-03-26T03:19:39.603941+0000","last_became_active":"2024-03-26T03:14:37.273658+0000","last_became_peered":"2024-03-26T03:14:37.273658+0000","last_unstale":"2024-03-26T03:19:39.603941+0000","last_undegraded":"2024-03-26T03:19:39.603941+0000","last_fullsized":"2024-03-26T03:19:39.603941+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:01.660326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:01.660326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:14.456341+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":749,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594552+0000","last_change":"2024-03-26T03:17:55.099141+0000","last_active":"2024-03-26T03:19:39.594552+0000","last_peered":"2024-03-26T03:19:39.594552+0000","last_clean":"2024-03-26T03:19:39.594552+0000","last_became_active":"2024-03-26T03:16:50.205905+0000","last_became_peered":"2024-03-26T03:16:50.205905+0000","last_unstale":"2024-03-26T03:19:39.594552+0000","last_undegraded":"2024-03-26T03:19:39.594552+0000","last_fullsized":"2024-03-26T03:19:39.594552+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:55.099060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.099060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":375,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593524+0000","last_change":"2024-03-26T03:17:45.079338+0000","last_active":"2024-03-26T03:19:39.593524+0000","last_peered":"2024-03-26T03:19:39.593524+0000","last_clean":"2024-03-26T03:19:39.593524+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:19:39.593524+0000","last_undegraded":"2024-03-26T03:19:39.593524+0000","last_fullsized":"2024-03-26T03:19:39.593524+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:45.079209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.079209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"446'24","reported_seq":621,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.603047+0000","last_change":"2024-03-26T03:16:40.927377+0000","last_active":"2024-03-26T03:19:39.603047+0000","last_peered":"2024-03-26T03:19:39.603047+0000","last_clean":"2024-03-26T03:19:39.603047+0000","last_became_active":"2024-03-26T03:14:37.269408+0000","last_became_peered":"2024-03-26T03:14:37.269408+0000","last_unstale":"2024-03-26T03:19:39.603047+0000","last_undegraded":"2024-03-26T03:19:39.603047+0000","last_fullsized":"2024-03-26T03:19:39.603047+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'18","last_scrub_stamp":"2024-03-26T03:16:40.927260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.927260+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":875,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597103+0000","last_change":"2024-03-26T03:17:56.570470+0000","last_active":"2024-03-26T03:19:39.597103+0000","last_peered":"2024-03-26T03:19:39.597103+0000","last_clean":"2024-03-26T03:19:39.597103+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:19:39.597103+0000","last_undegraded":"2024-03-26T03:19:39.597103+0000","last_fullsized":"2024-03-26T03:19:39.597103+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:59.707108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.707108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":847,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598881+0000","last_change":"2024-03-26T03:19:05.801891+0000","last_active":"2024-03-26T03:19:39.598881+0000","last_peered":"2024-03-26T03:19:39.598881+0000","last_clean":"2024-03-26T03:19:39.598881+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:19:39.598881+0000","last_undegraded":"2024-03-26T03:19:39.598881+0000","last_fullsized":"2024-03-26T03:19:39.598881+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:05.801789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:05.801789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:07.979149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":254,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596799+0000","last_change":"2024-03-26T03:19:00.728008+0000","last_active":"2024-03-26T03:19:39.596799+0000","last_peered":"2024-03-26T03:19:39.596799+0000","last_clean":"2024-03-26T03:19:39.596799+0000","last_became_active":"2024-03-26T03:14:37.333430+0000","last_became_peered":"2024-03-26T03:14:37.333430+0000","last_unstale":"2024-03-26T03:19:39.596799+0000","last_undegraded":"2024-03-26T03:19:39.596799+0000","last_fullsized":"2024-03-26T03:19:39.596799+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:00.727659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:00.727659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:09.168697+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":624,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597861+0000","last_change":"2024-03-26T03:19:16.735157+0000","last_active":"2024-03-26T03:19:39.597861+0000","last_peered":"2024-03-26T03:19:39.597861+0000","last_clean":"2024-03-26T03:19:39.597861+0000","last_became_active":"2024-03-26T03:14:24.567937+0000","last_became_peered":"2024-03-26T03:14:24.567937+0000","last_unstale":"2024-03-26T03:19:39.597861+0000","last_undegraded":"2024-03-26T03:19:39.597861+0000","last_fullsized":"2024-03-26T03:19:39.597861+0000","mapping_epoch":334,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:16.735031+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:19:16.735031+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:46.708237+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"446'49","reported_seq":644,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594922+0000","last_change":"2024-03-26T03:16:49.904794+0000","last_active":"2024-03-26T03:19:39.594922+0000","last_peered":"2024-03-26T03:19:39.594922+0000","last_clean":"2024-03-26T03:19:39.594922+0000","last_became_active":"2024-03-26T03:14:37.265466+0000","last_became_peered":"2024-03-26T03:14:37.265466+0000","last_unstale":"2024-03-26T03:19:39.594922+0000","last_undegraded":"2024-03-26T03:19:39.594922+0000","last_fullsized":"2024-03-26T03:19:39.594922+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.904637+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":881,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598120+0000","last_change":"2024-03-26T03:19:11.775706+0000","last_active":"2024-03-26T03:19:39.598120+0000","last_peered":"2024-03-26T03:19:39.598120+0000","last_clean":"2024-03-26T03:19:39.598120+0000","last_became_active":"2024-03-26T03:17:56.569537+0000","last_became_peered":"2024-03-26T03:17:56.569537+0000","last_unstale":"2024-03-26T03:19:39.598120+0000","last_undegraded":"2024-03-26T03:19:39.598120+0000","last_fullsized":"2024-03-26T03:19:39.598120+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:11.775591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:19:11.775591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:33.687560+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"333'13","reported_seq":1144,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599390+0000","last_change":"2024-03-26T03:18:09.760752+0000","last_active":"2024-03-26T03:19:39.599390+0000","last_peered":"2024-03-26T03:19:39.599390+0000","last_clean":"2024-03-26T03:19:39.599390+0000","last_became_active":"2024-03-26T03:14:22.577229+0000","last_became_peered":"2024-03-26T03:14:22.577229+0000","last_unstale":"2024-03-26T03:19:39.599390+0000","last_undegraded":"2024-03-26T03:19:39.599390+0000","last_fullsized":"2024-03-26T03:19:39.599390+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'13","last_scrub_stamp":"2024-03-26T03:18:09.760609+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:18:09.760609+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":584,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596566+0000","last_change":"2024-03-26T03:18:59.591864+0000","last_active":"2024-03-26T03:19:39.596566+0000","last_peered":"2024-03-26T03:19:39.596566+0000","last_clean":"2024-03-26T03:19:39.596566+0000","last_became_active":"2024-03-26T03:14:37.273988+0000","last_became_peered":"2024-03-26T03:14:37.273988+0000","last_unstale":"2024-03-26T03:19:39.596566+0000","last_undegraded":"2024-03-26T03:19:39.596566+0000","last_fullsized":"2024-03-26T03:19:39.596566+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:59.591751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:59.591751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:15.424958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":549,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595117+0000","last_change":"2024-03-26T03:18:54.739935+0000","last_active":"2024-03-26T03:19:39.595117+0000","last_peered":"2024-03-26T03:19:39.595117+0000","last_clean":"2024-03-26T03:19:39.595117+0000","last_became_active":"2024-03-26T03:14:40.173243+0000","last_became_peered":"2024-03-26T03:14:40.173243+0000","last_unstale":"2024-03-26T03:19:39.595117+0000","last_undegraded":"2024-03-26T03:19:39.595117+0000","last_fullsized":"2024-03-26T03:19:39.595117+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:54.739688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:18:54.739688+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:23.231065+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"446'27","reported_seq":686,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598898+0000","last_change":"2024-03-26T03:16:59.931429+0000","last_active":"2024-03-26T03:19:39.598898+0000","last_peered":"2024-03-26T03:19:39.598898+0000","last_clean":"2024-03-26T03:19:39.598898+0000","last_became_active":"2024-03-26T03:14:37.333426+0000","last_became_peered":"2024-03-26T03:14:37.333426+0000","last_unstale":"2024-03-26T03:19:39.598898+0000","last_undegraded":"2024-03-26T03:19:39.598898+0000","last_fullsized":"2024-03-26T03:19:39.598898+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'21","last_scrub_stamp":"2024-03-26T03:16:59.931283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.931283+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":609,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597236+0000","last_change":"2024-03-26T03:16:29.133272+0000","last_active":"2024-03-26T03:19:39.597236+0000","last_peered":"2024-03-26T03:19:39.597236+0000","last_clean":"2024-03-26T03:19:39.597236+0000","last_became_active":"2024-03-26T03:14:37.262766+0000","last_became_peered":"2024-03-26T03:14:37.262766+0000","last_unstale":"2024-03-26T03:19:39.597236+0000","last_undegraded":"2024-03-26T03:19:39.597236+0000","last_fullsized":"2024-03-26T03:19:39.597236+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:29.133174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.133174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"342'11","reported_seq":651,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595201+0000","last_change":"2024-03-26T03:17:50.265634+0000","last_active":"2024-03-26T03:19:39.595201+0000","last_peered":"2024-03-26T03:19:39.595201+0000","last_clean":"2024-03-26T03:19:39.595201+0000","last_became_active":"2024-03-26T03:14:37.264759+0000","last_became_peered":"2024-03-26T03:14:37.264759+0000","last_unstale":"2024-03-26T03:19:39.595201+0000","last_undegraded":"2024-03-26T03:19:39.595201+0000","last_fullsized":"2024-03-26T03:19:39.595201+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'11","last_scrub_stamp":"2024-03-26T03:17:50.265404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:50.265404+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592337+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598768+0000","last_change":"2024-03-26T03:17:48.227168+0000","last_active":"2024-03-26T03:19:39.598768+0000","last_peered":"2024-03-26T03:19:39.598768+0000","last_clean":"2024-03-26T03:19:39.598768+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:19:39.598768+0000","last_undegraded":"2024-03-26T03:19:39.598768+0000","last_fullsized":"2024-03-26T03:19:39.598768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:17:48.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.226947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307635+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":953,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599395+0000","last_change":"2024-03-26T03:17:55.577343+0000","last_active":"2024-03-26T03:19:39.599395+0000","last_peered":"2024-03-26T03:19:39.599395+0000","last_clean":"2024-03-26T03:19:39.599395+0000","last_became_active":"2024-03-26T03:14:40.170537+0000","last_became_peered":"2024-03-26T03:14:40.170537+0000","last_unstale":"2024-03-26T03:19:39.599395+0000","last_undegraded":"2024-03-26T03:19:39.599395+0000","last_fullsized":"2024-03-26T03:19:39.599395+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:55.577263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.577263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"446'25","reported_seq":889,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608077+0000","last_change":"2024-03-26T03:13:33.092136+0000","last_active":"2024-03-26T03:19:39.608077+0000","last_peered":"2024-03-26T03:19:39.608077+0000","last_clean":"2024-03-26T03:19:39.608077+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:19:39.608077+0000","last_undegraded":"2024-03-26T03:19:39.608077+0000","last_fullsized":"2024-03-26T03:19:39.608077+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-26T03:13:33.091975+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:13:33.091975+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093774+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":840,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599092+0000","last_change":"2024-03-26T03:16:33.748393+0000","last_active":"2024-03-26T03:19:39.599092+0000","last_peered":"2024-03-26T03:19:39.599092+0000","last_clean":"2024-03-26T03:19:39.599092+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:19:39.599092+0000","last_undegraded":"2024-03-26T03:19:39.599092+0000","last_fullsized":"2024-03-26T03:19:39.599092+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:33.748295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:33.748295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":530,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608241+0000","last_change":"2024-03-26T03:16:44.909663+0000","last_active":"2024-03-26T03:19:39.608241+0000","last_peered":"2024-03-26T03:19:39.608241+0000","last_clean":"2024-03-26T03:19:39.608241+0000","last_became_active":"2024-03-26T03:14:22.573232+0000","last_became_peered":"2024-03-26T03:14:22.573232+0000","last_unstale":"2024-03-26T03:19:39.608241+0000","last_undegraded":"2024-03-26T03:19:39.608241+0000","last_fullsized":"2024-03-26T03:19:39.608241+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.909523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.909523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":303,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.600845+0000","last_change":"2024-03-26T03:17:35.405461+0000","last_active":"2024-03-26T03:19:39.600845+0000","last_peered":"2024-03-26T03:19:39.600845+0000","last_clean":"2024-03-26T03:19:39.600845+0000","last_became_active":"2024-03-26T03:13:35.278675+0000","last_became_peered":"2024-03-26T03:13:35.278675+0000","last_unstale":"2024-03-26T03:19:39.600845+0000","last_undegraded":"2024-03-26T03:19:39.600845+0000","last_fullsized":"2024-03-26T03:19:39.600845+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.405139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.405139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":621,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601191+0000","last_change":"2024-03-26T03:14:22.578069+0000","last_active":"2024-03-26T03:19:39.601191+0000","last_peered":"2024-03-26T03:19:39.601191+0000","last_clean":"2024-03-26T03:19:39.601191+0000","last_became_active":"2024-03-26T03:14:22.577614+0000","last_became_peered":"2024-03-26T03:14:22.577614+0000","last_unstale":"2024-03-26T03:19:39.601191+0000","last_undegraded":"2024-03-26T03:19:39.601191+0000","last_fullsized":"2024-03-26T03:19:39.601191+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.940687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.940687+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093997+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"446'27","reported_seq":892,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596127+0000","last_change":"2024-03-26T03:16:53.170995+0000","last_active":"2024-03-26T03:19:39.596127+0000","last_peered":"2024-03-26T03:19:39.596127+0000","last_clean":"2024-03-26T03:19:39.596127+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:19:39.596127+0000","last_undegraded":"2024-03-26T03:19:39.596127+0000","last_fullsized":"2024-03-26T03:19:39.596127+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:16:53.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:53.170849+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":877,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599900+0000","last_change":"2024-03-26T03:17:34.417755+0000","last_active":"2024-03-26T03:19:39.599900+0000","last_peered":"2024-03-26T03:19:39.599900+0000","last_clean":"2024-03-26T03:19:39.599900+0000","last_became_active":"2024-03-26T03:13:35.274830+0000","last_became_peered":"2024-03-26T03:13:35.274830+0000","last_unstale":"2024-03-26T03:19:39.599900+0000","last_undegraded":"2024-03-26T03:19:39.599900+0000","last_fullsized":"2024-03-26T03:19:39.599900+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.417610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.417610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105270+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":824,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608245+0000","last_change":"2024-03-26T03:15:14.862264+0000","last_active":"2024-03-26T03:19:39.608245+0000","last_peered":"2024-03-26T03:19:39.608245+0000","last_clean":"2024-03-26T03:19:39.608245+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:19:39.608245+0000","last_undegraded":"2024-03-26T03:19:39.608245+0000","last_fullsized":"2024-03-26T03:19:39.608245+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:14.862148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:14.862148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094318+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":534,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595304+0000","last_change":"2024-03-26T03:14:22.577571+0000","last_active":"2024-03-26T03:19:39.595304+0000","last_peered":"2024-03-26T03:19:39.595304+0000","last_clean":"2024-03-26T03:19:39.595304+0000","last_became_active":"2024-03-26T03:14:22.577265+0000","last_became_peered":"2024-03-26T03:14:22.577265+0000","last_unstale":"2024-03-26T03:19:39.595304+0000","last_undegraded":"2024-03-26T03:19:39.595304+0000","last_fullsized":"2024-03-26T03:19:39.595304+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:19.903253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:14:19.903253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.600843+0000","last_change":"2024-03-26T03:14:40.950126+0000","last_active":"2024-03-26T03:19:39.600843+0000","last_peered":"2024-03-26T03:19:39.600843+0000","last_clean":"2024-03-26T03:19:39.600843+0000","last_became_active":"2024-03-26T03:14:40.166745+0000","last_became_peered":"2024-03-26T03:14:40.166745+0000","last_unstale":"2024-03-26T03:19:39.600843+0000","last_undegraded":"2024-03-26T03:19:39.600843+0000","last_fullsized":"2024-03-26T03:19:39.600843+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:14:40.950003+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:14:40.950003+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"446'12","reported_seq":557,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595252+0000","last_change":"2024-03-26T03:16:34.843257+0000","last_active":"2024-03-26T03:19:39.595252+0000","last_peered":"2024-03-26T03:19:39.595252+0000","last_clean":"2024-03-26T03:19:39.595252+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:19:39.595252+0000","last_undegraded":"2024-03-26T03:19:39.595252+0000","last_fullsized":"2024-03-26T03:19:39.595252+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'8","last_scrub_stamp":"2024-03-26T03:16:34.843159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:34.843159+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588341+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":886,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599094+0000","last_change":"2024-03-26T03:19:18.786789+0000","last_active":"2024-03-26T03:19:39.599094+0000","last_peered":"2024-03-26T03:19:39.599094+0000","last_clean":"2024-03-26T03:19:39.599094+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:19:39.599094+0000","last_undegraded":"2024-03-26T03:19:39.599094+0000","last_fullsized":"2024-03-26T03:19:39.599094+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:18.786671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:19:18.786671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.252575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":881,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597017+0000","last_change":"2024-03-26T03:17:57.629803+0000","last_active":"2024-03-26T03:19:39.597017+0000","last_peered":"2024-03-26T03:19:39.597017+0000","last_clean":"2024-03-26T03:19:39.597017+0000","last_became_active":"2024-03-26T03:17:56.569576+0000","last_became_peered":"2024-03-26T03:17:56.569576+0000","last_unstale":"2024-03-26T03:19:39.597017+0000","last_undegraded":"2024-03-26T03:19:39.597017+0000","last_fullsized":"2024-03-26T03:19:39.597017+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.629637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.629637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":246,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593708+0000","last_change":"2024-03-26T03:16:45.148252+0000","last_active":"2024-03-26T03:19:39.593708+0000","last_peered":"2024-03-26T03:19:39.593708+0000","last_clean":"2024-03-26T03:19:39.593708+0000","last_became_active":"2024-03-26T03:14:37.260920+0000","last_became_peered":"2024-03-26T03:14:37.260920+0000","last_unstale":"2024-03-26T03:19:39.593708+0000","last_undegraded":"2024-03-26T03:19:39.593708+0000","last_fullsized":"2024-03-26T03:19:39.593708+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.148129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.148129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":617,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594448+0000","last_change":"2024-03-26T03:17:59.621631+0000","last_active":"2024-03-26T03:19:39.594448+0000","last_peered":"2024-03-26T03:19:39.594448+0000","last_clean":"2024-03-26T03:19:39.594448+0000","last_became_active":"2024-03-26T03:17:59.621344+0000","last_became_peered":"2024-03-26T03:17:59.621344+0000","last_unstale":"2024-03-26T03:19:39.594448+0000","last_undegraded":"2024-03-26T03:19:39.594448+0000","last_fullsized":"2024-03-26T03:19:39.594448+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:45.232258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.232258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"446'39","reported_seq":626,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595389+0000","last_change":"2024-03-26T03:16:56.991143+0000","last_active":"2024-03-26T03:19:39.595389+0000","last_peered":"2024-03-26T03:19:39.595389+0000","last_clean":"2024-03-26T03:19:39.595389+0000","last_became_active":"2024-03-26T03:14:37.273380+0000","last_became_peered":"2024-03-26T03:14:37.273380+0000","last_unstale":"2024-03-26T03:19:39.595389+0000","last_undegraded":"2024-03-26T03:19:39.595389+0000","last_fullsized":"2024-03-26T03:19:39.595389+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'31","last_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.991031+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":863,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597618+0000","last_change":"2024-03-26T03:17:58.647438+0000","last_active":"2024-03-26T03:19:39.597618+0000","last_peered":"2024-03-26T03:19:39.597618+0000","last_clean":"2024-03-26T03:19:39.597618+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:19:39.597618+0000","last_undegraded":"2024-03-26T03:19:39.597618+0000","last_fullsized":"2024-03-26T03:19:39.597618+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:58.647326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:58.647326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":249,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592296+0000","last_change":"2024-03-26T03:17:32.002736+0000","last_active":"2024-03-26T03:19:39.592296+0000","last_peered":"2024-03-26T03:19:39.592296+0000","last_clean":"2024-03-26T03:19:39.592296+0000","last_became_active":"2024-03-26T03:14:37.249377+0000","last_became_peered":"2024-03-26T03:14:37.249377+0000","last_unstale":"2024-03-26T03:19:39.592296+0000","last_undegraded":"2024-03-26T03:19:39.592296+0000","last_fullsized":"2024-03-26T03:19:39.592296+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.002635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.002635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":528,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608063+0000","last_change":"2024-03-26T03:19:04.682708+0000","last_active":"2024-03-26T03:19:39.608063+0000","last_peered":"2024-03-26T03:19:39.608063+0000","last_clean":"2024-03-26T03:19:39.608063+0000","last_became_active":"2024-03-26T03:14:22.603198+0000","last_became_peered":"2024-03-26T03:14:22.603198+0000","last_unstale":"2024-03-26T03:19:39.608063+0000","last_undegraded":"2024-03-26T03:19:39.608063+0000","last_fullsized":"2024-03-26T03:19:39.608063+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:04.682583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:04.682583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.009193+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":864,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598166+0000","last_change":"2024-03-26T03:18:52.795890+0000","last_active":"2024-03-26T03:19:39.598166+0000","last_peered":"2024-03-26T03:19:39.598166+0000","last_clean":"2024-03-26T03:19:39.598166+0000","last_became_active":"2024-03-26T03:13:38.309830+0000","last_became_peered":"2024-03-26T03:13:38.309830+0000","last_unstale":"2024-03-26T03:19:39.598166+0000","last_undegraded":"2024-03-26T03:19:39.598166+0000","last_fullsized":"2024-03-26T03:19:39.598166+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:52.795807+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:18:52.795807+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:21.039399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"446'36","reported_seq":748,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596365+0000","last_change":"2024-03-26T03:19:11.558002+0000","last_active":"2024-03-26T03:19:39.596365+0000","last_peered":"2024-03-26T03:19:39.596365+0000","last_clean":"2024-03-26T03:19:39.596365+0000","last_became_active":"2024-03-26T03:14:37.274931+0000","last_became_peered":"2024-03-26T03:14:37.274931+0000","last_unstale":"2024-03-26T03:19:39.596365+0000","last_undegraded":"2024-03-26T03:19:39.596365+0000","last_fullsized":"2024-03-26T03:19:39.596365+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:19:11.557859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:11.557859+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:35.928237+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":892,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599341+0000","last_change":"2024-03-26T03:19:04.803767+0000","last_active":"2024-03-26T03:19:39.599341+0000","last_peered":"2024-03-26T03:19:39.599341+0000","last_clean":"2024-03-26T03:19:39.599341+0000","last_became_active":"2024-03-26T03:14:37.266519+0000","last_became_peered":"2024-03-26T03:14:37.266519+0000","last_unstale":"2024-03-26T03:19:39.599341+0000","last_undegraded":"2024-03-26T03:19:39.599341+0000","last_fullsized":"2024-03-26T03:19:39.599341+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:04.803650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:19:04.803650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:13.283931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"428'11","reported_seq":1147,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597779+0000","last_change":"2024-03-26T03:17:56.576319+0000","last_active":"2024-03-26T03:19:39.597779+0000","last_peered":"2024-03-26T03:19:39.597779+0000","last_clean":"2024-03-26T03:19:39.597779+0000","last_became_active":"2024-03-26T03:17:56.569871+0000","last_became_peered":"2024-03-26T03:17:56.569871+0000","last_unstale":"2024-03-26T03:19:39.597779+0000","last_undegraded":"2024-03-26T03:19:39.597779+0000","last_fullsized":"2024-03-26T03:19:39.597779+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'10","last_scrub_stamp":"2024-03-26T03:16:56.648648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.648648+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":636,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594522+0000","last_change":"2024-03-26T03:19:17.489071+0000","last_active":"2024-03-26T03:19:39.594522+0000","last_peered":"2024-03-26T03:19:39.594522+0000","last_clean":"2024-03-26T03:19:39.594522+0000","last_became_active":"2024-03-26T03:16:50.206818+0000","last_became_peered":"2024-03-26T03:16:50.206818+0000","last_unstale":"2024-03-26T03:19:39.594522+0000","last_undegraded":"2024-03-26T03:19:39.594522+0000","last_fullsized":"2024-03-26T03:19:39.594522+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:19:17.488941+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:19:17.488941+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:27.176056+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593428+0000","last_change":"2024-03-26T03:17:59.611282+0000","last_active":"2024-03-26T03:19:39.593428+0000","last_peered":"2024-03-26T03:19:39.593428+0000","last_clean":"2024-03-26T03:19:39.593428+0000","last_became_active":"2024-03-26T03:17:59.610887+0000","last_became_peered":"2024-03-26T03:17:59.610887+0000","last_unstale":"2024-03-26T03:19:39.593428+0000","last_undegraded":"2024-03-26T03:19:39.593428+0000","last_fullsized":"2024-03-26T03:19:39.593428+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.803243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.803243+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"446'33","reported_seq":915,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598545+0000","last_change":"2024-03-26T03:18:49.800384+0000","last_active":"2024-03-26T03:19:39.598545+0000","last_peered":"2024-03-26T03:19:39.598545+0000","last_clean":"2024-03-26T03:19:39.598545+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:19:39.598545+0000","last_undegraded":"2024-03-26T03:19:39.598545+0000","last_fullsized":"2024-03-26T03:19:39.598545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:18:49.800256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.800256+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:06.949716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":501,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596105+0000","last_change":"2024-03-26T03:17:31.999279+0000","last_active":"2024-03-26T03:19:39.596105+0000","last_peered":"2024-03-26T03:19:39.596105+0000","last_clean":"2024-03-26T03:19:39.596105+0000","last_became_active":"2024-03-26T03:14:22.576816+0000","last_became_peered":"2024-03-26T03:14:22.576816+0000","last_unstale":"2024-03-26T03:19:39.596105+0000","last_undegraded":"2024-03-26T03:19:39.596105+0000","last_fullsized":"2024-03-26T03:19:39.596105+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:17:31.999158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:31.999158+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.592426+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":241,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.601076+0000","last_change":"2024-03-26T03:14:37.255762+0000","last_active":"2024-03-26T03:19:39.601076+0000","last_peered":"2024-03-26T03:19:39.601076+0000","last_clean":"2024-03-26T03:19:39.601076+0000","last_became_active":"2024-03-26T03:14:37.255400+0000","last_became_peered":"2024-03-26T03:14:37.255400+0000","last_unstale":"2024-03-26T03:19:39.601076+0000","last_undegraded":"2024-03-26T03:19:39.601076+0000","last_fullsized":"2024-03-26T03:19:39.601076+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:25.571608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:13:25.571608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":855,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599658+0000","last_change":"2024-03-26T03:17:56.570706+0000","last_active":"2024-03-26T03:19:39.599658+0000","last_peered":"2024-03-26T03:19:39.599658+0000","last_clean":"2024-03-26T03:19:39.599658+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:19:39.599658+0000","last_undegraded":"2024-03-26T03:19:39.599658+0000","last_fullsized":"2024-03-26T03:19:39.599658+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.953482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.953482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094517+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":746,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596217+0000","last_change":"2024-03-26T03:14:40.162652+0000","last_active":"2024-03-26T03:19:39.596217+0000","last_peered":"2024-03-26T03:19:39.596217+0000","last_clean":"2024-03-26T03:19:39.596217+0000","last_became_active":"2024-03-26T03:14:40.162258+0000","last_became_peered":"2024-03-26T03:14:40.162258+0000","last_unstale":"2024-03-26T03:19:39.596217+0000","last_undegraded":"2024-03-26T03:19:39.596217+0000","last_fullsized":"2024-03-26T03:19:39.596217+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:13:20.546156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:13:20.546156+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588148+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"446'39","reported_seq":548,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.614988+0000","last_change":"2024-03-26T03:16:50.211157+0000","last_active":"2024-03-26T03:19:39.614988+0000","last_peered":"2024-03-26T03:19:39.614988+0000","last_clean":"2024-03-26T03:19:39.614988+0000","last_became_active":"2024-03-26T03:16:50.209508+0000","last_became_peered":"2024-03-26T03:16:50.209508+0000","last_unstale":"2024-03-26T03:19:39.614988+0000","last_undegraded":"2024-03-26T03:19:39.614988+0000","last_fullsized":"2024-03-26T03:19:39.614988+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'27","last_scrub_stamp":"2024-03-26T03:16:36.265181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.265181+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":877,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596095+0000","last_change":"2024-03-26T03:17:47.637363+0000","last_active":"2024-03-26T03:19:39.596095+0000","last_peered":"2024-03-26T03:19:39.596095+0000","last_clean":"2024-03-26T03:19:39.596095+0000","last_became_active":"2024-03-26T03:14:37.249066+0000","last_became_peered":"2024-03-26T03:14:37.249066+0000","last_unstale":"2024-03-26T03:19:39.596095+0000","last_undegraded":"2024-03-26T03:19:39.596095+0000","last_fullsized":"2024-03-26T03:19:39.596095+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:47.637246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:47.637246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":142,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596431+0000","last_change":"2024-03-26T03:17:56.573995+0000","last_active":"2024-03-26T03:19:39.596431+0000","last_peered":"2024-03-26T03:19:39.596431+0000","last_clean":"2024-03-26T03:19:39.596431+0000","last_became_active":"2024-03-26T03:17:56.571840+0000","last_became_peered":"2024-03-26T03:17:56.571840+0000","last_unstale":"2024-03-26T03:19:39.596431+0000","last_undegraded":"2024-03-26T03:19:39.596431+0000","last_fullsized":"2024-03-26T03:19:39.596431+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.507151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.507151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598369+0000","last_change":"2024-03-26T03:13:29.068724+0000","last_active":"2024-03-26T03:19:39.598369+0000","last_peered":"2024-03-26T03:19:39.598369+0000","last_clean":"2024-03-26T03:19:39.598369+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:19:39.598369+0000","last_undegraded":"2024-03-26T03:19:39.598369+0000","last_fullsized":"2024-03-26T03:19:39.598369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.068502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.068502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":649,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599478+0000","last_change":"2024-03-26T03:18:48.870805+0000","last_active":"2024-03-26T03:19:39.599478+0000","last_peered":"2024-03-26T03:19:39.599478+0000","last_clean":"2024-03-26T03:19:39.599478+0000","last_became_active":"2024-03-26T03:14:40.162583+0000","last_became_peered":"2024-03-26T03:14:40.162583+0000","last_unstale":"2024-03-26T03:19:39.599478+0000","last_undegraded":"2024-03-26T03:19:39.599478+0000","last_fullsized":"2024-03-26T03:19:39.599478+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:18:48.870709+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:18:48.870709+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:01.732773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"446'36","reported_seq":601,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599176+0000","last_change":"2024-03-26T03:16:48.957169+0000","last_active":"2024-03-26T03:19:39.599176+0000","last_peered":"2024-03-26T03:19:39.599176+0000","last_clean":"2024-03-26T03:19:39.599176+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:19:39.599176+0000","last_undegraded":"2024-03-26T03:19:39.599176+0000","last_fullsized":"2024-03-26T03:19:39.599176+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'30","last_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.957049+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.590720+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":559,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.608337+0000","last_change":"2024-03-26T03:16:39.889851+0000","last_active":"2024-03-26T03:19:39.608337+0000","last_peered":"2024-03-26T03:19:39.608337+0000","last_clean":"2024-03-26T03:19:39.608337+0000","last_became_active":"2024-03-26T03:14:37.275278+0000","last_became_peered":"2024-03-26T03:14:37.275278+0000","last_unstale":"2024-03-26T03:19:39.608337+0000","last_undegraded":"2024-03-26T03:19:39.608337+0000","last_fullsized":"2024-03-26T03:19:39.608337+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:39.889770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.889770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":822,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.594641+0000","last_change":"2024-03-26T03:14:36.271942+0000","last_active":"2024-03-26T03:19:39.594641+0000","last_peered":"2024-03-26T03:19:39.594641+0000","last_clean":"2024-03-26T03:19:39.594641+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:19:39.594641+0000","last_undegraded":"2024-03-26T03:19:39.594641+0000","last_fullsized":"2024-03-26T03:19:39.594641+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.271810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.271810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.304876+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":522,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.603758+0000","last_change":"2024-03-26T03:18:57.829715+0000","last_active":"2024-03-26T03:19:39.603758+0000","last_peered":"2024-03-26T03:19:39.603758+0000","last_clean":"2024-03-26T03:19:39.603758+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:19:39.603758+0000","last_undegraded":"2024-03-26T03:19:39.603758+0000","last_fullsized":"2024-03-26T03:19:39.603758+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:57.829604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:57.829604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:15.026242+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"446'39","reported_seq":94,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593329+0000","last_change":"2024-03-26T03:17:57.160567+0000","last_active":"2024-03-26T03:19:39.593329+0000","last_peered":"2024-03-26T03:19:39.593329+0000","last_clean":"2024-03-26T03:19:39.593329+0000","last_became_active":"2024-03-26T03:17:56.567557+0000","last_became_peered":"2024-03-26T03:17:56.567557+0000","last_unstale":"2024-03-26T03:19:39.593329+0000","last_undegraded":"2024-03-26T03:19:39.593329+0000","last_fullsized":"2024-03-26T03:19:39.593329+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'33","last_scrub_stamp":"2024-03-26T03:17:57.160442+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.160442+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307719+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":253,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599401+0000","last_change":"2024-03-26T03:19:14.394760+0000","last_active":"2024-03-26T03:19:39.599401+0000","last_peered":"2024-03-26T03:19:39.599401+0000","last_clean":"2024-03-26T03:19:39.599401+0000","last_became_active":"2024-03-26T03:14:37.243678+0000","last_became_peered":"2024-03-26T03:14:37.243678+0000","last_unstale":"2024-03-26T03:19:39.599401+0000","last_undegraded":"2024-03-26T03:19:39.599401+0000","last_fullsized":"2024-03-26T03:19:39.599401+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:14.394668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:19:14.394668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:33.267097+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":894,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598343+0000","last_change":"2024-03-26T03:18:01.638296+0000","last_active":"2024-03-26T03:19:39.598343+0000","last_peered":"2024-03-26T03:19:39.598343+0000","last_clean":"2024-03-26T03:19:39.598343+0000","last_became_active":"2024-03-26T03:17:56.570631+0000","last_became_peered":"2024-03-26T03:17:56.570631+0000","last_unstale":"2024-03-26T03:19:39.598343+0000","last_undegraded":"2024-03-26T03:19:39.598343+0000","last_fullsized":"2024-03-26T03:19:39.598343+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:01.638157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:01.638157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":313,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.600655+0000","last_change":"2024-03-26T03:17:56.571755+0000","last_active":"2024-03-26T03:19:39.600655+0000","last_peered":"2024-03-26T03:19:39.600655+0000","last_clean":"2024-03-26T03:19:39.600655+0000","last_became_active":"2024-03-26T03:17:56.571178+0000","last_became_peered":"2024-03-26T03:17:56.571178+0000","last_unstale":"2024-03-26T03:19:39.600655+0000","last_undegraded":"2024-03-26T03:19:39.600655+0000","last_fullsized":"2024-03-26T03:19:39.600655+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:13:29.660527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:13:29.660527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.093941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":588,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595931+0000","last_change":"2024-03-26T03:17:59.595137+0000","last_active":"2024-03-26T03:19:39.595931+0000","last_peered":"2024-03-26T03:19:39.595931+0000","last_clean":"2024-03-26T03:19:39.595931+0000","last_became_active":"2024-03-26T03:17:59.594759+0000","last_became_peered":"2024-03-26T03:17:59.594759+0000","last_unstale":"2024-03-26T03:19:39.595931+0000","last_undegraded":"2024-03-26T03:19:39.595931+0000","last_fullsized":"2024-03-26T03:19:39.595931+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:32.825864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.825864+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:44.588280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"446'15","reported_seq":74,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595837+0000","last_change":"2024-03-26T03:17:56.580380+0000","last_active":"2024-03-26T03:19:39.595837+0000","last_peered":"2024-03-26T03:19:39.595837+0000","last_clean":"2024-03-26T03:19:39.595837+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:19:39.595837+0000","last_undegraded":"2024-03-26T03:19:39.595837+0000","last_fullsized":"2024-03-26T03:19:39.595837+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'13","last_scrub_stamp":"2024-03-26T03:17:45.281375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.281375+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307532+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":850,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596745+0000","last_change":"2024-03-26T03:16:38.824092+0000","last_active":"2024-03-26T03:19:39.596745+0000","last_peered":"2024-03-26T03:19:39.596745+0000","last_clean":"2024-03-26T03:19:39.596745+0000","last_became_active":"2024-03-26T03:14:37.373600+0000","last_became_peered":"2024-03-26T03:14:37.373600+0000","last_unstale":"2024-03-26T03:19:39.596745+0000","last_undegraded":"2024-03-26T03:19:39.596745+0000","last_fullsized":"2024-03-26T03:19:39.596745+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:38.823985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:16:38.823985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":246,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593974+0000","last_change":"2024-03-26T03:16:51.182163+0000","last_active":"2024-03-26T03:19:39.593974+0000","last_peered":"2024-03-26T03:19:39.593974+0000","last_clean":"2024-03-26T03:19:39.593974+0000","last_became_active":"2024-03-26T03:14:37.253707+0000","last_became_peered":"2024-03-26T03:14:37.253707+0000","last_unstale":"2024-03-26T03:19:39.593974+0000","last_undegraded":"2024-03-26T03:19:39.593974+0000","last_fullsized":"2024-03-26T03:19:39.593974+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.182029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":843,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598164+0000","last_change":"2024-03-26T03:19:13.804053+0000","last_active":"2024-03-26T03:19:39.598164+0000","last_peered":"2024-03-26T03:19:39.598164+0000","last_clean":"2024-03-26T03:19:39.598164+0000","last_became_active":"2024-03-26T03:14:21.596806+0000","last_became_peered":"2024-03-26T03:14:21.596806+0000","last_unstale":"2024-03-26T03:19:39.598164+0000","last_undegraded":"2024-03-26T03:19:39.598164+0000","last_fullsized":"2024-03-26T03:19:39.598164+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:13.803940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:13.803940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:36.864830+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":648,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597976+0000","last_change":"2024-03-26T03:17:59.610214+0000","last_active":"2024-03-26T03:19:39.597976+0000","last_peered":"2024-03-26T03:19:39.597976+0000","last_clean":"2024-03-26T03:19:39.597976+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:19:39.597976+0000","last_undegraded":"2024-03-26T03:19:39.597976+0000","last_fullsized":"2024-03-26T03:19:39.597976+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:51.516127+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:17:51.516127+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.105428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"446'27","reported_seq":895,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592250+0000","last_change":"2024-03-26T03:16:46.178208+0000","last_active":"2024-03-26T03:19:39.592250+0000","last_peered":"2024-03-26T03:19:39.592250+0000","last_clean":"2024-03-26T03:19:39.592250+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:19:39.592250+0000","last_undegraded":"2024-03-26T03:19:39.592250+0000","last_fullsized":"2024-03-26T03:19:39.592250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:16:46.178099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.178099+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":531,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.596804+0000","last_change":"2024-03-26T03:19:06.601810+0000","last_active":"2024-03-26T03:19:39.596804+0000","last_peered":"2024-03-26T03:19:39.596804+0000","last_clean":"2024-03-26T03:19:39.596804+0000","last_became_active":"2024-03-26T03:14:22.571857+0000","last_became_peered":"2024-03-26T03:14:22.571857+0000","last_unstale":"2024-03-26T03:19:39.596804+0000","last_undegraded":"2024-03-26T03:19:39.596804+0000","last_fullsized":"2024-03-26T03:19:39.596804+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:06.601677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:06.601677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:30.452453+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":853,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.597799+0000","last_change":"2024-03-26T03:15:20.809430+0000","last_active":"2024-03-26T03:19:39.597799+0000","last_peered":"2024-03-26T03:19:39.597799+0000","last_clean":"2024-03-26T03:19:39.597799+0000","last_became_active":"2024-03-26T03:14:37.242540+0000","last_became_peered":"2024-03-26T03:14:37.242540+0000","last_unstale":"2024-03-26T03:19:39.597799+0000","last_undegraded":"2024-03-26T03:19:39.597799+0000","last_fullsized":"2024-03-26T03:19:39.597799+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:20.809176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:20.809176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:41.094292+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":849,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.593789+0000","last_change":"2024-03-26T03:17:43.076516+0000","last_active":"2024-03-26T03:19:39.593789+0000","last_peered":"2024-03-26T03:19:39.593789+0000","last_clean":"2024-03-26T03:19:39.593789+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:19:39.593789+0000","last_undegraded":"2024-03-26T03:19:39.593789+0000","last_fullsized":"2024-03-26T03:19:39.593789+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.076334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.076334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":553,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.599247+0000","last_change":"2024-03-26T03:19:01.651338+0000","last_active":"2024-03-26T03:19:39.599247+0000","last_peered":"2024-03-26T03:19:39.599247+0000","last_clean":"2024-03-26T03:19:39.599247+0000","last_became_active":"2024-03-26T03:14:40.174111+0000","last_became_peered":"2024-03-26T03:14:40.174111+0000","last_unstale":"2024-03-26T03:19:39.599247+0000","last_undegraded":"2024-03-26T03:19:39.599247+0000","last_fullsized":"2024-03-26T03:19:39.599247+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:01.651232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:01.651232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.733163+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"446'64","reported_seq":638,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.595988+0000","last_change":"2024-03-26T03:19:09.592737+0000","last_active":"2024-03-26T03:19:39.595988+0000","last_peered":"2024-03-26T03:19:39.595988+0000","last_clean":"2024-03-26T03:19:39.595988+0000","last_became_active":"2024-03-26T03:14:22.576815+0000","last_became_peered":"2024-03-26T03:14:22.576815+0000","last_unstale":"2024-03-26T03:19:39.595988+0000","last_undegraded":"2024-03-26T03:19:39.595988+0000","last_fullsized":"2024-03-26T03:19:39.595988+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'64","last_scrub_stamp":"2024-03-26T03:19:09.592406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:09.592406+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:20.478394+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":249,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.598426+0000","last_change":"2024-03-26T03:16:48.135457+0000","last_active":"2024-03-26T03:19:39.598426+0000","last_peered":"2024-03-26T03:19:39.598426+0000","last_clean":"2024-03-26T03:19:39.598426+0000","last_became_active":"2024-03-26T03:14:37.248588+0000","last_became_peered":"2024-03-26T03:14:37.248588+0000","last_unstale":"2024-03-26T03:19:39.598426+0000","last_undegraded":"2024-03-26T03:19:39.598426+0000","last_fullsized":"2024-03-26T03:19:39.598426+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:48.135354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.135354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.305233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":63,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.592487+0000","last_change":"2024-03-26T03:17:56.572431+0000","last_active":"2024-03-26T03:19:39.592487+0000","last_peered":"2024-03-26T03:19:39.592487+0000","last_clean":"2024-03-26T03:19:39.592487+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:19:39.592487+0000","last_undegraded":"2024-03-26T03:19:39.592487+0000","last_fullsized":"2024-03-26T03:19:39.592487+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.615644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.615644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:19:42.307417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":523,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T03:19:39.602445+0000","last_change":"2024-03-26T03:18:58.798963+0000","last_active":"2024-03-26T03:19:39.602445+0000","last_peered":"2024-03-26T03:19:39.602445+0000","last_clean":"2024-03-26T03:19:39.602445+0000","last_became_active":"2024-03-26T03:14:37.265472+0000","last_became_peered":"2024-03-26T03:14:37.265472+0000","last_unstale":"2024-03-26T03:19:39.602445+0000","last_undegraded":"2024-03-26T03:19:39.602445+0000","last_fullsized":"2024-03-26T03:19:39.602445+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:58.798827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:58.798827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:18.573112+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3232235544,"num_objects":49321,"num_object_clones":0,"num_object_copies":147963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214471,"num_write_kb":8441281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20605,"num_bytes_recovered":1350238208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10034819072,"data_stored":10034806856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53180,"ondisk_log_size":53180,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1232,"num_read_kb":1227,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1828,"num_read_kb":1553,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":3014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1244,"ondisk_log_size":1244,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430465,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3005672,"kb_used_data":2172064,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":90739480,"statfs":{"total":95995035648,"available":92917227520,"internally_reserved":0,"allocated":2224193536,"data_stored":2223817022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":853604806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575130,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3202520,"kb_used_data":2164432,"kb_used_omap":1,"kb_used_meta":1037694,"kb_avail":90542632,"statfs":{"total":95995035648,"available":92715655168,"internally_reserved":0,"allocated":2216378368,"data_stored":2215961841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1062599109},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738655,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411604,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2297020,"kb_used_data":1403700,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":91448132,"statfs":{"total":95995035648,"available":93642887168,"internally_reserved":0,"allocated":1437388800,"data_stored":1437000252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175041,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2226116,"kb_used_data":1780412,"kb_used_omap":4,"kb_used_meta":445499,"kb_avail":91519036,"statfs":{"total":95995035648,"available":93715492864,"internally_reserved":0,"allocated":1823141888,"data_stored":1822737323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":456191651},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986443,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2304048,"kb_used_data":1420392,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91441104,"statfs":{"total":95995035648,"available":93635690496,"internally_reserved":0,"allocated":1454481408,"data_stored":1454097502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692103,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1975560,"kb_used_data":991996,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":91769592,"statfs":{"total":95995035648,"available":93972062208,"internally_reserved":0,"allocated":1015803904,"data_stored":1015406643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1007155653},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2173173760,"data_stored":2173173760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1407782912,"data_stored":1407778840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1820327936,"data_stored":1820327936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1432293376,"data_stored":1432289304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1014366208,"data_stored":1014366208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2186874880,"data_stored":2186870808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:19:47.115 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T03:19:47.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:19:47.116 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-26T03:19:47.116 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:19:47.116 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:19:47.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-26T03:19:47.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:47.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:47.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:47.303 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:47.401 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:19:47.401 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:19:47.401 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 0 2024-03-26T03:19:47.401 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:19:47.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:47.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:47.478 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:19:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:47 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:47 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:47 smithi192 ceph-mon[33110]: osdmap e457: 8 total, 8 up, 6 in 2024-03-26T03:19:47.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:47 smithi192 ceph-mon[33110]: pgmap v984: 178 pgs: 178 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-03-26T03:19:47.548 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:47.589 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[29666]: osdmap e457: 8 total, 8 up, 6 in 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[29666]: pgmap v984: 178 pgs: 178 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[37920]: osdmap e457: 8 total, 8 up, 6 in 2024-03-26T03:19:47.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:47 smithi066 ceph-mon[37920]: pgmap v984: 178 pgs: 178 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-03-26T03:19:47.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:47.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:47.824 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:47.887 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:47.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:47.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:48.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:48.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:48.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:48.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:48.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:48.390 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:48.432 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:48.500 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:19:48.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:48.545 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:48.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:48.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:48.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:48.796 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.866 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:48.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:48.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:48.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:49.100 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:49.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:49.241 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:49.286 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:49.388 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:49.430 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:49.529 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:19:49.529 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:19:49.529 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:19:49.532 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:49.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:49.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:49.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:49.792 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:49.806 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:19:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.838 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:49.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:49.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:49 smithi066 ceph-mon[29666]: pgmap v985: 178 pgs: 178 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 747 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T03:19:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:49 smithi066 ceph-mon[37920]: pgmap v985: 178 pgs: 178 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 747 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T03:19:49.866 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:49 smithi192 ceph-mon[33110]: pgmap v985: 178 pgs: 178 active+clean; 2.6 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 747 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-26T03:19:49.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:49.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:49.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:50.075 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.139 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:50.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:50.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:50.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:50.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:50.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:50.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:19:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:19:50.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:50.538 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:50.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:50.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:50.772 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:50.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:50.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:50.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:50.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:51.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:51.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:51.184 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:51.228 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:51.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:51.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:51.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:51.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:51.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:51.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:51.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:51.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:51.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:51.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:51.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:51 smithi066 ceph-mon[29666]: pgmap v986: 178 pgs: 178 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.62k op/s 2024-03-26T03:19:51.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:51.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:51 smithi066 ceph-mon[37920]: pgmap v986: 178 pgs: 178 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.62k op/s 2024-03-26T03:19:51.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:51.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:51.985 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:19:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:51 smithi192 ceph-mon[33110]: pgmap v986: 178 pgs: 178 active+clean; 2.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.62k op/s 2024-03-26T03:19:52.012 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:52.114 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:52.158 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:52.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:52.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:52.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:52.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:52.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:52.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:52.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:52.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:52.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:52.785 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.814 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:52.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:52.915 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:52.957 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:52.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:53.059 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.107 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:53.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:53.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:53.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:53.344 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.409 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:53.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:53.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:53.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:53.544 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:53.599 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:53 smithi066 ceph-mon[29666]: pgmap v987: 178 pgs: 178 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-03-26T03:19:53.599 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:53 smithi066 ceph-mon[37920]: pgmap v987: 178 pgs: 178 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-03-26T03:19:53.641 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:53.693 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:53.777 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:19:53.795 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:53.855 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:19:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:19:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:19:53.862 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:53.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:53 smithi192 ceph-mon[33110]: pgmap v987: 178 pgs: 178 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-03-26T03:19:53.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:53.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:54.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:54.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:54.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.277 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:54.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:54.378 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:54.421 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:54.523 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:54.564 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:54.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:54.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:54.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:54.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:54.848 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:54.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:54.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:54.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:55.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:55.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:55.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:55.215 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6] out_osds: [0, 7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:19:55.215 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:19:55.216 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-03-26T03:19:55.216 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd in 0 2024-03-26T03:19:55.221 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:55.311 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:55.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:55.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:55.466 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:55.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:55.568 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:55.573 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:19:55.619 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:55.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:55.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:55.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:55.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:55.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:55.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[29666]: pgmap v988: 178 pgs: 178 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2026012960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[29666]: from='client.35990 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[37920]: pgmap v988: 178 pgs: 178 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2026012960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[37920]: from='client.35990 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:19:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:55 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:19:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:55.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:55.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:55.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:55.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:55 smithi192 ceph-mon[33110]: pgmap v988: 178 pgs: 178 active+clean; 2.1 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-26T03:19:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2026012960' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:19:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:55 smithi192 ceph-mon[33110]: from='client.35990 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:19:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:19:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:55 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:19:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:55 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:19:56.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:56.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:56.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:56.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:56.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:56.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:56.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:56.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:56.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:56.651 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.704 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:56.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:56.805 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:56.851 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:56.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[29666]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[29666]: from='client.35990 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[29666]: osdmap e458: 8 total, 8 up, 6 in 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[37920]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[37920]: from='client.35990 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[37920]: osdmap e458: 8 total, 8 up, 6 in 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:19:56.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:56 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:19:56.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:56.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:56.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:56 smithi192 ceph-mon[33110]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:19:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:56 smithi192 ceph-mon[33110]: from='client.35990 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:19:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:56 smithi192 ceph-mon[33110]: osdmap e458: 8 total, 8 up, 6 in 2024-03-26T03:19:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:56 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:19:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:56 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:19:57.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:57.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:57.217 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:57.275 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:57.377 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:57.423 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:57.525 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:57.572 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:57.674 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:57.718 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:57.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:57.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[29666]: pgmap v990: 178 pgs: 178 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-03-26T03:19:57.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:19:57.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:19:57.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/587256317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:57.829 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:19:57.830 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[37920]: pgmap v990: 178 pgs: 178 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-03-26T03:19:57.830 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:19:57.830 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:19:57.830 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/587256317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:57.830 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:57 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:19:57.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:57.952 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:19:57.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:57 smithi192 ceph-mon[33110]: pgmap v990: 178 pgs: 178 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-03-26T03:19:57.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:57 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:19:57.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:57 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:19:57.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/587256317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:19:57.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:57 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:19:57.998 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:57.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:19:58.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:19:58.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:19:58.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:58.272 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:58.374 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:58.422 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:58.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:58.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:58.658 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:19:58.699 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:19:58.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:58.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[29666]: 3.1 scrub starts 2024-03-26T03:19:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[29666]: 3.1 scrub ok 2024-03-26T03:19:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:19:58.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:19:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[37920]: 3.1 scrub starts 2024-03-26T03:19:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[37920]: 3.1 scrub ok 2024-03-26T03:19:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:19:58.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:58 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:19:58.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:58.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:58.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:58 smithi192 ceph-mon[33110]: 3.1 scrub starts 2024-03-26T03:19:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:58 smithi192 ceph-mon[33110]: 3.1 scrub ok 2024-03-26T03:19:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:58 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:19:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:58 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:19:59.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:19:59.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:19:59.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:59.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:59.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:19:59.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:19:59.460 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.507 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:59.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:59.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:19:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:19:59.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:19:59.747 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:19:59.748 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:19:59.748 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:19:59.748 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:19:59.748 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:19:59.749 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:19:59.750 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:19:59.792 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:19:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: 11.1 scrub starts 2024-03-26T03:19:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: pgmap v991: 178 pgs: 178 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-03-26T03:19:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:19:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:19:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: 11.1 scrub ok 2024-03-26T03:19:59.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:19:59.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:19:59.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:59.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: 11.1 scrub starts 2024-03-26T03:19:59.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: pgmap v991: 178 pgs: 178 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-03-26T03:19:59.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:19:59.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:19:59.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: 11.1 scrub ok 2024-03-26T03:19:59.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:19:59.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:19:59.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:19:59 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:19:59.894 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:19:59.926 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:19:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:19:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:19:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: 11.1 scrub starts 2024-03-26T03:19:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: pgmap v991: 178 pgs: 178 active+clean; 1.7 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.48k op/s 2024-03-26T03:19:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:19:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:19:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: 11.1 scrub ok 2024-03-26T03:19:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:19:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:19:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:19:59 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:00.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:00.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:00.162 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:00.189 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:00.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:00.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:00.416 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:00.461 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:00.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:20:00.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:00.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:00.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:00.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:00.821 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[29666]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[29666]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:20:00.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[37920]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-26T03:20:00.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[37920]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T03:20:00.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:20:00.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:00 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:20:00.864 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:00.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:20:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:20:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-mon[33110]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-26T03:20:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-mon[33110]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T03:20:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:20:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:00 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:20:00.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:00.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:01.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:01.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:01.227 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:01.272 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:01.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:01.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:01.509 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:01.537 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:01.638 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:01.686 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:01.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:01.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:01.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[29666]: pgmap v992: 178 pgs: 178 active+clean; 1.3 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[29666]: 1.c scrub starts 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[29666]: 1.c scrub ok 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[37920]: pgmap v992: 178 pgs: 178 active+clean; 1.3 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[37920]: 1.c scrub starts 2024-03-26T03:20:01.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:01 smithi066 ceph-mon[37920]: 1.c scrub ok 2024-03-26T03:20:01.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:01.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:01.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:01 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:20:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:01 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:20:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:01 smithi192 ceph-mon[33110]: pgmap v992: 178 pgs: 178 active+clean; 1.3 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-26T03:20:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:01 smithi192 ceph-mon[33110]: 1.c scrub starts 2024-03-26T03:20:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:01 smithi192 ceph-mon[33110]: 1.c scrub ok 2024-03-26T03:20:02.064 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:02.106 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:02.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:02.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:02.344 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:02.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:02.493 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:02.537 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:02.638 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:02.681 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:02.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:02.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:02.892 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:20:02.893 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:02 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:20:02.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:02.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:02.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:02 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:20:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:02 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:20:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:02 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:20:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:02 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:20:03.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.094 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:03.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:03.195 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:03.247 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:03.348 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:03.395 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:03.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:03.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:03.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:03.657 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:03.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:03.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:03.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:03.914 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:03 smithi066 ceph-mon[29666]: pgmap v993: 178 pgs: 178 active+clean; 1.2 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s 2024-03-26T03:20:03.914 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:03 smithi066 ceph-mon[37920]: pgmap v993: 178 pgs: 178 active+clean; 1.2 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s 2024-03-26T03:20:03.915 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:20:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:20:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:20:03.928 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:03.940 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:03.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:03.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:03 smithi192 ceph-mon[33110]: pgmap v993: 178 pgs: 178 active+clean; 1.2 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s 2024-03-26T03:20:04.042 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.091 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:04.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:04.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:04.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:04.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:04.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:04.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:04.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:04.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:04.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:04.706 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:04 smithi066 ceph-mon[29666]: 6.3 scrub starts 2024-03-26T03:20:04.706 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:04 smithi066 ceph-mon[29666]: 6.3 scrub ok 2024-03-26T03:20:04.706 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:04 smithi066 ceph-mon[29666]: pgmap v994: 178 pgs: 178 active+clean; 1.2 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s 2024-03-26T03:20:04.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:04 smithi066 ceph-mon[37920]: 6.3 scrub starts 2024-03-26T03:20:04.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:04 smithi066 ceph-mon[37920]: 6.3 scrub ok 2024-03-26T03:20:04.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:04 smithi066 ceph-mon[37920]: pgmap v994: 178 pgs: 178 active+clean; 1.2 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s 2024-03-26T03:20:04.722 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:04.773 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:04.871 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:20:04.871 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:20:04.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:04.905 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:04.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:04.940 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:20:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:04 smithi192 ceph-mon[33110]: 6.3 scrub starts 2024-03-26T03:20:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:04 smithi192 ceph-mon[33110]: 6.3 scrub ok 2024-03-26T03:20:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:04 smithi192 ceph-mon[33110]: pgmap v994: 178 pgs: 178 active+clean; 1.2 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.44k op/s 2024-03-26T03:20:05.008 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:05.052 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:05.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:05.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:05.288 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.337 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:05.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:05.438 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:05.484 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:05.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:05.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:05.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:05.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:05.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[29666]: 11.3 scrub starts 2024-03-26T03:20:05.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[29666]: 1.19 scrub starts 2024-03-26T03:20:05.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[29666]: 1.19 scrub ok 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[29666]: 11.3 scrub ok 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[37920]: 11.3 scrub starts 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[37920]: 1.19 scrub starts 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[37920]: 1.19 scrub ok 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[37920]: 11.3 scrub ok 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:20:05.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:05 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:20:05.861 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:05.908 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:05.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:05.972 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:20:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:05 smithi192 ceph-mon[33110]: 11.3 scrub starts 2024-03-26T03:20:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:05 smithi192 ceph-mon[33110]: 1.19 scrub starts 2024-03-26T03:20:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:05 smithi192 ceph-mon[33110]: 1.19 scrub ok 2024-03-26T03:20:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:05 smithi192 ceph-mon[33110]: 11.3 scrub ok 2024-03-26T03:20:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:05 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:20:05.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:05 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:20:06.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:06.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:06.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:06.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:06.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:06.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:06.403 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:06.449 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:06.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:06.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:06.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:06.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:06.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:06.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:06.945 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:06 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:20:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:06 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:20:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:06 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:20:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:06 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:20:06.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:06 smithi192 ceph-mon[33110]: pgmap v995: 178 pgs: 178 active+clean; 793 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-03-26T03:20:06.990 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:06.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:07.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:07.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:20:07.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:20:07.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:20:07.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:20:07.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[29666]: pgmap v995: 178 pgs: 178 active+clean; 793 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-03-26T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:20:07.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:06 smithi066 ceph-mon[37920]: pgmap v995: 178 pgs: 178 active+clean; 793 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-03-26T03:20:07.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:07.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:07.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:07.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:07.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:07.494 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:07.542 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:07.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:07.685 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:07.787 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:07.827 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:07.929 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:07.973 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:07.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:07 smithi192 ceph-mon[33110]: 1.17 scrub starts 2024-03-26T03:20:07.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:07 smithi192 ceph-mon[33110]: 1.17 scrub ok 2024-03-26T03:20:08.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:08.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:07 smithi066 ceph-mon[29666]: 1.17 scrub starts 2024-03-26T03:20:08.080 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:07 smithi066 ceph-mon[29666]: 1.17 scrub ok 2024-03-26T03:20:08.080 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:07 smithi066 ceph-mon[37920]: 1.17 scrub starts 2024-03-26T03:20:08.080 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:07 smithi066 ceph-mon[37920]: 1.17 scrub ok 2024-03-26T03:20:08.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:08.131 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:20:08.206 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:08.237 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:08.339 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:08.384 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:08.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:08.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:08.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:08.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:08.745 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:08.809 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:08.829 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:08.834 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:08.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:08.918 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:08.962 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:08.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:08.970 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:08 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:20:08.970 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:08 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:20:08.970 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:08 smithi066 ceph-mon[29666]: pgmap v996: 178 pgs: 178 active+clean; 793 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T03:20:08.970 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:08 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:20:08.970 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:08 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:20:08.970 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:08 smithi066 ceph-mon[37920]: pgmap v996: 178 pgs: 178 active+clean; 793 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T03:20:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:08 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:20:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:08 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:20:08.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:08 smithi192 ceph-mon[33110]: pgmap v996: 178 pgs: 178 active+clean; 793 MiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T03:20:09.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:09.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:09.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:09.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:09.326 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.379 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:09.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:09.480 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:09.536 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:09.638 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:09.685 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:09.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:09.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:09.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:09 smithi066 ceph-mon[29666]: 6.11 scrub starts 2024-03-26T03:20:09.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:09 smithi066 ceph-mon[29666]: 6.11 scrub ok 2024-03-26T03:20:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:09 smithi066 ceph-mon[37920]: 6.11 scrub starts 2024-03-26T03:20:09.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:09 smithi066 ceph-mon[37920]: 6.11 scrub ok 2024-03-26T03:20:09.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:09.963 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:09.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:09 smithi192 ceph-mon[33110]: 6.11 scrub starts 2024-03-26T03:20:09.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:09 smithi192 ceph-mon[33110]: 6.11 scrub ok 2024-03-26T03:20:10.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:10.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:10.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:10.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:10.324 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:10.374 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:10.478 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:10.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:20:10.521 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:10.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:10.672 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:10.774 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:10.819 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: 1.8 scrub starts 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: 1.8 scrub ok 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[29666]: pgmap v997: 178 pgs: 178 active+clean; 457 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.65k op/s 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: 1.8 scrub starts 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: 1.8 scrub ok 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:20:10.828 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:20:10.829 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:10.829 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:10.829 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-mon[37920]: pgmap v997: 178 pgs: 178 active+clean; 457 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.65k op/s 2024-03-26T03:20:10.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:10.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:10.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: 1.8 scrub starts 2024-03-26T03:20:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: 1.8 scrub ok 2024-03-26T03:20:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:20:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:20:10.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:10.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:10.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:10 smithi192 ceph-mon[33110]: pgmap v997: 178 pgs: 178 active+clean; 457 MiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.65k op/s 2024-03-26T03:20:11.054 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:11.095 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:11.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:10 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29662]: 2024-03-26T03:20:10.825+0000 7f94a71dc700 -1 mon.a@0(leader).osd e458 definitely_dead 0 2024-03-26T03:20:11.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:11.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:11.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:11.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:11.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:11.469 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:11.515 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:11.617 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.662 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:11.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:11.695 INFO:teuthology.orchestra.run.smithi066.stderr:marked in osd.0. 2024-03-26T03:20:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:20:11.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[29666]: 2.1b scrub starts 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[29666]: 2.1b scrub ok 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2706543844' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[37920]: 2.1b scrub starts 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[37920]: 2.1b scrub ok 2024-03-26T03:20:11.726 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2706543844' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-26T03:20:11.764 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:11.791 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:11.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:11.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:11.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:11 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:20:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:11 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:20:11.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:11 smithi192 ceph-mon[33110]: 2.1b scrub starts 2024-03-26T03:20:11.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:11 smithi192 ceph-mon[33110]: 2.1b scrub ok 2024-03-26T03:20:11.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2706543844' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-26T03:20:12.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:12.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:12.149 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:12.176 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:12.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:12.305 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:12.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:12.434 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:12.536 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:12.563 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:12.665 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:12.693 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:12.795 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:12.822 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:12.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:12.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:12.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2706543844' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-26T03:20:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:12 smithi192 ceph-mon[33110]: osdmap e459: 8 total, 8 up, 7 in 2024-03-26T03:20:12.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:12 smithi192 ceph-mon[33110]: pgmap v999: 178 pgs: 178 active+clean; 351 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-26T03:20:13.050 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:13.078 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:13.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:13.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2706543844' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-26T03:20:13.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:12 smithi066 ceph-mon[29666]: osdmap e459: 8 total, 8 up, 7 in 2024-03-26T03:20:13.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:12 smithi066 ceph-mon[29666]: pgmap v999: 178 pgs: 178 active+clean; 351 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-26T03:20:13.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2706543844' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-26T03:20:13.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:12 smithi066 ceph-mon[37920]: osdmap e459: 8 total, 8 up, 7 in 2024-03-26T03:20:13.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:12 smithi066 ceph-mon[37920]: pgmap v999: 178 pgs: 178 active+clean; 351 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-26T03:20:13.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:13.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:13.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:13.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:13.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:13.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:13.469 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-03-26T03:20:13.556 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:13.585 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:13.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:13.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:13.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:13.841 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:13.943 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:13 smithi192 ceph-mon[33110]: osdmap e460: 8 total, 8 up, 7 in 2024-03-26T03:20:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:13 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:20:13.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:13 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:20:13.989 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:13.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:14.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:14.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-mon[29666]: osdmap e460: 8 total, 8 up, 7 in 2024-03-26T03:20:14.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:20:14.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:20:14.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:20:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:20:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-mon[37920]: osdmap e460: 8 total, 8 up, 7 in 2024-03-26T03:20:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:20:14.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:13 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:20:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:14.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:14.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:14.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:14.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:14.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:14.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:14.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:14.622 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:14.669 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:14.771 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:14.818 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:14.920 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:14.967 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:14.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:14.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 1.f scrub starts 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 1.f scrub ok 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 1.1e scrub starts 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 1.1e scrub ok 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 458'16557 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16312 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: osdmap e461: 8 total, 8 up, 7 in 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16650 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 458'16051 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16497 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 458'16181 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16247 2024-03-26T03:20:14.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:20:14.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:20:14.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:20:14.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:20:14.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: osdmap e462: 8 total, 8 up, 7 in 2024-03-26T03:20:14.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:14 smithi192 ceph-mon[33110]: pgmap v1003: 178 pgs: 178 active+clean; 351 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 423 op/s 2024-03-26T03:20:15.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:15.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 1.f scrub starts 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 1.f scrub ok 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 1.1e scrub starts 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 1.1e scrub ok 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 458'16557 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16312 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: osdmap e461: 8 total, 8 up, 7 in 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16650 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 458'16051 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16497 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 458'16181 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16247 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:20:15.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: osdmap e462: 8 total, 8 up, 7 in 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[29666]: pgmap v1003: 178 pgs: 178 active+clean; 351 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 423 op/s 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 1.f scrub starts 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 1.f scrub ok 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 1.1e scrub starts 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 1.1e scrub ok 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 458'16557 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16312 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: osdmap e461: 8 total, 8 up, 7 in 2024-03-26T03:20:15.110 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16650 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 458'16051 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16497 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 458'16181 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 458'16247 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: osdmap e462: 8 total, 8 up, 7 in 2024-03-26T03:20:15.111 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:14 smithi066 ceph-mon[37920]: pgmap v1003: 178 pgs: 178 active+clean; 351 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 423 op/s 2024-03-26T03:20:15.119 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:15.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:15.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:15.351 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:15.398 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:15.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:15.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:15.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:15.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:15.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.781 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:15.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:15.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:15.910 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:15.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:16.014 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:16.042 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:16.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:16.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:16.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:16.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:16.304 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:20:16.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:16.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:16.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:16.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:16.558 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: osdmap e463: 8 total, 8 up, 7 in 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2296664592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[29666]: from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: osdmap e463: 8 total, 8 up, 7 in 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:20:16.559 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2296664592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:16.560 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:16 smithi066 ceph-mon[37920]: from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:16.613 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:20:16.614 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:20:16.614 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:20:16.616 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:20:16.636 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:20:16.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:16.655 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:20:16.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:16.675 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:20:16.695 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:20:16.696 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:20:16.696 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:20:16.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:20:16.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:20:16.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:20:16.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:20:16.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:20:16.697 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:20:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:20:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:20:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: osdmap e463: 8 total, 8 up, 7 in 2024-03-26T03:20:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:20:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:20:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2296664592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:16 smithi192 ceph-mon[33110]: from='client.36002 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:16.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:16.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:16.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:16.927 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:16.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:17.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:17.053 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:17.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:17.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:17.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:17.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:17.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:17.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:17.542 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:17.591 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 1.16 scrub starts 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 1.16 scrub ok 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: osdmap e464: 8 total, 8 up, 7 in 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: pgmap v1006: 178 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 5 peering, 166 active+clean; 128 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.2 KiB/s rd, 29 KiB/s wr, 950 op/s; 584/6801 objects misplaced (8.587%); 6.1 MiB/s, 118 objects/s recovering 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1873405538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 1.9 scrub starts 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 1.9 scrub ok 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 1.16 scrub starts 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 1.16 scrub ok 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:20:17.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: osdmap e464: 8 total, 8 up, 7 in 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: pgmap v1006: 178 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 5 peering, 166 active+clean; 128 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.2 KiB/s rd, 29 KiB/s wr, 950 op/s; 584/6801 objects misplaced (8.587%); 6.1 MiB/s, 118 objects/s recovering 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1873405538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 1.9 scrub starts 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 1.9 scrub ok 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:20:17.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:17 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:20:17.693 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 1.16 scrub starts 2024-03-26T03:20:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 1.16 scrub ok 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: from='client.36002 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: osdmap e464: 8 total, 8 up, 7 in 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: pgmap v1006: 178 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 5 peering, 166 active+clean; 128 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 4.2 KiB/s rd, 29 KiB/s wr, 950 op/s; 584/6801 objects misplaced (8.587%); 6.1 MiB/s, 118 objects/s recovering 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1873405538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 1.9 scrub starts 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 1.9 scrub ok 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:20:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:17 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:20:17.741 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:17.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:17.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:17.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:17.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:17.972 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.008 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:18.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:18.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:18.151 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:18.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:18.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:18.350 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Removed 131895 objects 2024-03-26T03:20:18.351 INFO:tasks.radosbench.radosbench.0.smithi066.stdout:Clean up completed and total clean up time :99.815 2024-03-26T03:20:18.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:18.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:18.470 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 6, 0] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-26T03:20:18.470 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:20:18.470 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T03:20:18.470 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:20:18.507 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:18.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[29666]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:20:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[29666]: osdmap e465: 8 total, 8 up, 7 in 2024-03-26T03:20:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:20:18.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:20:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[37920]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:20:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[37920]: osdmap e465: 8 total, 8 up, 7 in 2024-03-26T03:20:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:20:18.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:18 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:20:18.643 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:18.672 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:18 smithi192 ceph-mon[33110]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:20:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:18 smithi192 ceph-mon[33110]: osdmap e465: 8 total, 8 up, 7 in 2024-03-26T03:20:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:18 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:20:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:18 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:20:18.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:18.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:18.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:18.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:18.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:19.027 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.056 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:19.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:19.157 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:19.184 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:19.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:19.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:19.413 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:19.440 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:19.542 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:19.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: pgmap v1008: 178 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 5 peering, 166 active+clean; 128 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 3.8 KiB/s rd, 26 KiB/s wr, 854 op/s; 584/6801 objects misplaced (8.587%); 5.5 MiB/s, 106 objects/s recovering 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:20:19.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:20:19.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:20:19.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: pgmap v1008: 178 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 5 peering, 166 active+clean; 128 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 3.8 KiB/s rd, 26 KiB/s wr, 854 op/s; 584/6801 objects misplaced (8.587%); 5.5 MiB/s, 106 objects/s recovering 2024-03-26T03:20:19.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:20:19.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:19 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:20:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.571 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:19.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:19.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:19.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: pgmap v1008: 178 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 5 peering, 166 active+clean; 128 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 3.8 KiB/s rd, 26 KiB/s wr, 854 op/s; 584/6801 objects misplaced (8.587%); 5.5 MiB/s, 106 objects/s recovering 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:20:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:19 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:20:19.799 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:19.831 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:19.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:19.850 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:280126 9.5 9:a020b14b:::benchmark_data_smithi066_125102_object133825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:02.244060+0000", 2024-03-26T03:20:19.851 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 497.60502010900001, 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.060487632999999999, 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 280126 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244060+0000", 2024-03-26T03:20:19.852 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244060+0000", 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244062+0000", 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5910000000000001e-06 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.853 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244067+0000", 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6010000000000002e-06 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244084+0000", 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6582000000000001e-05 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.854 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.855 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244093+0000", 2024-03-26T03:20:19.855 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2799999999999992e-06 2024-03-26T03:20:19.855 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.855 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.855 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244154+0000", 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1308000000000003e-05 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244285+0000", 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000130147 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.856 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.244402+0000", 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117142 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.251689+0000", 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0072876080000000001 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.290229+0000", 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.038539963000000003 2024-03-26T03:20:19.857 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304414+0000", 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014184275999999999 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304479+0000", 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5512000000000006e-05 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.858 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304548+0000", 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8622999999999997e-05 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:280129 9.5 9:adb46974:::benchmark_data_smithi066_125102_object133828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:02.249127+0000", 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 497.599953368, 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055577005999999998, 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.859 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 280129 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249127+0000", 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.860 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249127+0000", 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249129+0000", 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0999999999999998e-06 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249135+0000", 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3130000000000004e-06 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.861 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249147+0000", 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1406999999999999e-05 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249155+0000", 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2579999999999995e-06 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249197+0000", 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2280000000000002e-05 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.862 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249297+0000", 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9832000000000002e-05 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.249388+0000", 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1291000000000004e-05 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.251902+0000", 2024-03-26T03:20:19.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002514124 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.296709+0000", 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.044806954000000003 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304609+0000", 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0079000600000000004 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304662+0000", 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2311000000000001e-05 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:02.304704+0000", 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2076000000000002e-05 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281395 9.5 9:a71a042d:::benchmark_data_smithi066_125102_object135094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.201446+0000", 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 496.64763393999999, 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.110794741, 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.866 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281395 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201446+0000", 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201446+0000", 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201448+0000", 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.415e-06 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201452+0000", 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6550000000000003e-06 2024-03-26T03:20:19.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201461+0000", 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6589999999999996e-06 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201469+0000", 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8650000000000001e-06 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201508+0000", 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9436e-05 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201596+0000", 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7429999999999995e-05 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.201675+0000", 2024-03-26T03:20:19.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9371999999999998e-05 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.204385+0000", 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0027099519999999998 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.209026+0000", 2024-03-26T03:20:19.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046409779999999996 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.312123+0000", 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10309655199999999 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.312198+0000", 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5066000000000003e-05 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.312241+0000", 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3361e-05 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281402 9.5 9:a3d91319:::benchmark_data_smithi066_125102_object135101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.205723+0000", 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 496.643357552, 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.11434588699999999, 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.873 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281402 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205723+0000", 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205723+0000", 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205724+0000", 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.124e-06 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.875 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.877 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.877 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205728+0000", 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2100000000000003e-06 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205737+0000", 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.4949999999999997e-06 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205746+0000", 2024-03-26T03:20:19.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1980000000000001e-06 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205801+0000", 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.486e-05 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.205984+0000", 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018371800000000001 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.206096+0000", 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011207800000000001 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.209983+0000", 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038868790000000002 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214312+0000", 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043288220000000004 2024-03-26T03:20:19.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.319873+0000", 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.10556109900000001 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320005+0000", 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013166400000000001 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320069+0000", 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3739999999999996e-05 2024-03-26T03:20:19.881 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281415 9.8 9:1daf5fd6:::benchmark_data_smithi066_125102_object135114:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.211863+0000", 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 496.637217561, 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055859276999999999, 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281415 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211863+0000", 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211863+0000", 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211864+0000", 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.751e-06 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211870+0000", 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4389999999999999e-06 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211885+0000", 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5216e-05 2024-03-26T03:20:19.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211898+0000", 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3339000000000001e-05 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.211950+0000", 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1882000000000002e-05 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.212067+0000", 2024-03-26T03:20:19.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011676300000000001 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5", 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.212531+0000", 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00046378700000000001 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.215891+0000", 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0033606780000000002 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267589+0000", 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.051697369 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267623+0000", 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3893999999999998e-05 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267722+0000", 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9159000000000004e-05 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281419 9.b 9:d26b497b:::benchmark_data_smithi066_125102_object135118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.214285+0000", 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 496.63479558, 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.056169565999999997, 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281419 2024-03-26T03:20:19.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214285+0000", 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214285+0000", 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214286+0000", 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.756e-06 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214292+0000", 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4489999999999998e-06 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214302+0000", 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7710000000000001e-06 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214310+0000", 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9389999999999997e-06 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214365+0000", 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5590999999999997e-05 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214502+0000", 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000136797 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,6", 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.214645+0000", 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014285799999999999 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.217148+0000", 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0025030439999999998 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.217314+0000", 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016595299999999999 2024-03-26T03:20:19.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270382+0000", 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.053067847000000001 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270417+0000", 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4962999999999997e-05 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270454+0000", 2024-03-26T03:20:19.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7598000000000002e-05 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281425 9.5 9:abc5880a:::benchmark_data_smithi066_125102_object135124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.267780+0000", 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 496.581300075, 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052647581999999998, 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.894 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281425 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267780+0000", 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267780+0000", 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267781+0000", 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1260000000000001e-06 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267786+0000", 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6589999999999998e-06 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267797+0000", 2024-03-26T03:20:19.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0679999999999999e-05 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267804+0000", 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6170000000000001e-06 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267858+0000", 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4046999999999999e-05 2024-03-26T03:20:19.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.267964+0000", 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010581099999999999 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.268038+0000", 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3696999999999996e-05 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270273+0000", 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022354649999999998 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.270644+0000", 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00037056600000000002 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320315+0000", 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.049671445000000002 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.899 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320380+0000", 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5005999999999994e-05 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.320428+0000", 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7463000000000003e-05 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:281777 9.5 9:a406715e:::benchmark_data_smithi066_125102_object135476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:03.547662+0000", 2024-03-26T03:20:19.900 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 496.30141826099998, 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.054275181999999998, 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 281777 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547662+0000", 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547662+0000", 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547663+0000", 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2399999999999997e-07 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547666+0000", 2024-03-26T03:20:19.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3500000000000001e-06 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547672+0000", 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2620000000000004e-06 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.547677+0000", 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5349999999999998e-06 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.549501+0000", 2024-03-26T03:20:19.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001823849 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.549581+0000", 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0561000000000003e-05 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.549659+0000", 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8082000000000002e-05 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.551034+0000", 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013747799999999999 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.576143+0000", 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.025109267000000001 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.601865+0000", 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.025721895000000002 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.601910+0000", 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4332999999999997e-05 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:03.601937+0000", 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7444e-05 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.906 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290173 9.9 9:90f0ff23:::benchmark_data_smithi066_125102_object143872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.532237+0000", 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 490.31684307, 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.061708225999999998, 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290173 2024-03-26T03:20:19.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532237+0000", 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532237+0000", 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532239+0000", 2024-03-26T03:20:19.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3920000000000001e-06 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532245+0000", 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8150000000000002e-06 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532260+0000", 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4543e-05 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532268+0000", 2024-03-26T03:20:19.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4710000000000007e-06 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532326+0000", 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7299999999999997e-05 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532472+0000", 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014627999999999999 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.910 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532590+0000", 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011835199999999999 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539034+0000", 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0064440890000000001 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.912 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539158+0000", 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012363599999999999 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.593818+0000", 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.054660042999999998 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.593896+0000", 2024-03-26T03:20:19.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7730000000000003e-05 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.593945+0000", 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9574999999999999e-05 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290174 9.9 9:971ef230:::benchmark_data_smithi066_125102_object143873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:20:19.914 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.532503+0000", 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 490.31657708500001, 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.062196251000000001, 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290174 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532503+0000", 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532503+0000", 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532504+0000", 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3489999999999999e-06 2024-03-26T03:20:19.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532509+0000", 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8459999999999999e-06 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532519+0000", 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7529999999999996e-06 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.532527+0000", 2024-03-26T03:20:19.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2220000000000002e-06 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535557+0000", 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030293690000000001 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535657+0000", 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100417 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535765+0000", 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010826 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539145+0000", 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003379309 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.539287+0000", 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014264 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.594557+0000", 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055269791999999998 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.594652+0000", 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5237999999999999e-05 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.594699+0000", 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7055999999999998e-05 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.920 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290180 9.5 9:a6fd05c4:::benchmark_data_smithi066_125102_object143879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.535448+0000", 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 490.31363243300001, 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.056686799000000003, 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.921 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290180 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535448+0000", 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535448+0000", 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535449+0000", 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0270000000000001e-06 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535453+0000", 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1899999999999997e-06 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535466+0000", 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3179e-05 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535473+0000", 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3509999999999996e-06 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535530+0000", 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6192e-05 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.924 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535647+0000", 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117118 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.535765+0000", 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118508 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.538830+0000", 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030647019999999999 2024-03-26T03:20:19.925 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.538974+0000", 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000144186 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.591996+0000", 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.053021336000000002 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.592081+0000", 2024-03-26T03:20:19.926 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5105999999999995e-05 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.592135+0000", 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3903999999999998e-05 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15360.0:290185 9.9 9:94769a65:::benchmark_data_smithi066_125102_object143884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:09.541601+0000", 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 490.30747947200001, 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.056043854999999997, 2024-03-26T03:20:19.927 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15360", 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 290185 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541601+0000", 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.928 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541601+0000", 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541603+0000", 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.858e-06 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541608+0000", 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.04e-06 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.929 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541619+0000", 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1257000000000001e-05 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541625+0000", 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3319999999999997e-06 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541676+0000", 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0759e-05 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.930 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541802+0000", 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012582000000000001 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.541943+0000", 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014159599999999999 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.544680+0000", 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0027369769999999998 2024-03-26T03:20:19.931 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.544784+0000", 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000103658 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.597572+0000", 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052788195000000003 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.597606+0000", 2024-03-26T03:20:19.932 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3733999999999999e-05 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:09.597645+0000", 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8628999999999999e-05 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:177 10.6 10:6dd0b257:::benchmark_data_smithi066_148944_object176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:12:53.173704+0000", 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 446.67537594200002, 2024-03-26T03:20:19.933 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.054475559999999999, 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 177 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173704+0000", 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.934 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173704+0000", 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173706+0000", 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5149999999999999e-06 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.935 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173726+0000", 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0684999999999999e-05 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173737+0000", 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0604999999999999e-05 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173745+0000", 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9100000000000005e-06 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.936 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173789+0000", 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3627000000000001e-05 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.173929+0000", 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014060900000000001 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5", 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.174328+0000", 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00039881500000000001 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.937 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.174823+0000", 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00049515300000000002 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.175595+0000", 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00077191399999999996 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.228066+0000", 2024-03-26T03:20:19.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.052470623000000001 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.228136+0000", 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0370999999999994e-05 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:12:53.228180+0000", 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3733000000000003e-05 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:62024 10.8 10:13aea929:::benchmark_data_smithi066_148944_object62023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:20.143956+0000", 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 419.70512400899997, 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.055962244000000001, 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62024 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.940 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143956+0000", 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143956+0000", 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143958+0000", 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3039999999999999e-06 2024-03-26T03:20:19.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.143997+0000", 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8958999999999997e-05 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144024+0000", 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6916000000000001e-05 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144034+0000", 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2609999999999999e-06 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144090+0000", 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6790999999999999e-05 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.144225+0000", 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000134707 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:20:19.943 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.191365+0000", 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.047139970000000003 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.196677+0000", 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053120759999999998 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199505+0000", 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028275449999999999 2024-03-26T03:20:19.944 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199813+0000", 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00030792800000000001 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199860+0000", 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7327000000000003e-05 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199918+0000", 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8459999999999999e-05 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:62027 10.8 10:1d6ddba7:::benchmark_data_smithi066_148944_object62026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:20.146816+0000", 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 419.702264496, 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.053254153999999998, 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.946 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 62027 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146816+0000", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146816+0000", 2024-03-26T03:20:19.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146818+0000", 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1610000000000001e-06 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.146845+0000", 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7620999999999999e-05 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.191260+0000", 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.044414431999999997 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.191275+0000", 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4673e-05 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.194261+0000", 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0029868780000000001 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.194419+0000", 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000157625 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.194607+0000", 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000188395 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199510+0000", 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004902689 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199699+0000", 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00018911399999999999 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.199966+0000", 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00026690399999999999 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.200031+0000", 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4632000000000001e-05 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.951 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:20.200070+0000", 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9029999999999997e-05 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:88937 10.c e319/314 10:344b2b03:::benchmark_data_smithi066_148944_object88936:head v 319'5454, mlcod=319'5452)", 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.555823+0000", 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 397.29325727399998, 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.042912618, 2024-03-26T03:20:19.952 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555823+0000", 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555823+0000", 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555824+0000", 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.468e-06 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.555864+0000", 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9490000000000003e-05 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591050+0000", 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.035186533999999998 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591061+0000", 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1058e-05 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591114+0000", 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2670000000000002e-05 2024-03-26T03:20:19.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591136+0000", 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2340999999999998e-05 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.597711+0000", 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.006574781 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.598736+0000", 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001024276 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:88939 10.4 e319/314 10:25b58809:::benchmark_data_smithi066_148944_object88938:head v 319'5508, mlcod=319'5506)", 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:13:42.557928+0000", 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 397.29115176900001, 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.042445721999999998, 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557928+0000", 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557928+0000", 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557930+0000", 2024-03-26T03:20:19.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.474e-06 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.557989+0000", 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9305000000000002e-05 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591131+0000", 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033141362000000001 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591141+0000", 2024-03-26T03:20:19.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0163000000000001e-05 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591412+0000", 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00027127500000000002 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.591437+0000", 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4924000000000001e-05 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.600112+0000", 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0086753009999999998 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:13:42.600374+0000", 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00026191799999999998 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.961 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_repop(client.25522.0:188973 10.4 e363/331 10:28715c38:::benchmark_data_smithi066_148944_object64326:head v 363'11663, mlcod=362'11659)", 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:32.576404+0000", 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 287.27267630799997, 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.040717334000000001, 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576404+0000", 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576404+0000", 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576405+0000", 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.16e-07 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576411+0000", 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2389999999999999e-06 2024-03-26T03:20:19.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576418+0000", 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5590000000000002e-06 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.576424+0000", 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4570000000000002e-06 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.580727+0000", 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043029330000000001 2024-03-26T03:20:19.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.580790+0000", 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2564000000000006e-05 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.616987+0000", 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.036197571999999997 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:32.617121+0000", 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013409399999999999 2024-03-26T03:20:19.965 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "rep_scrubmap(6.18 e459 from shard 1)", 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:20:17.859526+0000", 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 1.9895543120000001, 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.049292913000000001, 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.859526+0000", 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.859526+0000", 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.859527+0000", 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.006e-06 2024-03-26T03:20:19.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.859532+0000", 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5969999999999999e-06 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.859546+0000", 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3331e-05 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.859553+0000", 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9489999999999999e-06 2024-03-26T03:20:19.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.908788+0000", 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.049235510000000003 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.908800+0000", 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1439e-05 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.908819+0000", 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9080999999999999e-05 2024-03-26T03:20:19.969 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "rep_scrubmap(6.18 e459 from shard 2)", 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:20:17.862498+0000", 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 1.9865821239999999, 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.046393946999999998, 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "started", 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:19.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.862498+0000", 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.862498+0000", 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.862499+0000", 2024-03-26T03:20:19.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3030000000000001e-06 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.862505+0000", 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5380000000000002e-06 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.862513+0000", 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8699999999999992e-06 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.862520+0000", 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8059999999999997e-06 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.908845+0000", 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.046325676000000003 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.908855+0000", 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9399999999999997e-06 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:19.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:20:17.908892+0000", 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6813999999999997e-05 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:19.974 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:20:19.974 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.002 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:20.103 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.131 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:20.233 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:20.260 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:20.362 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.391 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:20.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:20:20.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 2.7 deep-scrub starts 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 2.7 deep-scrub ok 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:20:20.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:20 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:20:20.491 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:20.520 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:20.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:20:20.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:20:20.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 2.7 deep-scrub starts 2024-03-26T03:20:20.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 2.7 deep-scrub ok 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 2.7 deep-scrub starts 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 2.7 deep-scrub ok 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:20:20.573 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:20:20.574 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:20:20.574 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:20 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:20:20.623 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:20.651 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:20.742 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:20:20.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:20.813 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:20.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:20.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:20.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:21.045 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.074 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:21.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:21.175 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:21.203 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:21.305 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:21.333 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:21.435 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:21.463 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:21.485 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:21.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:21.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:21.586 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[29666]: pgmap v1009: 178 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 175 active+clean; 23 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 878 op/s; 82/1773 objects misplaced (4.625%); 6.5 MiB/s, 117 objects/s recovering 2024-03-26T03:20:21.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[29666]: 4.10 deep-scrub starts 2024-03-26T03:20:21.587 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[29666]: 4.10 deep-scrub ok 2024-03-26T03:20:21.587 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:21.587 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[37920]: pgmap v1009: 178 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 175 active+clean; 23 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 878 op/s; 82/1773 objects misplaced (4.625%); 6.5 MiB/s, 117 objects/s recovering 2024-03-26T03:20:21.587 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[37920]: 4.10 deep-scrub starts 2024-03-26T03:20:21.587 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:21 smithi066 ceph-mon[37920]: 4.10 deep-scrub ok 2024-03-26T03:20:21.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:21.596 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:20:21.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:21.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:21 smithi192 ceph-mon[33110]: pgmap v1009: 178 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 175 active+clean; 23 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 2.7 KiB/s rd, 18 KiB/s wr, 878 op/s; 82/1773 objects misplaced (4.625%); 6.5 MiB/s, 117 objects/s recovering 2024-03-26T03:20:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:21 smithi192 ceph-mon[33110]: 4.10 deep-scrub starts 2024-03-26T03:20:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:21 smithi192 ceph-mon[33110]: 4.10 deep-scrub ok 2024-03-26T03:20:21.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:21.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:21.941 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:20:21.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:21.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:21.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:22.080 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:22.108 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:22.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:22.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:22.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:22.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:22.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:22.454 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:22.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:22.493 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-26T03:20:22.494 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-26T03:20:22.503 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:22.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[29666]: 1.18 scrub starts 2024-03-26T03:20:22.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[29666]: 1.18 scrub ok 2024-03-26T03:20:22.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:20:22.531 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:20:22.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[37920]: 1.18 scrub starts 2024-03-26T03:20:22.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[37920]: 1.18 scrub ok 2024-03-26T03:20:22.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:20:22.532 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:22 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:20:22.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:22.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: "epoch": 465, 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: "fsid": "eba7a548-eb1b-11ee-95d0-87774f69a715", 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: "created": "2024-03-26T02:53:28.887014+0000", 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: "modified": "2024-03-26T03:20:17.290773+0000", 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: "last_up_change": "2024-03-26T03:14:20.518826+0000", 2024-03-26T03:20:22.663 INFO:teuthology.orchestra.run.smithi066.stdout: "last_in_change": "2024-03-26T03:20:11.669367+0000", 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_num": 5799936, 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_set": [ 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "pglog_hardlimit", 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "purged_snapdirs", 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "recovery_deletes", 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "sortbitwise" 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_version": 19, 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_snapshot": "", 2024-03-26T03:20:22.664 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_max": 11, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "max_osd": 8, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "min_compat_client": "luminous", 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "require_osd_release": "squid", 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "allow_crimson": false, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "pools": [ 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 1, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".mgr", 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:56:43.176262+0000", 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.665 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 31, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 31, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 31, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 31, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 31, 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.666 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "305", 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.667 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "303", 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.668 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.669 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:20:22.670 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_max": 32, 2024-03-26T03:20:22.671 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_min": 1 2024-03-26T03:20:22.671 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.671 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.671 INFO:teuthology.orchestra.run.smithi066.stdout: "mgr": {} 2024-03-26T03:20:22.671 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.5299999713897705, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.5299999713897705, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.3500000238418579, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.3500000238418579, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.672 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 2, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": ".rgw.root", 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.077781+0000", 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.673 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.674 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "65", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.675 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.676 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.677 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.678 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2400000095367432, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2400000095367432, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 3, 2024-03-26T03:20:22.679 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "datapool", 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:55.118069+0000", 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 8193, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 3, 2024-03-26T03:20:22.680 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 3, 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 3, 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 3, 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 3, 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.681 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "68", 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:20:22.682 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 2, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 68, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.683 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.684 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "rbd": {} 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.6500000953674316, 2024-03-26T03:20:22.685 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.6500000953674316, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 4, 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:20:22.686 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:56.987641+0000", 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:20:22.687 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "67", 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.688 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.689 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.690 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.691 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.2400000095367432, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.2400000095367432, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 5, 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:58:58.940665+0000", 2024-03-26T03:20:22.692 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:20:22.693 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "69", 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.694 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.695 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.696 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:20:22.697 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 1.75, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 1.75, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 6, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T02:59:01.006117+0000", 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:20:22.698 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 32, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 32, 2024-03-26T03:20:22.699 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 32, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 32, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 32, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "71", 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.700 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.701 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.702 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "options": { 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_bias": 4 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "rgw": {} 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.703 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.4900000095367432, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.4900000095367432, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "pool": 11, 2024-03-26T03:20:22.704 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_name": "unique_pool_4", 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "create_time": "2024-03-26T03:16:54.948187+0000", 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "flags": 1, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "flags_names": "hashpspool", 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "type": 1, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 3, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "min_size": 2, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_rule": 0, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "object_hash": 2, 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:20:22.705 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num": 16, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num": 16, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_placement_num_target": 16, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_target": 16, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_num_pending": 16, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "last_pg_merge_meta": { 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "source_pgid": "0.0", 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "ready_epoch": 0, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_started": 0, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "last_epoch_clean": 0, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "source_version": "0'0", 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "target_version": "0'0" 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.706 INFO:teuthology.orchestra.run.smithi066.stdout: "last_change": "409", 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend": "0", 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "auid": 0, 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_seq": 0, 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "snap_epoch": 0, 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "pool_snaps": [], 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps": "[]", 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_bytes": 0, 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "quota_max_objects": 0, 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "tiers": [], 2024-03-26T03:20:22.707 INFO:teuthology.orchestra.run.smithi066.stdout: "tier_of": -1, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "read_tier": -1, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "write_tier": -1, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_mode": "none", 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_bytes": 0, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "target_max_objects": 0, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_flush_age": 0, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "cache_min_evict_age": 0, 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profile": "", 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_params": { 2024-03-26T03:20:22.708 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "none" 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_period": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_count": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "use_gmt_hitset": true, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "grade_table": [], 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "stripe_width": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "expected_num_objects": 0, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "fast_read": false, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "options": {}, 2024-03-26T03:20:22.709 INFO:teuthology.orchestra.run.smithi066.stdout: "application_metadata": { 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "rados": {} 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "read_balance": { 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "score_type": "Fair distribution", 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "score_acting": 2.4900000095367432, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "score_stable": 2.4900000095367432, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity_weighted": 0.82522201538085938, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-26T03:20:22.710 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "90c9bf50-0b39-4fd1-9f00-b02800f07bf9", 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 8, 2024-03-26T03:20:22.711 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 464, 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6801", 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.712 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6802", 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6804", 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.713 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6803", 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1028658447 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6801/1028658447", 2024-03-26T03:20:22.714 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6802/1028658447", 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6804/1028658447", 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6803/1028658447", 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9", 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.715 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 13, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 463, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6805", 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.716 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6806", 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.717 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6808", 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6807", 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3170415936 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.718 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6805/3170415936", 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6806/3170415936", 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6808/3170415936", 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6807/3170415936", 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:20:22.719 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "76639ea3-00c7-4693-9eb2-3421f234283b", 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.005157470703125, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 18, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 461, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.720 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6809", 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6810", 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.721 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6812", 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.722 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6811", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2520642381 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6809/2520642381", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6810/2520642381", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6812/2520642381", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6811/2520642381", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.723 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2", 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 26, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 197, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 331, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 464, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 198, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.724 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6813", 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6814", 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.725 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6816", 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.726 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.66:6815", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 52636497 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.66:6813/52636497", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.66:6814/52636497", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.66:6816/52636497", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.66:6815/52636497", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.727 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "4fae794b-e89c-4931-9ed3-0cc32b445883", 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 31, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 463, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.728 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6800", 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.729 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6801", 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6803", 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.730 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6802", 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 1766416869 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6800/1766416869", 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6801/1766416869", 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6803/1766416869", 2024-03-26T03:20:22.731 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6802/1766416869", 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "04b178c3-9be8-466c-822c-0acfca8a92b0", 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 0.59661865234375, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.732 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 40, 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 462, 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6804", 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:20:22.733 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.734 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.734 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.734 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.734 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.734 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.734 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6805", 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6807", 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.735 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6806", 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 2546913262 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6804/2546913262", 2024-03-26T03:20:22.736 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6805/2546913262", 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2546913262", 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6806/2546913262", 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "fc63b52a-e4fe-4396-bf54-2e38696aedcb", 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 1, 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 1, 2024-03-26T03:20:22.737 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 48, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 462, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6808", 2024-03-26T03:20:22.738 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6809", 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.739 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6811", 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6810", 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 4039268292 2024-03-26T03:20:22.740 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6808/4039268292", 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6809/4039268292", 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6811/4039268292", 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6810/4039268292", 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.741 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "uuid": "e2f4cedd-43af-4261-9f86-c4b72c97bd1b", 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "up": 1, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "in": 0, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "weight": 0, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_affinity": 1, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_begin": 0, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "last_clean_end": 0, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "up_from": 58, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "up_thru": 341, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "down_at": 0, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "lost_at": 0, 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addrs": { 2024-03-26T03:20:22.742 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6812", 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addrs": { 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6813", 2024-03-26T03:20:22.743 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addrs": { 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6815", 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addrs": { 2024-03-26T03:20:22.744 INFO:teuthology.orchestra.run.smithi066.stdout: "addrvec": [ 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "type": "v1", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "addr": "172.21.15.192:6814", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "nonce": 3100313121 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "public_addr": "172.21.15.192:6812/3100313121", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "cluster_addr": "172.21.15.192:6813/3100313121", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3100313121", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "heartbeat_front_addr": "172.21.15.192:6814/3100313121", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "state": [ 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "exists", 2024-03-26T03:20:22.745 INFO:teuthology.orchestra.run.smithi066.stdout: "up" 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "osd_xinfo": [ 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 0, 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:07.294615+0000", 2024-03-26T03:20:22.746 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 1, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:45.527445+0000", 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.747 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 2, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:44.669457+0000", 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 3, 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.748 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:21.343317+0000", 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 197 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 4, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.749 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:38.710352+0000", 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 5, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:55.368779+0000", 2024-03-26T03:20:22.750 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 6, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 0, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:01.055265+0000", 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.751 INFO:teuthology.orchestra.run.smithi066.stdout: "osd": 7, 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "down_stamp": "0.000000", 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_probability": 0, 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "laggy_interval": 0, 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "features": 4540701547738038271, 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "old_weight": 65536, 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "last_purged_snaps_scrub": "2024-03-26T03:19:09.478599+0000", 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "dead_epoch": 0 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap": [ 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.f", 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:20:22.752 INFO:teuthology.orchestra.run.smithi066.stdout: 4, 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: 7 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_items": [ 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "1.12", 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 5, 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 4 2024-03-26T03:20:22.753 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.9", 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 5 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.754 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.a", 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "4.17", 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:20:22.755 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 2 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "5.1d", 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: "mappings": [ 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: "from": 1, 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: "to": 6 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_upmap_primaries": [], 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: "pg_temp": [ 2024-03-26T03:20:22.756 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "11.1", 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: 6, 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: 3, 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: 2 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: "pgid": "11.f", 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: "osds": [ 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: 1, 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: 2, 2024-03-26T03:20:22.757 INFO:teuthology.orchestra.run.smithi066.stdout: 4 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: ], 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "primary_temp": [], 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "blocklist": { 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2473255846": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3359773304": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3105618869": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/1321389422": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3068874001": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2719981294": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/858010278": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2558488368": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:20:22.758 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2101239378": "2024-03-27T02:59:37.893358+0000", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/2641107787": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/5961410": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/3720619020": "2024-03-27T02:54:01.878928+0000", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:6800/2059693071": "2024-03-27T02:54:29.078830+0000", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "172.21.15.66:0/952321413": "2024-03-27T02:54:29.078830+0000" 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "range_blocklist": {}, 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "erasure_code_profiles": { 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "default": { 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "crush-failure-domain": "osd", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "k": "2", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "m": "1", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "plugin": "jerasure", 2024-03-26T03:20:22.759 INFO:teuthology.orchestra.run.smithi066.stdout: "technique": "reed_sol_van" 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "removed_snaps_queue": [], 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "new_removed_snaps": [], 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "new_purged_snaps": [], 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "crush_node_flags": {}, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "device_class_flags": {}, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode": { 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_enabled": false, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_bucket_count": 0, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: "stretch_mode_bucket": 0 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:22.760 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:20:22.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:22.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:22 smithi192 ceph-mon[33110]: 1.18 scrub starts 2024-03-26T03:20:22.762 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:22 smithi192 ceph-mon[33110]: 1.18 scrub ok 2024-03-26T03:20:22.762 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:22 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:20:22.762 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:22 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:20:22.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:22.890 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:22.917 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:22.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:23.019 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:23.047 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:23.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:23.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:23.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:23.306 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:23.408 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:23.435 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:23.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:23.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:23 smithi066 ceph-mon[29666]: pgmap v1010: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 996 B/s rd, 0 B/s wr, 391 op/s; 3.0 MiB/s, 54 objects/s recovering 2024-03-26T03:20:23.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:23 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1140776488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:20:23.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:23 smithi066 ceph-mon[37920]: pgmap v1010: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 996 B/s rd, 0 B/s wr, 391 op/s; 3.0 MiB/s, 54 objects/s recovering 2024-03-26T03:20:23.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:23 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1140776488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:20:23.666 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:23.693 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:23.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:23 smithi192 ceph-mon[33110]: pgmap v1010: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 996 B/s rd, 0 B/s wr, 391 op/s; 3.0 MiB/s, 54 objects/s recovering 2024-03-26T03:20:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:23 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1140776488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:20:23.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:23.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:23.845 INFO:tasks.thrashosds.thrasher:j is {'epoch': 465, 'fsid': 'eba7a548-eb1b-11ee-95d0-87774f69a715', 'created': '2024-03-26T02:53:28.887014+0000', 'modified': '2024-03-26T03:20:17.290773+0000', 'last_up_change': '2024-03-26T03:14:20.518826+0000', 'last_in_change': '2024-03-26T03:20:11.669367+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:56:43.176262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '305', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '303', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5299999713897705, 'score_stable': 1.5299999713897705, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.350000023841858, 'raw_score_stable': 1.350000023841858, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T02:58:55.077781+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-26T02:58:55.118069+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T02:58:56.987641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T02:58:58.940665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T02:59:01.006117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-26T03:16:54.948187+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '409', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8252220153808594, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '90c9bf50-0b39-4fd1-9f00-b02800f07bf9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 464, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6801', 'nonce': 1028658447}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6802', 'nonce': 1028658447}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6804', 'nonce': 1028658447}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6803', 'nonce': 1028658447}]}, 'public_addr': '172.21.15.66:6801/1028658447', 'cluster_addr': '172.21.15.66:6802/1028658447', 'heartbeat_back_addr': '172.21.15.66:6804/1028658447', 'heartbeat_front_addr': '172.21.15.66:6803/1028658447', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 463, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6805', 'nonce': 3170415936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6806', 'nonce': 3170415936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6808', 'nonce': 3170415936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6807', 'nonce': 3170415936}]}, 'public_addr': '172.21.15.66:6805/3170415936', 'cluster_addr': '172.21.15.66:6806/3170415936', 'heartbeat_back_addr': '172.21.15.66:6808/3170415936', 'heartbeat_front_addr': '172.21.15.66:6807/3170415936', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '76639ea3-00c7-4693-9eb2-3421f234283b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.005157470703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 461, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6809', 'nonce': 2520642381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6810', 'nonce': 2520642381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6812', 'nonce': 2520642381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6811', 'nonce': 2520642381}]}, 'public_addr': '172.21.15.66:6809/2520642381', 'cluster_addr': '172.21.15.66:6810/2520642381', 'heartbeat_back_addr': '172.21.15.66:6812/2520642381', 'heartbeat_front_addr': '172.21.15.66:6811/2520642381', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 197, 'up_from': 331, 'up_thru': 464, 'down_at': 198, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6813', 'nonce': 52636497}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6814', 'nonce': 52636497}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6816', 'nonce': 52636497}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.66:6815', 'nonce': 52636497}]}, 'public_addr': '172.21.15.66:6813/52636497', 'cluster_addr': '172.21.15.66:6814/52636497', 'heartbeat_back_addr': '172.21.15.66:6816/52636497', 'heartbeat_front_addr': '172.21.15.66:6815/52636497', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '4fae794b-e89c-4931-9ed3-0cc32b445883', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 463, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6800', 'nonce': 1766416869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 1766416869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 1766416869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6802', 'nonce': 1766416869}]}, 'public_addr': '172.21.15.192:6800/1766416869', 'cluster_addr': '172.21.15.192:6801/1766416869', 'heartbeat_back_addr': '172.21.15.192:6803/1766416869', 'heartbeat_front_addr': '172.21.15.192:6802/1766416869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '04b178c3-9be8-466c-822c-0acfca8a92b0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.59661865234375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6804', 'nonce': 2546913262}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2546913262}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2546913262}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6806', 'nonce': 2546913262}]}, 'public_addr': '172.21.15.192:6804/2546913262', 'cluster_addr': '172.21.15.192:6805/2546913262', 'heartbeat_back_addr': '172.21.15.192:6807/2546913262', 'heartbeat_front_addr': '172.21.15.192:6806/2546913262', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fc63b52a-e4fe-4396-bf54-2e38696aedcb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6808', 'nonce': 4039268292}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 4039268292}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 4039268292}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6810', 'nonce': 4039268292}]}, 'public_addr': '172.21.15.192:6808/4039268292', 'cluster_addr': '172.21.15.192:6809/4039268292', 'heartbeat_back_addr': '172.21.15.192:6811/4039268292', 'heartbeat_front_addr': '172.21.15.192:6810/4039268292', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e2f4cedd-43af-4261-9f86-c4b72c97bd1b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6812', 'nonce': 3100313121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3100313121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3100313121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.192:6814', 'nonce': 3100313121}]}, 'public_addr': '172.21.15.192:6812/3100313121', 'cluster_addr': '172.21.15.192:6813/3100313121', 'heartbeat_back_addr': '172.21.15.192:6815/3100313121', 'heartbeat_front_addr': '172.21.15.192:6814/3100313121', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:07.294615+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:45.527445+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:44.669457+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:21.343317+0000', 'dead_epoch': 197}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:38.710352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:55.368779+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:19:01.055265+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T03:19:09.478599+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [4, 1, 7]}], 'pg_upmap_items': [{'pgid': '1.12', 'mappings': [{'from': 5, 'to': 4}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.1d', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.1', 'osds': [6, 3, 2]}, {'pgid': '11.f', 'osds': [1, 2, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.66:6800/2473255846': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3359773304': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3105618869': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/1321389422': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/3068874001': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2719981294': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/858010278': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:6800/2558488368': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/2101239378': '2024-03-27T02:59:37.893358+0000', '172.21.15.66:0/2641107787': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/5961410': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:0/3720619020': '2024-03-27T02:54:01.878928+0000', '172.21.15.66:6800/2059693071': '2024-03-27T02:54:29.078830+0000', '172.21.15.66:0/952321413': '2024-03-27T02:54:29.078830+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:20:23.845 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:20:23.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:23.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:23.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:24.054 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:24.063 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:20:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:20:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:20:24.082 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:24.184 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:24.212 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:24.311 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:24.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:24.316 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[29666]: 2.9 scrub starts 2024-03-26T03:20:24.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[29666]: 2.9 scrub ok 2024-03-26T03:20:24.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[29666]: 5.16 scrub starts 2024-03-26T03:20:24.317 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[29666]: 5.16 scrub ok 2024-03-26T03:20:24.321 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:24.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:24.441 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:24.469 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:24.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:24.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[37920]: 2.9 scrub starts 2024-03-26T03:20:24.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[37920]: 2.9 scrub ok 2024-03-26T03:20:24.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[37920]: 5.16 scrub starts 2024-03-26T03:20:24.593 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:24 smithi066 ceph-mon[37920]: 5.16 scrub ok 2024-03-26T03:20:24.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:24.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:24.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:24 smithi192 ceph-mon[33110]: 2.9 scrub starts 2024-03-26T03:20:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:24 smithi192 ceph-mon[33110]: 2.9 scrub ok 2024-03-26T03:20:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:24 smithi192 ceph-mon[33110]: 5.16 scrub starts 2024-03-26T03:20:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:24 smithi192 ceph-mon[33110]: 5.16 scrub ok 2024-03-26T03:20:24.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:24.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:24.958 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:24.987 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:24.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:25.089 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.117 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:25.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:25.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:25.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:25.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:25.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:25.473 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:25.501 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:25.603 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: 1.1b scrub starts 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: 1.1b scrub ok 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: pgmap v1011: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 247 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:25.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: 1.1b scrub starts 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: 1.1b scrub ok 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: pgmap v1011: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 247 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:25.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:25.630 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:25.732 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: 1.1b scrub starts 2024-03-26T03:20:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: 1.1b scrub ok 2024-03-26T03:20:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:20:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:20:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: pgmap v1011: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 247 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-03-26T03:20:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:25.759 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:25.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:25.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:25.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:25.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:26.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:26.123 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:26.151 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:26.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:26.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:26.358 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[29666]: 4.17 scrub starts 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[29666]: 4.17 scrub ok 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[37920]: 4.17 scrub starts 2024-03-26T03:20:26.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:26 smithi066 ceph-mon[37920]: 4.17 scrub ok 2024-03-26T03:20:26.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:26.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:26.512 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:26.541 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:26.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:26.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:26 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:20:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:26 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:20:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:26 smithi192 ceph-mon[33110]: 4.17 scrub starts 2024-03-26T03:20:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:26 smithi192 ceph-mon[33110]: 4.17 scrub ok 2024-03-26T03:20:26.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:26.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:26.902 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:26.930 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:26.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:27.032 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:27.059 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:27.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:27.188 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:27.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:27.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:27.408 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:20:27.420 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:27.428 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:20:27.452 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:27.466 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:20:27.488 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:20:27.508 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:20:27.529 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: osdmap e466: 8 total, 8 up, 7 in 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: pgmap v1013: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 220 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-26T03:20:27.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: 2.13 scrub starts 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[29666]: 2.13 scrub ok 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: osdmap e466: 8 total, 8 up, 7 in 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: pgmap v1013: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 220 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: 2.13 scrub starts 2024-03-26T03:20:27.530 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:27 smithi066 ceph-mon[37920]: 2.13 scrub ok 2024-03-26T03:20:27.549 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:20:27.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:27.569 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:20:27.571 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:20:27.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:27.683 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:27.711 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:20:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:20:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:20:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:20:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: osdmap e466: 8 total, 8 up, 7 in 2024-03-26T03:20:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: pgmap v1013: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 220 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-26T03:20:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: 2.13 scrub starts 2024-03-26T03:20:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:27 smithi192 ceph-mon[33110]: 2.13 scrub ok 2024-03-26T03:20:27.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:27.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:27.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:27.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:27.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:28.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:28.100 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:28.202 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:28.229 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:28.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:28.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:28.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:28.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:28.590 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:28.606 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:28 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3801135637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:28.606 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:28 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3801135637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:28.618 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:28.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:28 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3801135637' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:28.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:28.849 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:28.877 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:28.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:28.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:29.109 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:29.137 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:29.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:29.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:29.369 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:29.396 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:29.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:29.524 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:29.527 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 1.15 scrub starts 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 1.15 scrub ok 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: pgmap v1014: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 202 op/s; 1.6 MiB/s, 25 objects/s recovering 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 2.1 scrub starts 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 2.1 scrub ok 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 5.4 deep-scrub starts 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[37920]: 5.4 deep-scrub ok 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 1.15 scrub starts 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 1.15 scrub ok 2024-03-26T03:20:29.528 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: pgmap v1014: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 202 op/s; 1.6 MiB/s, 25 objects/s recovering 2024-03-26T03:20:29.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 2.1 scrub starts 2024-03-26T03:20:29.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 2.1 scrub ok 2024-03-26T03:20:29.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 5.4 deep-scrub starts 2024-03-26T03:20:29.529 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:29 smithi066 ceph-mon[29666]: 5.4 deep-scrub ok 2024-03-26T03:20:29.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:29.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 1.15 scrub starts 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 1.15 scrub ok 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: pgmap v1014: 178 pgs: 2 active+remapped+backfill_toofull, 176 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s rd, 0 B/s wr, 202 op/s; 1.6 MiB/s, 25 objects/s recovering 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 2.1 scrub starts 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 2.1 scrub ok 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 5.4 deep-scrub starts 2024-03-26T03:20:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:29 smithi192 ceph-mon[33110]: 5.4 deep-scrub ok 2024-03-26T03:20:29.756 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:29.784 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:29.886 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:29.915 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:29.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:30.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:20:30.038 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:30.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:20:30.147 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:30.175 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:30.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:30.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:30.379 INFO:teuthology.orchestra.run.smithi066.stderr:pool 'unique_pool_4' removed 2024-03-26T03:20:30.407 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 2.a scrub starts 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 2.a scrub ok 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 4.0 scrub starts 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 4.0 scrub ok 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3894399116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:20:30.420 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[37920]: from='client.36026 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 2.a scrub starts 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 2.a scrub ok 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 4.0 scrub starts 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 4.0 scrub ok 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3894399116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:20:30.421 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:30 smithi066 ceph-mon[29666]: from='client.36026 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:20:30.441 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:30.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:20:30.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 2.a scrub starts 2024-03-26T03:20:30.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 2.a scrub ok 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 4.0 scrub starts 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 4.0 scrub ok 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3894399116' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:20:30.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:30 smithi192 ceph-mon[33110]: from='client.36026 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:20:30.543 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:20:30.582 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:20:30.684 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:30.723 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:30.825 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:20:30.854 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:20:30.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:30.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:30.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:31.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:20:31.109 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:20:31.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:31.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:31.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:31.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:20:31.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:20:31.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:31.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:20:31.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:20:31.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[37920]: from='client.36026 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[37920]: osdmap e467: 8 total, 8 up, 7 in 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[37920]: pgmap v1016: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[29666]: from='client.36026 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[29666]: osdmap e467: 8 total, 8 up, 7 in 2024-03-26T03:20:31.495 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:31 smithi066 ceph-mon[29666]: pgmap v1016: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:20:31.558 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:20:31.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:31.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:31.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:31.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:31 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:20:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:31 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:20:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:31 smithi192 ceph-mon[33110]: from='client.36026 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:20:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:31 smithi192 ceph-mon[33110]: osdmap e467: 8 total, 8 up, 7 in 2024-03-26T03:20:31.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:31 smithi192 ceph-mon[33110]: pgmap v1016: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:20:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:31.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:20:31.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:20:31.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:20:31.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-03-26T03:20:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:31.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:20:32.089 DEBUG:teuthology.orchestra.run.smithi066:> sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:20:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:20:32.118 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: docker: command not found 2024-03-26T03:20:32.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi066 with status 1: 'sudo docker kill -s 1 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:20:32.161 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-26T03:20:32.172 INFO:tasks.thrashosds:joining thrashosds 2024-03-26T03:20:32.249 INFO:teuthology.orchestra.run.smithi066.stdout:ERROR: (22) Invalid argument 2024-03-26T03:20:32.249 INFO:teuthology.orchestra.run.smithi066.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:20:32.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[29666]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T03:20:32.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:20:32.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[29666]: 1.4 scrub starts 2024-03-26T03:20:32.568 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[29666]: 1.4 scrub ok 2024-03-26T03:20:32.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[37920]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T03:20:32.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:20:32.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[37920]: 1.4 scrub starts 2024-03-26T03:20:32.569 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:32 smithi066 ceph-mon[37920]: 1.4 scrub ok 2024-03-26T03:20:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:32 smithi192 ceph-mon[33110]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-26T03:20:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:32 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:20:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:32 smithi192 ceph-mon[33110]: 1.4 scrub starts 2024-03-26T03:20:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:32 smithi192 ceph-mon[33110]: 1.4 scrub ok 2024-03-26T03:20:33.010 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:33.377 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is set 2024-03-26T03:20:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: 6.4 scrub starts 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: 6.4 scrub ok 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: 5.11 scrub starts 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: 5.11 scrub ok 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: pgmap v1017: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: 4.1d scrub starts 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: 4.1d scrub ok 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2042741175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:20:33.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-mon[33110]: from='client.36032 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:20:33.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: 6.4 scrub starts 2024-03-26T03:20:33.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: 6.4 scrub ok 2024-03-26T03:20:33.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: 5.11 scrub starts 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: 5.11 scrub ok 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: pgmap v1017: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: 4.1d scrub starts 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: 4.1d scrub ok 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2042741175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[29666]: from='client.36032 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: 6.4 scrub starts 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: 6.4 scrub ok 2024-03-26T03:20:33.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: 5.11 scrub starts 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: 5.11 scrub ok 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: pgmap v1017: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: 4.1d scrub starts 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: 4.1d scrub ok 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2042741175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:20:33.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-mon[37920]: from='client.36032 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:20:34.030 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:20:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:20:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:20:34.237 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:20:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=cleanup t=2024-03-26T03:20:33.960210918Z level=info msg="Completed cleanup jobs" duration=11.810115ms 2024-03-26T03:20:34.531 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:20:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:34 smithi192 ceph-mon[33110]: 4.4 scrub starts 2024-03-26T03:20:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:34 smithi192 ceph-mon[33110]: 4.4 scrub ok 2024-03-26T03:20:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:34 smithi192 ceph-mon[33110]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:20:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:34 smithi192 ceph-mon[33110]: from='client.36032 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:20:34.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:34 smithi192 ceph-mon[33110]: osdmap e468: 8 total, 8 up, 7 in 2024-03-26T03:20:34.758 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[29666]: 4.4 scrub starts 2024-03-26T03:20:34.758 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[29666]: 4.4 scrub ok 2024-03-26T03:20:34.758 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[29666]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:20:34.758 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[29666]: from='client.36032 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:20:34.758 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[29666]: osdmap e468: 8 total, 8 up, 7 in 2024-03-26T03:20:34.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[37920]: 4.4 scrub starts 2024-03-26T03:20:34.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[37920]: 4.4 scrub ok 2024-03-26T03:20:34.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[37920]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:20:34.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[37920]: from='client.36032 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:20:34.759 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:34 smithi066 ceph-mon[37920]: osdmap e468: 8 total, 8 up, 7 in 2024-03-26T03:20:35.354 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:20:35.358 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:20:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:35 smithi066 ceph-mon[29666]: pgmap v1019: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:20:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:35 smithi066 ceph-mon[37920]: pgmap v1019: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:20:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:35 smithi192 ceph-mon[33110]: pgmap v1019: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:20:36.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:36 smithi066 ceph-mon[29666]: from='client.16191 v1:172.21.15.66:0/4125554417' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:20:36.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:36 smithi066 ceph-mon[37920]: from='client.16191 v1:172.21.15.66:0/4125554417' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:20:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:36 smithi192 ceph-mon[33110]: from='client.16191 v1:172.21.15.66:0/4125554417' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:20:37.271 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1019,"stamp":"2024-03-26T03:20:34.479836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3124,"num_read_kb":2844,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":1254196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5328,"ondisk_log_size":5328,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":530,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6394232,"kb_used_data":274964,"kb_used_omap":13,"kb_used_meta":6119026,"kb_avail":649821832,"statfs":{"total":671965249536,"available":665417555968,"internally_reserved":0,"allocated":281563136,"data_stored":278993027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14014,"internal_metadata":6265882946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-263792,"num_write_kb":-8441281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-21226,"num_bytes_recovered":-1390739456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001630"},"pg_stats":[{"pgid":"1.d","version":"82'128","reported_seq":788,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948729+0000","last_change":"2024-03-26T03:16:31.121425+0000","last_active":"2024-03-26T03:20:26.948729+0000","last_peered":"2024-03-26T03:20:26.948729+0000","last_clean":"2024-03-26T03:20:26.948729+0000","last_became_active":"2024-03-26T03:14:40.174462+0000","last_became_peered":"2024-03-26T03:14:40.174462+0000","last_unstale":"2024-03-26T03:20:26.948729+0000","last_undegraded":"2024-03-26T03:20:26.948729+0000","last_fullsized":"2024-03-26T03:20:26.948729+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:31.121331+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:16:31.121331+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"342'10","reported_seq":507,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207788+0000","last_change":"2024-03-26T03:20:24.316048+0000","last_active":"2024-03-26T03:20:27.207788+0000","last_peered":"2024-03-26T03:20:27.207788+0000","last_clean":"2024-03-26T03:20:27.207788+0000","last_became_active":"2024-03-26T03:14:37.273000+0000","last_became_peered":"2024-03-26T03:14:37.273000+0000","last_unstale":"2024-03-26T03:20:27.207788+0000","last_undegraded":"2024-03-26T03:20:27.207788+0000","last_fullsized":"2024-03-26T03:20:27.207788+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'10","last_scrub_stamp":"2024-03-26T03:20:24.315879+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:20:24.315879+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:33.272404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"446'51","reported_seq":1026,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329303+0000","last_change":"2024-03-26T03:20:15.019691+0000","last_active":"2024-03-26T03:20:26.329303+0000","last_peered":"2024-03-26T03:20:26.329303+0000","last_clean":"2024-03-26T03:20:26.329303+0000","last_became_active":"2024-03-26T03:14:37.267205+0000","last_became_peered":"2024-03-26T03:14:37.267205+0000","last_unstale":"2024-03-26T03:20:26.329303+0000","last_undegraded":"2024-03-26T03:20:26.329303+0000","last_fullsized":"2024-03-26T03:20:26.329303+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'51","last_scrub_stamp":"2024-03-26T03:20:15.019461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:20:15.019461+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:33.964753+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":298,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330598+0000","last_change":"2024-03-26T03:17:57.225859+0000","last_active":"2024-03-26T03:20:26.330598+0000","last_peered":"2024-03-26T03:20:26.330598+0000","last_clean":"2024-03-26T03:20:26.330598+0000","last_became_active":"2024-03-26T03:14:37.264771+0000","last_became_peered":"2024-03-26T03:14:37.264771+0000","last_unstale":"2024-03-26T03:20:26.330598+0000","last_undegraded":"2024-03-26T03:20:26.330598+0000","last_fullsized":"2024-03-26T03:20:26.330598+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.225755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.225755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":709,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207503+0000","last_change":"2024-03-26T03:20:12.720618+0000","last_active":"2024-03-26T03:20:27.207503+0000","last_peered":"2024-03-26T03:20:27.207503+0000","last_clean":"2024-03-26T03:20:27.207503+0000","last_became_active":"2024-03-26T03:20:12.720274+0000","last_became_peered":"2024-03-26T03:20:12.720274+0000","last_unstale":"2024-03-26T03:20:27.207503+0000","last_undegraded":"2024-03-26T03:20:27.207503+0000","last_fullsized":"2024-03-26T03:20:27.207503+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:18:07.660520+0000","last_clean_scrub_stamp":"2024-03-26T03:18:07.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":393,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948307+0000","last_change":"2024-03-26T03:19:16.437899+0000","last_active":"2024-03-26T03:20:26.948307+0000","last_peered":"2024-03-26T03:20:26.948307+0000","last_clean":"2024-03-26T03:20:26.948307+0000","last_became_active":"2024-03-26T03:17:59.592113+0000","last_became_peered":"2024-03-26T03:17:59.592113+0000","last_unstale":"2024-03-26T03:20:26.948307+0000","last_undegraded":"2024-03-26T03:20:26.948307+0000","last_fullsized":"2024-03-26T03:20:26.948307+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_clean_scrub_stamp":"2024-03-26T03:19:16.437810+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:44.087975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"446'32","reported_seq":37,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.355050+0000","last_change":"2024-03-26T03:20:12.770910+0000","last_active":"2024-03-26T03:20:30.355050+0000","last_peered":"2024-03-26T03:20:30.355050+0000","last_clean":"2024-03-26T03:20:30.355050+0000","last_became_active":"2024-03-26T03:20:12.725584+0000","last_became_peered":"2024-03-26T03:20:12.725584+0000","last_unstale":"2024-03-26T03:20:30.355050+0000","last_undegraded":"2024-03-26T03:20:30.355050+0000","last_fullsized":"2024-03-26T03:20:30.355050+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'26","last_scrub_stamp":"2024-03-26T03:17:38.464358+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:17:38.464358+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":879,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546368+0000","last_change":"2024-03-26T03:17:56.571530+0000","last_active":"2024-03-26T03:20:30.546368+0000","last_peered":"2024-03-26T03:20:30.546368+0000","last_clean":"2024-03-26T03:20:30.546368+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:20:30.546368+0000","last_undegraded":"2024-03-26T03:20:30.546368+0000","last_fullsized":"2024-03-26T03:20:30.546368+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:16.933002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:15:16.933002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":88,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947536+0000","last_change":"2024-03-26T03:17:56.568649+0000","last_active":"2024-03-26T03:20:26.947536+0000","last_peered":"2024-03-26T03:20:26.947536+0000","last_clean":"2024-03-26T03:20:26.947536+0000","last_became_active":"2024-03-26T03:17:56.568322+0000","last_became_peered":"2024-03-26T03:17:56.568322+0000","last_unstale":"2024-03-26T03:20:26.947536+0000","last_undegraded":"2024-03-26T03:20:26.947536+0000","last_fullsized":"2024-03-26T03:20:26.947536+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.222175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.222175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353696+0000","last_change":"2024-03-26T03:20:12.743615+0000","last_active":"2024-03-26T03:20:30.353696+0000","last_peered":"2024-03-26T03:20:30.353696+0000","last_clean":"2024-03-26T03:20:30.353696+0000","last_became_active":"2024-03-26T03:20:12.743303+0000","last_became_peered":"2024-03-26T03:20:12.743303+0000","last_unstale":"2024-03-26T03:20:30.353696+0000","last_undegraded":"2024-03-26T03:20:30.353696+0000","last_fullsized":"2024-03-26T03:20:30.353696+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:55.664193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:19:55.664193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:13.574000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":972,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208332+0000","last_change":"2024-03-26T03:20:13.215674+0000","last_active":"2024-03-26T03:20:27.208332+0000","last_peered":"2024-03-26T03:20:27.208332+0000","last_clean":"2024-03-26T03:20:27.208332+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:20:27.208332+0000","last_undegraded":"2024-03-26T03:20:27.208332+0000","last_fullsized":"2024-03-26T03:20:27.208332+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:13.215586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:13.215586+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:22.436908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"446'57","reported_seq":1031,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207990+0000","last_change":"2024-03-26T03:18:02.671958+0000","last_active":"2024-03-26T03:20:27.207990+0000","last_peered":"2024-03-26T03:20:27.207990+0000","last_clean":"2024-03-26T03:20:27.207990+0000","last_became_active":"2024-03-26T03:14:37.266473+0000","last_became_peered":"2024-03-26T03:14:37.266473+0000","last_unstale":"2024-03-26T03:20:27.207990+0000","last_undegraded":"2024-03-26T03:20:27.207990+0000","last_fullsized":"2024-03-26T03:20:27.207990+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'41","last_scrub_stamp":"2024-03-26T03:18:02.671821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:02.671821+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353201+0000","last_change":"2024-03-26T03:20:12.740287+0000","last_active":"2024-03-26T03:20:30.353201+0000","last_peered":"2024-03-26T03:20:30.353201+0000","last_clean":"2024-03-26T03:20:30.353201+0000","last_became_active":"2024-03-26T03:20:12.739985+0000","last_became_peered":"2024-03-26T03:20:12.739985+0000","last_unstale":"2024-03-26T03:20:30.353201+0000","last_undegraded":"2024-03-26T03:20:30.353201+0000","last_fullsized":"2024-03-26T03:20:30.353201+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:54.517230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.517230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":924,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352421+0000","last_change":"2024-03-26T03:20:12.720171+0000","last_active":"2024-03-26T03:20:30.352421+0000","last_peered":"2024-03-26T03:20:30.352421+0000","last_clean":"2024-03-26T03:20:30.352421+0000","last_became_active":"2024-03-26T03:20:12.719811+0000","last_became_peered":"2024-03-26T03:20:12.719811+0000","last_unstale":"2024-03-26T03:20:30.352421+0000","last_undegraded":"2024-03-26T03:20:30.352421+0000","last_fullsized":"2024-03-26T03:20:30.352421+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:58.614060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.227664+0000","last_clean_scrub_stamp":"2024-03-26T03:19:58.614060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:15.521180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":828,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331693+0000","last_change":"2024-03-26T03:17:40.190354+0000","last_active":"2024-03-26T03:20:26.331693+0000","last_peered":"2024-03-26T03:20:26.331693+0000","last_clean":"2024-03-26T03:20:26.331693+0000","last_became_active":"2024-03-26T03:14:37.270129+0000","last_became_peered":"2024-03-26T03:14:37.270129+0000","last_unstale":"2024-03-26T03:20:26.331693+0000","last_undegraded":"2024-03-26T03:20:26.331693+0000","last_fullsized":"2024-03-26T03:20:26.331693+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:40.189873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:40.189873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"446'53","reported_seq":127,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948577+0000","last_change":"2024-03-26T03:17:56.585807+0000","last_active":"2024-03-26T03:20:26.948577+0000","last_peered":"2024-03-26T03:20:26.948577+0000","last_clean":"2024-03-26T03:20:26.948577+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:20:26.948577+0000","last_undegraded":"2024-03-26T03:20:26.948577+0000","last_fullsized":"2024-03-26T03:20:26.948577+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:17:36.443102+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:17:36.443102+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":88,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948135+0000","last_change":"2024-03-26T03:17:56.568076+0000","last_active":"2024-03-26T03:20:26.948135+0000","last_peered":"2024-03-26T03:20:26.948135+0000","last_clean":"2024-03-26T03:20:26.948135+0000","last_became_active":"2024-03-26T03:17:56.567051+0000","last_became_peered":"2024-03-26T03:17:56.567051+0000","last_unstale":"2024-03-26T03:20:26.948135+0000","last_undegraded":"2024-03-26T03:20:26.948135+0000","last_fullsized":"2024-03-26T03:20:26.948135+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:37.464363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:37.464363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":886,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.547175+0000","last_change":"2024-03-26T03:20:12.721118+0000","last_active":"2024-03-26T03:20:30.547175+0000","last_peered":"2024-03-26T03:20:30.547175+0000","last_clean":"2024-03-26T03:20:30.547175+0000","last_became_active":"2024-03-26T03:20:12.720725+0000","last_became_peered":"2024-03-26T03:20:12.720725+0000","last_unstale":"2024-03-26T03:20:30.547175+0000","last_undegraded":"2024-03-26T03:20:30.547175+0000","last_fullsized":"2024-03-26T03:20:30.547175+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:46.947118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.947118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":292,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331026+0000","last_change":"2024-03-26T03:18:50.863004+0000","last_active":"2024-03-26T03:20:26.331026+0000","last_peered":"2024-03-26T03:20:26.331026+0000","last_clean":"2024-03-26T03:20:26.331026+0000","last_became_active":"2024-03-26T03:14:21.586859+0000","last_became_peered":"2024-03-26T03:14:21.586859+0000","last_unstale":"2024-03-26T03:20:26.331026+0000","last_undegraded":"2024-03-26T03:20:26.331026+0000","last_fullsized":"2024-03-26T03:20:26.331026+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:50.862872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:50.862872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":310,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":594,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329366+0000","last_change":"2024-03-26T03:20:09.552174+0000","last_active":"2024-03-26T03:20:26.329366+0000","last_peered":"2024-03-26T03:20:26.329366+0000","last_clean":"2024-03-26T03:20:26.329366+0000","last_became_active":"2024-03-26T03:14:21.590629+0000","last_became_peered":"2024-03-26T03:14:21.590629+0000","last_unstale":"2024-03-26T03:20:26.329366+0000","last_undegraded":"2024-03-26T03:20:26.329366+0000","last_fullsized":"2024-03-26T03:20:26.329366+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:09.551937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:09.551937+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:32.478260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":761,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329329+0000","last_change":"2024-03-26T03:20:16.540083+0000","last_active":"2024-03-26T03:20:26.329329+0000","last_peered":"2024-03-26T03:20:26.329329+0000","last_clean":"2024-03-26T03:20:26.329329+0000","last_became_active":"2024-03-26T03:14:40.169332+0000","last_became_peered":"2024-03-26T03:14:40.169332+0000","last_unstale":"2024-03-26T03:20:26.329329+0000","last_undegraded":"2024-03-26T03:20:26.329329+0000","last_fullsized":"2024-03-26T03:20:26.329329+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:16.539931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:16.539931+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:26.148106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"446'30","reported_seq":917,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948693+0000","last_change":"2024-03-26T03:16:39.220608+0000","last_active":"2024-03-26T03:20:26.948693+0000","last_peered":"2024-03-26T03:20:26.948693+0000","last_clean":"2024-03-26T03:20:26.948693+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:20:26.948693+0000","last_undegraded":"2024-03-26T03:20:26.948693+0000","last_fullsized":"2024-03-26T03:20:26.948693+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'20","last_scrub_stamp":"2024-03-26T03:16:39.220501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:39.220501+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":873,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545364+0000","last_change":"2024-03-26T03:20:29.054078+0000","last_active":"2024-03-26T03:20:30.545364+0000","last_peered":"2024-03-26T03:20:30.545364+0000","last_clean":"2024-03-26T03:20:30.545364+0000","last_became_active":"2024-03-26T03:14:37.251633+0000","last_became_peered":"2024-03-26T03:14:37.251633+0000","last_unstale":"2024-03-26T03:20:30.545364+0000","last_undegraded":"2024-03-26T03:20:30.545364+0000","last_fullsized":"2024-03-26T03:20:30.545364+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:29.054006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:29.054006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:46.812769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"460'12","reported_seq":34,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353079+0000","last_change":"2024-03-26T03:20:12.760262+0000","last_active":"2024-03-26T03:20:30.353079+0000","last_peered":"2024-03-26T03:20:30.353079+0000","last_clean":"2024-03-26T03:20:30.353079+0000","last_became_active":"2024-03-26T03:20:12.740533+0000","last_became_peered":"2024-03-26T03:20:12.740533+0000","last_unstale":"2024-03-26T03:20:30.353079+0000","last_undegraded":"2024-03-26T03:20:30.353079+0000","last_fullsized":"2024-03-26T03:20:30.353079+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'11","last_scrub_stamp":"2024-03-26T03:19:15.476016+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:19:15.476016+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":88,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948434+0000","last_change":"2024-03-26T03:17:56.574248+0000","last_active":"2024-03-26T03:20:26.948434+0000","last_peered":"2024-03-26T03:20:26.948434+0000","last_clean":"2024-03-26T03:20:26.948434+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:20:26.948434+0000","last_undegraded":"2024-03-26T03:20:26.948434+0000","last_fullsized":"2024-03-26T03:20:26.948434+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:32.853807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:32.853807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":451,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207342+0000","last_change":"2024-03-26T03:19:17.349314+0000","last_active":"2024-03-26T03:20:27.207342+0000","last_peered":"2024-03-26T03:20:27.207342+0000","last_clean":"2024-03-26T03:20:27.207342+0000","last_became_active":"2024-03-26T03:16:50.205474+0000","last_became_peered":"2024-03-26T03:16:50.205474+0000","last_unstale":"2024-03-26T03:20:27.207342+0000","last_undegraded":"2024-03-26T03:20:27.207342+0000","last_fullsized":"2024-03-26T03:20:27.207342+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:17.349212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:17.349212+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.072644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"446'39","reported_seq":947,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.547828+0000","last_change":"2024-03-26T03:16:28.788850+0000","last_active":"2024-03-26T03:20:30.547828+0000","last_peered":"2024-03-26T03:20:30.547828+0000","last_clean":"2024-03-26T03:20:30.547828+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:20:30.547828+0000","last_undegraded":"2024-03-26T03:20:30.547828+0000","last_fullsized":"2024-03-26T03:20:30.547828+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'27","last_scrub_stamp":"2024-03-26T03:16:28.788749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:28.788749+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":873,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.544967+0000","last_change":"2024-03-26T03:20:23.014708+0000","last_active":"2024-03-26T03:20:30.544967+0000","last_peered":"2024-03-26T03:20:30.544967+0000","last_clean":"2024-03-26T03:20:30.544967+0000","last_became_active":"2024-03-26T03:14:37.249790+0000","last_became_peered":"2024-03-26T03:14:37.249790+0000","last_unstale":"2024-03-26T03:20:30.544967+0000","last_undegraded":"2024-03-26T03:20:30.544967+0000","last_fullsized":"2024-03-26T03:20:30.544967+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:23.014602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:23.014602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:31.591032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"460'13","reported_seq":143,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948559+0000","last_change":"2024-03-26T03:20:19.317479+0000","last_active":"2024-03-26T03:20:26.948559+0000","last_peered":"2024-03-26T03:20:26.948559+0000","last_clean":"2024-03-26T03:20:26.948559+0000","last_became_active":"2024-03-26T03:20:12.733250+0000","last_became_peered":"2024-03-26T03:20:12.733250+0000","last_unstale":"2024-03-26T03:20:26.948559+0000","last_undegraded":"2024-03-26T03:20:26.948559+0000","last_fullsized":"2024-03-26T03:20:26.948559+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'13","last_scrub_stamp":"2024-03-26T03:20:19.317395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:20:19.317395+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:43.991009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":695,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545939+0000","last_change":"2024-03-26T03:20:12.720217+0000","last_active":"2024-03-26T03:20:30.545939+0000","last_peered":"2024-03-26T03:20:30.545939+0000","last_clean":"2024-03-26T03:20:30.545939+0000","last_became_active":"2024-03-26T03:20:12.719849+0000","last_became_peered":"2024-03-26T03:20:12.719849+0000","last_unstale":"2024-03-26T03:20:30.545939+0000","last_undegraded":"2024-03-26T03:20:30.545939+0000","last_fullsized":"2024-03-26T03:20:30.545939+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:19.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:19:19.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:37.205716+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":1015,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546555+0000","last_change":"2024-03-26T03:20:15.313881+0000","last_active":"2024-03-26T03:20:30.546555+0000","last_peered":"2024-03-26T03:20:30.546555+0000","last_clean":"2024-03-26T03:20:30.546555+0000","last_became_active":"2024-03-26T03:20:15.312092+0000","last_became_peered":"2024-03-26T03:20:15.312092+0000","last_unstale":"2024-03-26T03:20:30.546555+0000","last_undegraded":"2024-03-26T03:20:30.546555+0000","last_fullsized":"2024-03-26T03:20:30.546555+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:39.489210+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:17:39.489210+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"446'39","reported_seq":943,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948809+0000","last_change":"2024-03-26T03:17:42.061990+0000","last_active":"2024-03-26T03:20:26.948809+0000","last_peered":"2024-03-26T03:20:26.948809+0000","last_clean":"2024-03-26T03:20:26.948809+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:20:26.948809+0000","last_undegraded":"2024-03-26T03:20:26.948809+0000","last_fullsized":"2024-03-26T03:20:26.948809+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'31","last_scrub_stamp":"2024-03-26T03:17:42.061780+0000","last_deep_scrub":"291'31","last_deep_scrub_stamp":"2024-03-26T03:12:37.144848+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.061780+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352521+0000","last_change":"2024-03-26T03:20:18.832722+0000","last_active":"2024-03-26T03:20:30.352521+0000","last_peered":"2024-03-26T03:20:30.352521+0000","last_clean":"2024-03-26T03:20:30.352521+0000","last_became_active":"2024-03-26T03:20:12.731753+0000","last_became_peered":"2024-03-26T03:20:12.731753+0000","last_unstale":"2024-03-26T03:20:30.352521+0000","last_undegraded":"2024-03-26T03:20:30.352521+0000","last_fullsized":"2024-03-26T03:20:30.352521+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:18.832637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:18.832637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:32.469000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":684,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948821+0000","last_change":"2024-03-26T03:16:55.201337+0000","last_active":"2024-03-26T03:20:26.948821+0000","last_peered":"2024-03-26T03:20:26.948821+0000","last_clean":"2024-03-26T03:20:26.948821+0000","last_became_active":"2024-03-26T03:14:37.249741+0000","last_became_peered":"2024-03-26T03:14:37.249741+0000","last_unstale":"2024-03-26T03:20:26.948821+0000","last_undegraded":"2024-03-26T03:20:26.948821+0000","last_fullsized":"2024-03-26T03:20:26.948821+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:55.201190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:55.201190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":505,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330155+0000","last_change":"2024-03-26T03:17:44.249680+0000","last_active":"2024-03-26T03:20:26.330155+0000","last_peered":"2024-03-26T03:20:26.330155+0000","last_clean":"2024-03-26T03:20:26.330155+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:20:26.330155+0000","last_undegraded":"2024-03-26T03:20:26.330155+0000","last_fullsized":"2024-03-26T03:20:26.330155+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:44.249561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:17:44.249561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":706,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329309+0000","last_change":"2024-03-26T03:17:55.204987+0000","last_active":"2024-03-26T03:20:26.329309+0000","last_peered":"2024-03-26T03:20:26.329309+0000","last_clean":"2024-03-26T03:20:26.329309+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:20:26.329309+0000","last_undegraded":"2024-03-26T03:20:26.329309+0000","last_fullsized":"2024-03-26T03:20:26.329309+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:17:55.204836+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.204836+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"446'48","reported_seq":958,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947593+0000","last_change":"2024-03-26T03:16:56.199180+0000","last_active":"2024-03-26T03:20:26.947593+0000","last_peered":"2024-03-26T03:20:26.947593+0000","last_clean":"2024-03-26T03:20:26.947593+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:20:26.947593+0000","last_undegraded":"2024-03-26T03:20:26.947593+0000","last_fullsized":"2024-03-26T03:20:26.947593+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-03-26T03:16:56.199057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.199057+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":902,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208528+0000","last_change":"2024-03-26T03:20:19.268776+0000","last_active":"2024-03-26T03:20:27.208528+0000","last_peered":"2024-03-26T03:20:27.208528+0000","last_clean":"2024-03-26T03:20:27.208528+0000","last_became_active":"2024-03-26T03:14:37.276008+0000","last_became_peered":"2024-03-26T03:14:37.276008+0000","last_unstale":"2024-03-26T03:20:27.208528+0000","last_undegraded":"2024-03-26T03:20:27.208528+0000","last_fullsized":"2024-03-26T03:20:27.208528+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:19.268683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:19.268683+0000","last_clean_scrub_stamp":"2024-03-26T03:20:19.268683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:26.159427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":278,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207769+0000","last_change":"2024-03-26T03:20:03.377705+0000","last_active":"2024-03-26T03:20:27.207769+0000","last_peered":"2024-03-26T03:20:27.207769+0000","last_clean":"2024-03-26T03:20:27.207769+0000","last_became_active":"2024-03-26T03:14:37.248955+0000","last_became_peered":"2024-03-26T03:14:37.248955+0000","last_unstale":"2024-03-26T03:20:27.207769+0000","last_undegraded":"2024-03-26T03:20:27.207769+0000","last_fullsized":"2024-03-26T03:20:27.207769+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:03.377553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:49.921407+0000","last_clean_scrub_stamp":"2024-03-26T03:20:03.377553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:25.428661+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":675,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330631+0000","last_change":"2024-03-26T03:16:29.852173+0000","last_active":"2024-03-26T03:20:26.330631+0000","last_peered":"2024-03-26T03:20:26.330631+0000","last_clean":"2024-03-26T03:20:26.330631+0000","last_became_active":"2024-03-26T03:14:40.166583+0000","last_became_peered":"2024-03-26T03:14:40.166583+0000","last_unstale":"2024-03-26T03:20:26.330631+0000","last_undegraded":"2024-03-26T03:20:26.330631+0000","last_fullsized":"2024-03-26T03:20:26.330631+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.852073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.852073+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":860,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947842+0000","last_change":"2024-03-26T03:20:12.732441+0000","last_active":"2024-03-26T03:20:26.947842+0000","last_peered":"2024-03-26T03:20:26.947842+0000","last_clean":"2024-03-26T03:20:26.947842+0000","last_became_active":"2024-03-26T03:20:12.732079+0000","last_became_peered":"2024-03-26T03:20:12.732079+0000","last_unstale":"2024-03-26T03:20:26.947842+0000","last_undegraded":"2024-03-26T03:20:26.947842+0000","last_fullsized":"2024-03-26T03:20:26.947842+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:36.206634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.206634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"446'66","reported_seq":724,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:29.573602+0000","last_change":"2024-03-26T03:20:29.573661+0000","last_active":"2024-03-26T03:20:29.573602+0000","last_peered":"2024-03-26T03:20:29.573602+0000","last_clean":"2024-03-26T03:20:29.573602+0000","last_became_active":"2024-03-26T03:20:12.750931+0000","last_became_peered":"2024-03-26T03:20:12.750931+0000","last_unstale":"2024-03-26T03:20:29.573602+0000","last_undegraded":"2024-03-26T03:20:29.573602+0000","last_fullsized":"2024-03-26T03:20:29.573602+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'66","last_scrub_stamp":"2024-03-26T03:20:29.573524+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:20:29.573524+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:40.633573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":892,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546521+0000","last_change":"2024-03-26T03:20:02.080141+0000","last_active":"2024-03-26T03:20:30.546521+0000","last_peered":"2024-03-26T03:20:30.546521+0000","last_clean":"2024-03-26T03:20:30.546521+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:20:30.546521+0000","last_undegraded":"2024-03-26T03:20:30.546521+0000","last_fullsized":"2024-03-26T03:20:30.546521+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:20:02.079849+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:06:31.299819+0000","last_clean_scrub_stamp":"2024-03-26T03:20:02.079849+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:06.270929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":88,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947609+0000","last_change":"2024-03-26T03:17:56.567975+0000","last_active":"2024-03-26T03:20:26.947609+0000","last_peered":"2024-03-26T03:20:26.947609+0000","last_clean":"2024-03-26T03:20:26.947609+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:20:26.947609+0000","last_undegraded":"2024-03-26T03:20:26.947609+0000","last_fullsized":"2024-03-26T03:20:26.947609+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:52.179410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:52.179410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":987,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948797+0000","last_change":"2024-03-26T03:17:54.095189+0000","last_active":"2024-03-26T03:20:26.948797+0000","last_peered":"2024-03-26T03:20:26.948797+0000","last_clean":"2024-03-26T03:20:26.948797+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:20:26.948797+0000","last_undegraded":"2024-03-26T03:20:26.948797+0000","last_fullsized":"2024-03-26T03:20:26.948797+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:54.095058+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:17:54.095058+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":901,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208147+0000","last_change":"2024-03-26T03:20:16.274034+0000","last_active":"2024-03-26T03:20:27.208147+0000","last_peered":"2024-03-26T03:20:27.208147+0000","last_clean":"2024-03-26T03:20:27.208147+0000","last_became_active":"2024-03-26T03:20:12.719873+0000","last_became_peered":"2024-03-26T03:20:12.719873+0000","last_unstale":"2024-03-26T03:20:27.208147+0000","last_undegraded":"2024-03-26T03:20:27.208147+0000","last_fullsized":"2024-03-26T03:20:27.208147+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:16.273928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:20:16.273928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:22.210799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"446'45","reported_seq":45,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.354939+0000","last_change":"2024-03-26T03:20:12.778629+0000","last_active":"2024-03-26T03:20:30.354939+0000","last_peered":"2024-03-26T03:20:30.354939+0000","last_clean":"2024-03-26T03:20:30.354939+0000","last_became_active":"2024-03-26T03:20:12.740734+0000","last_became_peered":"2024-03-26T03:20:12.740734+0000","last_unstale":"2024-03-26T03:20:30.354939+0000","last_undegraded":"2024-03-26T03:20:30.354939+0000","last_fullsized":"2024-03-26T03:20:30.354939+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'35","last_scrub_stamp":"2024-03-26T03:18:58.829458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:58.829458+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352942+0000","last_change":"2024-03-26T03:20:12.731202+0000","last_active":"2024-03-26T03:20:30.352942+0000","last_peered":"2024-03-26T03:20:30.352942+0000","last_clean":"2024-03-26T03:20:30.352942+0000","last_became_active":"2024-03-26T03:20:12.730837+0000","last_became_peered":"2024-03-26T03:20:12.730837+0000","last_unstale":"2024-03-26T03:20:30.352942+0000","last_undegraded":"2024-03-26T03:20:30.352942+0000","last_fullsized":"2024-03-26T03:20:30.352942+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:06.296617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:06.296617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:15.861957+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":882,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545067+0000","last_change":"2024-03-26T03:19:58.018301+0000","last_active":"2024-03-26T03:20:30.545067+0000","last_peered":"2024-03-26T03:20:30.545067+0000","last_clean":"2024-03-26T03:20:30.545067+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:20:30.545067+0000","last_undegraded":"2024-03-26T03:20:30.545067+0000","last_fullsized":"2024-03-26T03:20:30.545067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:58.018177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:19:58.018177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:11.169491+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":974,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208394+0000","last_change":"2024-03-26T03:20:15.306423+0000","last_active":"2024-03-26T03:20:27.208394+0000","last_peered":"2024-03-26T03:20:27.208394+0000","last_clean":"2024-03-26T03:20:27.208394+0000","last_became_active":"2024-03-26T03:20:15.306151+0000","last_became_peered":"2024-03-26T03:20:15.306151+0000","last_unstale":"2024-03-26T03:20:27.208394+0000","last_undegraded":"2024-03-26T03:20:27.208394+0000","last_fullsized":"2024-03-26T03:20:27.208394+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:53.734968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:18:53.734968+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":915,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330898+0000","last_change":"2024-03-26T03:20:12.733057+0000","last_active":"2024-03-26T03:20:26.330898+0000","last_peered":"2024-03-26T03:20:26.330898+0000","last_clean":"2024-03-26T03:20:26.330898+0000","last_became_active":"2024-03-26T03:20:12.732621+0000","last_became_peered":"2024-03-26T03:20:12.732621+0000","last_unstale":"2024-03-26T03:20:26.330898+0000","last_undegraded":"2024-03-26T03:20:26.330898+0000","last_fullsized":"2024-03-26T03:20:26.330898+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:09.827318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:09.827318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:34.358408+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"446'42","reported_seq":39,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.355784+0000","last_change":"2024-03-26T03:20:12.767646+0000","last_active":"2024-03-26T03:20:30.355784+0000","last_peered":"2024-03-26T03:20:30.355784+0000","last_clean":"2024-03-26T03:20:30.355784+0000","last_became_active":"2024-03-26T03:20:12.734007+0000","last_became_peered":"2024-03-26T03:20:12.734007+0000","last_unstale":"2024-03-26T03:20:30.355784+0000","last_undegraded":"2024-03-26T03:20:30.355784+0000","last_fullsized":"2024-03-26T03:20:30.355784+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'42","last_scrub_stamp":"2024-03-26T03:20:10.008973+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:20:10.008973+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:12.776726+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":814,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545048+0000","last_change":"2024-03-26T03:20:12.735751+0000","last_active":"2024-03-26T03:20:30.545048+0000","last_peered":"2024-03-26T03:20:30.545048+0000","last_clean":"2024-03-26T03:20:30.545048+0000","last_became_active":"2024-03-26T03:20:12.735383+0000","last_became_peered":"2024-03-26T03:20:12.735383+0000","last_unstale":"2024-03-26T03:20:30.545048+0000","last_undegraded":"2024-03-26T03:20:30.545048+0000","last_fullsized":"2024-03-26T03:20:30.545048+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:56.019845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.019845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.354895+0000","last_change":"2024-03-26T03:20:12.746249+0000","last_active":"2024-03-26T03:20:30.354895+0000","last_peered":"2024-03-26T03:20:30.354895+0000","last_clean":"2024-03-26T03:20:30.354895+0000","last_became_active":"2024-03-26T03:20:12.745764+0000","last_became_peered":"2024-03-26T03:20:12.745764+0000","last_unstale":"2024-03-26T03:20:30.354895+0000","last_undegraded":"2024-03-26T03:20:30.354895+0000","last_fullsized":"2024-03-26T03:20:30.354895+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:57.234593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:19:57.234593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:04.948831+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":856,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.547270+0000","last_change":"2024-03-26T03:15:17.827310+0000","last_active":"2024-03-26T03:20:30.547270+0000","last_peered":"2024-03-26T03:20:30.547270+0000","last_clean":"2024-03-26T03:20:30.547270+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:20:30.547270+0000","last_undegraded":"2024-03-26T03:20:30.547270+0000","last_fullsized":"2024-03-26T03:20:30.547270+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:17.827186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:15:17.827186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":995,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546661+0000","last_change":"2024-03-26T03:17:59.603321+0000","last_active":"2024-03-26T03:20:30.546661+0000","last_peered":"2024-03-26T03:20:30.546661+0000","last_clean":"2024-03-26T03:20:30.546661+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:20:30.546661+0000","last_undegraded":"2024-03-26T03:20:30.546661+0000","last_fullsized":"2024-03-26T03:20:30.546661+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:43.863351+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:43.863351+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.354210+0000","last_change":"2024-03-26T03:20:29.014513+0000","last_active":"2024-03-26T03:20:30.354210+0000","last_peered":"2024-03-26T03:20:30.354210+0000","last_clean":"2024-03-26T03:20:30.354210+0000","last_became_active":"2024-03-26T03:20:12.738392+0000","last_became_peered":"2024-03-26T03:20:12.738392+0000","last_unstale":"2024-03-26T03:20:30.354210+0000","last_undegraded":"2024-03-26T03:20:30.354210+0000","last_fullsized":"2024-03-26T03:20:30.354210+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:29.014428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:29.014428+0000","last_clean_scrub_stamp":"2024-03-26T03:20:29.014428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:32.902854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"460'42","reported_seq":1259,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208537+0000","last_change":"2024-03-26T03:20:12.769974+0000","last_active":"2024-03-26T03:20:27.208537+0000","last_peered":"2024-03-26T03:20:27.208537+0000","last_clean":"2024-03-26T03:20:27.208537+0000","last_became_active":"2024-03-26T03:20:12.734516+0000","last_became_peered":"2024-03-26T03:20:12.734516+0000","last_unstale":"2024-03-26T03:20:27.208537+0000","last_undegraded":"2024-03-26T03:20:27.208537+0000","last_fullsized":"2024-03-26T03:20:27.208537+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'41","last_scrub_stamp":"2024-03-26T03:19:08.477608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:08.477608+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:36.585039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":856,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353013+0000","last_change":"2024-03-26T03:20:15.307771+0000","last_active":"2024-03-26T03:20:30.353013+0000","last_peered":"2024-03-26T03:20:30.353013+0000","last_clean":"2024-03-26T03:20:30.353013+0000","last_became_active":"2024-03-26T03:20:15.307418+0000","last_became_peered":"2024-03-26T03:20:15.307418+0000","last_unstale":"2024-03-26T03:20:30.353013+0000","last_undegraded":"2024-03-26T03:20:30.353013+0000","last_fullsized":"2024-03-26T03:20:30.353013+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:05.588322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:05.588322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":932,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330015+0000","last_change":"2024-03-26T03:20:02.029002+0000","last_active":"2024-03-26T03:20:26.330015+0000","last_peered":"2024-03-26T03:20:26.330015+0000","last_clean":"2024-03-26T03:20:26.330015+0000","last_became_active":"2024-03-26T03:14:37.274568+0000","last_became_peered":"2024-03-26T03:14:37.274568+0000","last_unstale":"2024-03-26T03:20:26.330015+0000","last_undegraded":"2024-03-26T03:20:26.330015+0000","last_fullsized":"2024-03-26T03:20:26.330015+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:02.028867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:20:02.028867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:05.033568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":567,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330196+0000","last_change":"2024-03-26T03:19:03.628055+0000","last_active":"2024-03-26T03:20:26.330196+0000","last_peered":"2024-03-26T03:20:26.330196+0000","last_clean":"2024-03-26T03:20:26.330196+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:20:26.330196+0000","last_undegraded":"2024-03-26T03:20:26.330196+0000","last_fullsized":"2024-03-26T03:20:26.330196+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:19:03.627938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:03.627938+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":868,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330922+0000","last_change":"2024-03-26T03:20:00.229236+0000","last_active":"2024-03-26T03:20:26.330922+0000","last_peered":"2024-03-26T03:20:26.330922+0000","last_clean":"2024-03-26T03:20:26.330922+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:20:26.330922+0000","last_undegraded":"2024-03-26T03:20:26.330922+0000","last_fullsized":"2024-03-26T03:20:26.330922+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:00.229148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:20:00.229148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:25.839354+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":794,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330567+0000","last_change":"2024-03-26T03:20:18.047755+0000","last_active":"2024-03-26T03:20:26.330567+0000","last_peered":"2024-03-26T03:20:26.330567+0000","last_clean":"2024-03-26T03:20:26.330567+0000","last_became_active":"2024-03-26T03:16:50.206619+0000","last_became_peered":"2024-03-26T03:16:50.206619+0000","last_unstale":"2024-03-26T03:20:26.330567+0000","last_undegraded":"2024-03-26T03:20:26.330567+0000","last_fullsized":"2024-03-26T03:20:26.330567+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:18.047680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:20:18.047680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:19.432082+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"446'27","reported_seq":997,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.547988+0000","last_change":"2024-03-26T03:20:21.035311+0000","last_active":"2024-03-26T03:20:30.547988+0000","last_peered":"2024-03-26T03:20:30.547988+0000","last_clean":"2024-03-26T03:20:30.547988+0000","last_became_active":"2024-03-26T03:20:12.726379+0000","last_became_peered":"2024-03-26T03:20:12.726379+0000","last_unstale":"2024-03-26T03:20:30.547988+0000","last_undegraded":"2024-03-26T03:20:30.547988+0000","last_fullsized":"2024-03-26T03:20:30.547988+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'27","last_scrub_stamp":"2024-03-26T03:20:21.035217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:20:21.035217+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:28.385319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":741,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:28.611158+0000","last_change":"2024-03-26T03:20:28.611180+0000","last_active":"2024-03-26T03:20:28.611158+0000","last_peered":"2024-03-26T03:20:28.611158+0000","last_clean":"2024-03-26T03:20:28.611158+0000","last_became_active":"2024-03-26T03:20:12.746801+0000","last_became_peered":"2024-03-26T03:20:12.746801+0000","last_unstale":"2024-03-26T03:20:28.611158+0000","last_undegraded":"2024-03-26T03:20:28.611158+0000","last_fullsized":"2024-03-26T03:20:28.611158+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:28.611092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:28.611092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:43.139965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2191,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545305+0000","last_change":"2024-03-26T03:20:19.040147+0000","last_active":"2024-03-26T03:20:30.545305+0000","last_peered":"2024-03-26T03:20:30.545305+0000","last_clean":"2024-03-26T03:20:30.545305+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:20:30.545305+0000","last_undegraded":"2024-03-26T03:20:30.545305+0000","last_fullsized":"2024-03-26T03:20:30.545305+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:20:19.040023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:19.040023+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:31.946150+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1282,"num_read_kb":1277,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":844,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208358+0000","last_change":"2024-03-26T03:18:55.818417+0000","last_active":"2024-03-26T03:20:27.208358+0000","last_peered":"2024-03-26T03:20:27.208358+0000","last_clean":"2024-03-26T03:20:27.208358+0000","last_became_active":"2024-03-26T03:14:37.373684+0000","last_became_peered":"2024-03-26T03:14:37.373684+0000","last_unstale":"2024-03-26T03:20:27.208358+0000","last_undegraded":"2024-03-26T03:20:27.208358+0000","last_fullsized":"2024-03-26T03:20:27.208358+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:55.818297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:18:55.818297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":977,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353064+0000","last_change":"2024-03-26T03:20:19.857619+0000","last_active":"2024-03-26T03:20:30.353064+0000","last_peered":"2024-03-26T03:20:30.353064+0000","last_clean":"2024-03-26T03:20:30.353064+0000","last_became_active":"2024-03-26T03:20:15.304142+0000","last_became_peered":"2024-03-26T03:20:15.304142+0000","last_unstale":"2024-03-26T03:20:30.353064+0000","last_undegraded":"2024-03-26T03:20:30.353064+0000","last_fullsized":"2024-03-26T03:20:30.353064+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:19.857476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:19.857476+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:48.730003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353053+0000","last_change":"2024-03-26T03:20:12.822502+0000","last_active":"2024-03-26T03:20:30.353053+0000","last_peered":"2024-03-26T03:20:30.353053+0000","last_clean":"2024-03-26T03:20:30.353053+0000","last_became_active":"2024-03-26T03:20:12.731160+0000","last_became_peered":"2024-03-26T03:20:12.731160+0000","last_unstale":"2024-03-26T03:20:30.353053+0000","last_undegraded":"2024-03-26T03:20:30.353053+0000","last_fullsized":"2024-03-26T03:20:30.353053+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:12.822368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:20:12.822368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:40.656250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"446'71","reported_seq":1246,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546786+0000","last_change":"2024-03-26T03:15:12.789448+0000","last_active":"2024-03-26T03:20:30.546786+0000","last_peered":"2024-03-26T03:20:30.546786+0000","last_clean":"2024-03-26T03:20:30.546786+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:20:30.546786+0000","last_undegraded":"2024-03-26T03:20:30.546786+0000","last_fullsized":"2024-03-26T03:20:30.546786+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'59","last_scrub_stamp":"2024-03-26T03:15:12.789148+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:15:12.789148+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":683,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545262+0000","last_change":"2024-03-26T03:16:50.962056+0000","last_active":"2024-03-26T03:20:30.545262+0000","last_peered":"2024-03-26T03:20:30.545262+0000","last_clean":"2024-03-26T03:20:30.545262+0000","last_became_active":"2024-03-26T03:04:14.420236+0000","last_became_peered":"2024-03-26T03:04:14.420236+0000","last_unstale":"2024-03-26T03:20:30.545262+0000","last_undegraded":"2024-03-26T03:20:30.545262+0000","last_fullsized":"2024-03-26T03:20:30.545262+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:50.961975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:50.961975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":579,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330281+0000","last_change":"2024-03-26T03:19:12.537772+0000","last_active":"2024-03-26T03:20:26.330281+0000","last_peered":"2024-03-26T03:20:26.330281+0000","last_clean":"2024-03-26T03:20:26.330281+0000","last_became_active":"2024-03-26T03:14:37.274135+0000","last_became_peered":"2024-03-26T03:14:37.274135+0000","last_unstale":"2024-03-26T03:20:26.330281+0000","last_undegraded":"2024-03-26T03:20:26.330281+0000","last_fullsized":"2024-03-26T03:20:26.330281+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_clean_scrub_stamp":"2024-03-26T03:19:12.537519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:40.732976+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":356,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.353799+0000","last_change":"2024-03-26T03:20:15.812785+0000","last_active":"2024-03-26T03:20:30.353799+0000","last_peered":"2024-03-26T03:20:30.353799+0000","last_clean":"2024-03-26T03:20:30.353799+0000","last_became_active":"2024-03-26T03:20:15.305238+0000","last_became_peered":"2024-03-26T03:20:15.305238+0000","last_unstale":"2024-03-26T03:20:30.353799+0000","last_undegraded":"2024-03-26T03:20:30.353799+0000","last_fullsized":"2024-03-26T03:20:30.353799+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:15.812651+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:20:15.812651+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:31.229521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"460'13","reported_seq":34,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.355223+0000","last_change":"2024-03-26T03:20:12.757125+0000","last_active":"2024-03-26T03:20:30.355223+0000","last_peered":"2024-03-26T03:20:30.355223+0000","last_clean":"2024-03-26T03:20:30.355223+0000","last_became_active":"2024-03-26T03:20:12.737843+0000","last_became_peered":"2024-03-26T03:20:12.737843+0000","last_unstale":"2024-03-26T03:20:30.355223+0000","last_undegraded":"2024-03-26T03:20:30.355223+0000","last_fullsized":"2024-03-26T03:20:30.355223+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'12","last_scrub_stamp":"2024-03-26T03:17:42.576771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:42.576771+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"446'42","reported_seq":985,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545472+0000","last_change":"2024-03-26T03:17:56.581987+0000","last_active":"2024-03-26T03:20:30.545472+0000","last_peered":"2024-03-26T03:20:30.545472+0000","last_clean":"2024-03-26T03:20:30.545472+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:20:30.545472+0000","last_undegraded":"2024-03-26T03:20:30.545472+0000","last_fullsized":"2024-03-26T03:20:30.545472+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'32","last_scrub_stamp":"2024-03-26T03:16:40.828476+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-26T03:00:42.399566+0000","last_clean_scrub_stamp":"2024-03-26T03:16:40.828476+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546893+0000","last_change":"2024-03-26T03:20:12.725116+0000","last_active":"2024-03-26T03:20:30.546893+0000","last_peered":"2024-03-26T03:20:30.546893+0000","last_clean":"2024-03-26T03:20:30.546893+0000","last_became_active":"2024-03-26T03:20:12.724585+0000","last_became_peered":"2024-03-26T03:20:12.724585+0000","last_unstale":"2024-03-26T03:20:30.546893+0000","last_undegraded":"2024-03-26T03:20:30.546893+0000","last_fullsized":"2024-03-26T03:20:30.546893+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:05.061096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:05.061096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:21.818791+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352616+0000","last_change":"2024-03-26T03:20:12.746902+0000","last_active":"2024-03-26T03:20:30.352616+0000","last_peered":"2024-03-26T03:20:30.352616+0000","last_clean":"2024-03-26T03:20:30.352616+0000","last_became_active":"2024-03-26T03:20:12.746518+0000","last_became_peered":"2024-03-26T03:20:12.746518+0000","last_unstale":"2024-03-26T03:20:30.352616+0000","last_undegraded":"2024-03-26T03:20:30.352616+0000","last_fullsized":"2024-03-26T03:20:30.352616+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:49.512502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.512502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":966,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208375+0000","last_change":"2024-03-26T03:19:06.555275+0000","last_active":"2024-03-26T03:20:27.208375+0000","last_peered":"2024-03-26T03:20:27.208375+0000","last_clean":"2024-03-26T03:20:27.208375+0000","last_became_active":"2024-03-26T03:14:40.178828+0000","last_became_peered":"2024-03-26T03:14:40.178828+0000","last_unstale":"2024-03-26T03:20:27.208375+0000","last_undegraded":"2024-03-26T03:20:27.208375+0000","last_fullsized":"2024-03-26T03:20:27.208375+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:06.555052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:06.555052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":703,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329860+0000","last_change":"2024-03-26T03:20:15.309164+0000","last_active":"2024-03-26T03:20:26.329860+0000","last_peered":"2024-03-26T03:20:26.329860+0000","last_clean":"2024-03-26T03:20:26.329860+0000","last_became_active":"2024-03-26T03:20:15.308673+0000","last_became_peered":"2024-03-26T03:20:15.308673+0000","last_unstale":"2024-03-26T03:20:26.329860+0000","last_undegraded":"2024-03-26T03:20:26.329860+0000","last_fullsized":"2024-03-26T03:20:26.329860+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:00.547284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:00.547284+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:15.262051+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352760+0000","last_change":"2024-03-26T03:20:12.732105+0000","last_active":"2024-03-26T03:20:30.352760+0000","last_peered":"2024-03-26T03:20:30.352760+0000","last_clean":"2024-03-26T03:20:30.352760+0000","last_became_active":"2024-03-26T03:20:12.731709+0000","last_became_peered":"2024-03-26T03:20:12.731709+0000","last_unstale":"2024-03-26T03:20:30.352760+0000","last_undegraded":"2024-03-26T03:20:30.352760+0000","last_fullsized":"2024-03-26T03:20:30.352760+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:53.556429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:53.556429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"446'36","reported_seq":947,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947667+0000","last_change":"2024-03-26T03:14:37.290458+0000","last_active":"2024-03-26T03:20:26.947667+0000","last_peered":"2024-03-26T03:20:26.947667+0000","last_clean":"2024-03-26T03:20:26.947667+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:20:26.947667+0000","last_undegraded":"2024-03-26T03:20:26.947667+0000","last_fullsized":"2024-03-26T03:20:26.947667+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'30","last_scrub_stamp":"2024-03-26T03:14:37.290313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:14:37.290313+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":960,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948714+0000","last_change":"2024-03-26T03:20:12.761076+0000","last_active":"2024-03-26T03:20:26.948714+0000","last_peered":"2024-03-26T03:20:26.948714+0000","last_clean":"2024-03-26T03:20:26.948714+0000","last_became_active":"2024-03-26T03:20:12.719323+0000","last_became_peered":"2024-03-26T03:20:12.719323+0000","last_unstale":"2024-03-26T03:20:26.948714+0000","last_undegraded":"2024-03-26T03:20:26.948714+0000","last_fullsized":"2024-03-26T03:20:26.948714+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:17:37.106951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:37.106951+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":551,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331651+0000","last_change":"2024-03-26T03:19:00.815058+0000","last_active":"2024-03-26T03:20:26.331651+0000","last_peered":"2024-03-26T03:20:26.331651+0000","last_clean":"2024-03-26T03:20:26.331651+0000","last_became_active":"2024-03-26T03:14:37.265041+0000","last_became_peered":"2024-03-26T03:14:37.265041+0000","last_unstale":"2024-03-26T03:20:26.331651+0000","last_undegraded":"2024-03-26T03:20:26.331651+0000","last_fullsized":"2024-03-26T03:20:26.331651+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:00.814950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:00.814950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":1007,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546517+0000","last_change":"2024-03-26T03:17:59.607666+0000","last_active":"2024-03-26T03:20:30.546517+0000","last_peered":"2024-03-26T03:20:30.546517+0000","last_clean":"2024-03-26T03:20:30.546517+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:20:30.546517+0000","last_undegraded":"2024-03-26T03:20:30.546517+0000","last_fullsized":"2024-03-26T03:20:30.546517+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:44.879010+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.879010+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"446'33","reported_seq":39,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.364844+0000","last_change":"2024-03-26T03:20:12.780639+0000","last_active":"2024-03-26T03:20:30.364844+0000","last_peered":"2024-03-26T03:20:30.364844+0000","last_clean":"2024-03-26T03:20:30.364844+0000","last_became_active":"2024-03-26T03:20:12.743273+0000","last_became_peered":"2024-03-26T03:20:12.743273+0000","last_unstale":"2024-03-26T03:20:30.364844+0000","last_undegraded":"2024-03-26T03:20:30.364844+0000","last_fullsized":"2024-03-26T03:20:30.364844+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'33","last_scrub_stamp":"2024-03-26T03:19:07.549012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:07.549012+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.383786+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":173,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545123+0000","last_change":"2024-03-26T03:20:12.748353+0000","last_active":"2024-03-26T03:20:30.545123+0000","last_peered":"2024-03-26T03:20:30.545123+0000","last_clean":"2024-03-26T03:20:30.545123+0000","last_became_active":"2024-03-26T03:20:12.720153+0000","last_became_peered":"2024-03-26T03:20:12.720153+0000","last_unstale":"2024-03-26T03:20:30.545123+0000","last_undegraded":"2024-03-26T03:20:30.545123+0000","last_fullsized":"2024-03-26T03:20:30.545123+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:16:49.187353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.187353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":577,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331301+0000","last_change":"2024-03-26T03:20:12.746142+0000","last_active":"2024-03-26T03:20:26.331301+0000","last_peered":"2024-03-26T03:20:26.331301+0000","last_clean":"2024-03-26T03:20:26.331301+0000","last_became_active":"2024-03-26T03:20:12.745799+0000","last_became_peered":"2024-03-26T03:20:12.745799+0000","last_unstale":"2024-03-26T03:20:26.331301+0000","last_undegraded":"2024-03-26T03:20:26.331301+0000","last_fullsized":"2024-03-26T03:20:26.331301+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:01.660326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:19:01.660326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":779,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948163+0000","last_change":"2024-03-26T03:20:12.732557+0000","last_active":"2024-03-26T03:20:26.948163+0000","last_peered":"2024-03-26T03:20:26.948163+0000","last_clean":"2024-03-26T03:20:26.948163+0000","last_became_active":"2024-03-26T03:20:12.732140+0000","last_became_peered":"2024-03-26T03:20:12.732140+0000","last_unstale":"2024-03-26T03:20:26.948163+0000","last_undegraded":"2024-03-26T03:20:26.948163+0000","last_fullsized":"2024-03-26T03:20:26.948163+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:55.099060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:17:55.099060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":400,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947821+0000","last_change":"2024-03-26T03:17:45.079338+0000","last_active":"2024-03-26T03:20:26.947821+0000","last_peered":"2024-03-26T03:20:26.947821+0000","last_clean":"2024-03-26T03:20:26.947821+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:20:26.947821+0000","last_undegraded":"2024-03-26T03:20:26.947821+0000","last_fullsized":"2024-03-26T03:20:26.947821+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:45.079209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.079209+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"446'24","reported_seq":653,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330845+0000","last_change":"2024-03-26T03:20:25.586151+0000","last_active":"2024-03-26T03:20:26.330845+0000","last_peered":"2024-03-26T03:20:26.330845+0000","last_clean":"2024-03-26T03:20:26.330845+0000","last_became_active":"2024-03-26T03:14:37.269408+0000","last_became_peered":"2024-03-26T03:14:37.269408+0000","last_unstale":"2024-03-26T03:20:26.330845+0000","last_undegraded":"2024-03-26T03:20:26.330845+0000","last_fullsized":"2024-03-26T03:20:26.330845+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'24","last_scrub_stamp":"2024-03-26T03:20:25.586055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:20:25.586055+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:42.402362+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":908,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207602+0000","last_change":"2024-03-26T03:20:10.296608+0000","last_active":"2024-03-26T03:20:27.207602+0000","last_peered":"2024-03-26T03:20:27.207602+0000","last_clean":"2024-03-26T03:20:27.207602+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:20:27.207602+0000","last_undegraded":"2024-03-26T03:20:27.207602+0000","last_fullsized":"2024-03-26T03:20:27.207602+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:10.296415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:10.296415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:13.831965+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":879,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330421+0000","last_change":"2024-03-26T03:20:23.048757+0000","last_active":"2024-03-26T03:20:26.330421+0000","last_peered":"2024-03-26T03:20:26.330421+0000","last_clean":"2024-03-26T03:20:26.330421+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:20:26.330421+0000","last_undegraded":"2024-03-26T03:20:26.330421+0000","last_fullsized":"2024-03-26T03:20:26.330421+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:23.048651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:20:23.048651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:46.065147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":278,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208450+0000","last_change":"2024-03-26T03:19:00.728008+0000","last_active":"2024-03-26T03:20:27.208450+0000","last_peered":"2024-03-26T03:20:27.208450+0000","last_clean":"2024-03-26T03:20:27.208450+0000","last_became_active":"2024-03-26T03:14:37.333430+0000","last_became_peered":"2024-03-26T03:14:37.333430+0000","last_unstale":"2024-03-26T03:20:27.208450+0000","last_undegraded":"2024-03-26T03:20:27.208450+0000","last_fullsized":"2024-03-26T03:20:27.208450+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:00.727659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:00.727659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":328,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352592+0000","last_change":"2024-03-26T03:20:16.821528+0000","last_active":"2024-03-26T03:20:30.352592+0000","last_peered":"2024-03-26T03:20:30.352592+0000","last_clean":"2024-03-26T03:20:30.352592+0000","last_became_active":"2024-03-26T03:20:15.317985+0000","last_became_peered":"2024-03-26T03:20:15.317985+0000","last_unstale":"2024-03-26T03:20:30.352592+0000","last_undegraded":"2024-03-26T03:20:30.352592+0000","last_fullsized":"2024-03-26T03:20:30.352592+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:16.821442+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:20:16.821442+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:46.490221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"446'49","reported_seq":668,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330046+0000","last_change":"2024-03-26T03:16:49.904794+0000","last_active":"2024-03-26T03:20:26.330046+0000","last_peered":"2024-03-26T03:20:26.330046+0000","last_clean":"2024-03-26T03:20:26.330046+0000","last_became_active":"2024-03-26T03:14:37.265466+0000","last_became_peered":"2024-03-26T03:14:37.265466+0000","last_unstale":"2024-03-26T03:20:26.330046+0000","last_undegraded":"2024-03-26T03:20:26.330046+0000","last_fullsized":"2024-03-26T03:20:26.330046+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'39","last_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:16:49.904637+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":912,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329833+0000","last_change":"2024-03-26T03:20:12.725333+0000","last_active":"2024-03-26T03:20:26.329833+0000","last_peered":"2024-03-26T03:20:26.329833+0000","last_clean":"2024-03-26T03:20:26.329833+0000","last_became_active":"2024-03-26T03:20:12.725005+0000","last_became_peered":"2024-03-26T03:20:12.725005+0000","last_unstale":"2024-03-26T03:20:26.329833+0000","last_undegraded":"2024-03-26T03:20:26.329833+0000","last_fullsized":"2024-03-26T03:20:26.329833+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:11.775591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:19:11.775591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"460'14","reported_seq":1197,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330629+0000","last_change":"2024-03-26T03:20:14.018886+0000","last_active":"2024-03-26T03:20:26.330629+0000","last_peered":"2024-03-26T03:20:26.330629+0000","last_clean":"2024-03-26T03:20:26.330629+0000","last_became_active":"2024-03-26T03:20:12.719502+0000","last_became_peered":"2024-03-26T03:20:12.719502+0000","last_unstale":"2024-03-26T03:20:26.330629+0000","last_undegraded":"2024-03-26T03:20:26.330629+0000","last_fullsized":"2024-03-26T03:20:26.330629+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'14","last_scrub_stamp":"2024-03-26T03:20:14.018761+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:20:14.018761+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:43.833565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.364050+0000","last_change":"2024-03-26T03:20:12.747099+0000","last_active":"2024-03-26T03:20:30.364050+0000","last_peered":"2024-03-26T03:20:30.364050+0000","last_clean":"2024-03-26T03:20:30.364050+0000","last_became_active":"2024-03-26T03:20:12.746708+0000","last_became_peered":"2024-03-26T03:20:12.746708+0000","last_unstale":"2024-03-26T03:20:30.364050+0000","last_undegraded":"2024-03-26T03:20:30.364050+0000","last_fullsized":"2024-03-26T03:20:30.364050+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:59.591751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:59.591751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":573,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.206863+0000","last_change":"2024-03-26T03:18:54.739935+0000","last_active":"2024-03-26T03:20:27.206863+0000","last_peered":"2024-03-26T03:20:27.206863+0000","last_clean":"2024-03-26T03:20:27.206863+0000","last_became_active":"2024-03-26T03:14:40.173243+0000","last_became_peered":"2024-03-26T03:14:40.173243+0000","last_unstale":"2024-03-26T03:20:27.206863+0000","last_undegraded":"2024-03-26T03:20:27.206863+0000","last_fullsized":"2024-03-26T03:20:27.206863+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:18:54.739688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:18:54.739688+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"446'27","reported_seq":710,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331188+0000","last_change":"2024-03-26T03:16:59.931429+0000","last_active":"2024-03-26T03:20:26.331188+0000","last_peered":"2024-03-26T03:20:26.331188+0000","last_clean":"2024-03-26T03:20:26.331188+0000","last_became_active":"2024-03-26T03:14:37.333426+0000","last_became_peered":"2024-03-26T03:14:37.333426+0000","last_unstale":"2024-03-26T03:20:26.331188+0000","last_undegraded":"2024-03-26T03:20:26.331188+0000","last_fullsized":"2024-03-26T03:20:26.331188+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'21","last_scrub_stamp":"2024-03-26T03:16:59.931283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:59.931283+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.354826+0000","last_change":"2024-03-26T03:20:26.947350+0000","last_active":"2024-03-26T03:20:30.354826+0000","last_peered":"2024-03-26T03:20:30.354826+0000","last_clean":"2024-03-26T03:20:30.354826+0000","last_became_active":"2024-03-26T03:20:12.746773+0000","last_became_peered":"2024-03-26T03:20:12.746773+0000","last_unstale":"2024-03-26T03:20:30.354826+0000","last_undegraded":"2024-03-26T03:20:30.354826+0000","last_fullsized":"2024-03-26T03:20:30.354826+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:26.947056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:20:26.947056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:27.829777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"342'11","reported_seq":685,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330381+0000","last_change":"2024-03-26T03:17:50.265634+0000","last_active":"2024-03-26T03:20:26.330381+0000","last_peered":"2024-03-26T03:20:26.330381+0000","last_clean":"2024-03-26T03:20:26.330381+0000","last_became_active":"2024-03-26T03:14:37.264759+0000","last_became_peered":"2024-03-26T03:14:37.264759+0000","last_unstale":"2024-03-26T03:20:26.330381+0000","last_undegraded":"2024-03-26T03:20:26.330381+0000","last_fullsized":"2024-03-26T03:20:26.330381+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'11","last_scrub_stamp":"2024-03-26T03:17:50.265404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:50.265404+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":866,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948867+0000","last_change":"2024-03-26T03:17:48.227168+0000","last_active":"2024-03-26T03:20:26.948867+0000","last_peered":"2024-03-26T03:20:26.948867+0000","last_clean":"2024-03-26T03:20:26.948867+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:20:26.948867+0000","last_undegraded":"2024-03-26T03:20:26.948867+0000","last_fullsized":"2024-03-26T03:20:26.948867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:17:48.226947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:48.226947+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":994,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330641+0000","last_change":"2024-03-26T03:20:15.296572+0000","last_active":"2024-03-26T03:20:26.330641+0000","last_peered":"2024-03-26T03:20:26.330641+0000","last_clean":"2024-03-26T03:20:26.330641+0000","last_became_active":"2024-03-26T03:20:15.295795+0000","last_became_peered":"2024-03-26T03:20:15.295795+0000","last_unstale":"2024-03-26T03:20:26.330641+0000","last_undegraded":"2024-03-26T03:20:26.330641+0000","last_fullsized":"2024-03-26T03:20:26.330641+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:07.068141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:07.068141+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:14.953313+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"446'25","reported_seq":924,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545395+0000","last_change":"2024-03-26T03:19:57.066273+0000","last_active":"2024-03-26T03:20:30.545395+0000","last_peered":"2024-03-26T03:20:30.545395+0000","last_clean":"2024-03-26T03:20:30.545395+0000","last_became_active":"2024-03-26T02:58:58.927617+0000","last_became_peered":"2024-03-26T02:58:58.927617+0000","last_unstale":"2024-03-26T03:20:30.545395+0000","last_undegraded":"2024-03-26T03:20:30.545395+0000","last_fullsized":"2024-03-26T03:20:30.545395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'25","last_scrub_stamp":"2024-03-26T03:19:57.066118+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:19:57.066118+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:13.765003+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":873,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207858+0000","last_change":"2024-03-26T03:19:59.291519+0000","last_active":"2024-03-26T03:20:27.207858+0000","last_peered":"2024-03-26T03:20:27.207858+0000","last_clean":"2024-03-26T03:20:27.207858+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:20:27.207858+0000","last_undegraded":"2024-03-26T03:20:27.207858+0000","last_fullsized":"2024-03-26T03:20:27.207858+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:59.291437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:59.291437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:22.922597+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":560,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331394+0000","last_change":"2024-03-26T03:20:12.749509+0000","last_active":"2024-03-26T03:20:26.331394+0000","last_peered":"2024-03-26T03:20:26.331394+0000","last_clean":"2024-03-26T03:20:26.331394+0000","last_became_active":"2024-03-26T03:20:12.749158+0000","last_became_peered":"2024-03-26T03:20:12.749158+0000","last_unstale":"2024-03-26T03:20:26.331394+0000","last_undegraded":"2024-03-26T03:20:26.331394+0000","last_fullsized":"2024-03-26T03:20:26.331394+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:44.909523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:16:44.909523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352286+0000","last_change":"2024-03-26T03:20:12.736329+0000","last_active":"2024-03-26T03:20:30.352286+0000","last_peered":"2024-03-26T03:20:30.352286+0000","last_clean":"2024-03-26T03:20:30.352286+0000","last_became_active":"2024-03-26T03:20:12.736042+0000","last_became_peered":"2024-03-26T03:20:12.736042+0000","last_unstale":"2024-03-26T03:20:30.352286+0000","last_undegraded":"2024-03-26T03:20:30.352286+0000","last_fullsized":"2024-03-26T03:20:30.352286+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:35.405139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:17:35.405139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":664,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546395+0000","last_change":"2024-03-26T03:20:15.935819+0000","last_active":"2024-03-26T03:20:30.546395+0000","last_peered":"2024-03-26T03:20:30.546395+0000","last_clean":"2024-03-26T03:20:30.546395+0000","last_became_active":"2024-03-26T03:20:15.311071+0000","last_became_peered":"2024-03-26T03:20:15.311071+0000","last_unstale":"2024-03-26T03:20:30.546395+0000","last_undegraded":"2024-03-26T03:20:30.546395+0000","last_fullsized":"2024-03-26T03:20:30.546395+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:15.935709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:15.935709+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:45.044071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"446'27","reported_seq":917,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948290+0000","last_change":"2024-03-26T03:16:53.170995+0000","last_active":"2024-03-26T03:20:26.948290+0000","last_peered":"2024-03-26T03:20:26.948290+0000","last_clean":"2024-03-26T03:20:26.948290+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:20:26.948290+0000","last_undegraded":"2024-03-26T03:20:26.948290+0000","last_fullsized":"2024-03-26T03:20:26.948290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'19","last_scrub_stamp":"2024-03-26T03:16:53.170849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:53.170849+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546291+0000","last_change":"2024-03-26T03:20:12.720525+0000","last_active":"2024-03-26T03:20:30.546291+0000","last_peered":"2024-03-26T03:20:30.546291+0000","last_clean":"2024-03-26T03:20:30.546291+0000","last_became_active":"2024-03-26T03:20:12.720143+0000","last_became_peered":"2024-03-26T03:20:12.720143+0000","last_unstale":"2024-03-26T03:20:30.546291+0000","last_undegraded":"2024-03-26T03:20:30.546291+0000","last_fullsized":"2024-03-26T03:20:30.546291+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:34.417610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:02.578096+0000","last_clean_scrub_stamp":"2024-03-26T03:17:34.417610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":851,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545511+0000","last_change":"2024-03-26T03:15:14.862264+0000","last_active":"2024-03-26T03:20:30.545511+0000","last_peered":"2024-03-26T03:20:30.545511+0000","last_clean":"2024-03-26T03:20:30.545511+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:20:30.545511+0000","last_undegraded":"2024-03-26T03:20:30.545511+0000","last_fullsized":"2024-03-26T03:20:30.545511+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:14.862148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:14.862148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":572,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329602+0000","last_change":"2024-03-26T03:20:12.747156+0000","last_active":"2024-03-26T03:20:26.329602+0000","last_peered":"2024-03-26T03:20:26.329602+0000","last_clean":"2024-03-26T03:20:26.329602+0000","last_became_active":"2024-03-26T03:20:12.745809+0000","last_became_peered":"2024-03-26T03:20:12.745809+0000","last_unstale":"2024-03-26T03:20:26.329602+0000","last_undegraded":"2024-03-26T03:20:26.329602+0000","last_fullsized":"2024-03-26T03:20:26.329602+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:08.518278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:20:08.518278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:37.120570+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":579,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545643+0000","last_change":"2024-03-26T03:20:28.030160+0000","last_active":"2024-03-26T03:20:30.545643+0000","last_peered":"2024-03-26T03:20:30.545643+0000","last_clean":"2024-03-26T03:20:30.545643+0000","last_became_active":"2024-03-26T03:14:40.166745+0000","last_became_peered":"2024-03-26T03:14:40.166745+0000","last_unstale":"2024-03-26T03:20:30.545643+0000","last_undegraded":"2024-03-26T03:20:30.545643+0000","last_fullsized":"2024-03-26T03:20:30.545643+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:28.030072+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:20:28.030072+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:49.757815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"465'14","reported_seq":591,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331733+0000","last_change":"2024-03-26T03:20:20.501797+0000","last_active":"2024-03-26T03:20:26.331733+0000","last_peered":"2024-03-26T03:20:26.331733+0000","last_clean":"2024-03-26T03:20:26.331733+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:20:26.331733+0000","last_undegraded":"2024-03-26T03:20:26.331733+0000","last_fullsized":"2024-03-26T03:20:26.331733+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'14","last_scrub_stamp":"2024-03-26T03:20:20.501710+0000","last_deep_scrub":"465'14","last_deep_scrub_stamp":"2024-03-26T03:20:20.501710+0000","last_clean_scrub_stamp":"2024-03-26T03:20:20.501710+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:26.172332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":910,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330564+0000","last_change":"2024-03-26T03:19:18.786789+0000","last_active":"2024-03-26T03:20:26.330564+0000","last_peered":"2024-03-26T03:20:26.330564+0000","last_clean":"2024-03-26T03:20:26.330564+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:20:26.330564+0000","last_undegraded":"2024-03-26T03:20:26.330564+0000","last_fullsized":"2024-03-26T03:20:26.330564+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:18.786671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:19:18.786671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:31.252575+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":912,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207332+0000","last_change":"2024-03-26T03:20:12.720562+0000","last_active":"2024-03-26T03:20:27.207332+0000","last_peered":"2024-03-26T03:20:27.207332+0000","last_clean":"2024-03-26T03:20:27.207332+0000","last_became_active":"2024-03-26T03:20:12.720161+0000","last_became_peered":"2024-03-26T03:20:12.720161+0000","last_unstale":"2024-03-26T03:20:27.207332+0000","last_undegraded":"2024-03-26T03:20:27.207332+0000","last_fullsized":"2024-03-26T03:20:27.207332+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:57.629637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:13:29.300020+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.629637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":270,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948074+0000","last_change":"2024-03-26T03:16:45.148252+0000","last_active":"2024-03-26T03:20:26.948074+0000","last_peered":"2024-03-26T03:20:26.948074+0000","last_clean":"2024-03-26T03:20:26.948074+0000","last_became_active":"2024-03-26T03:14:37.260920+0000","last_became_peered":"2024-03-26T03:14:37.260920+0000","last_unstale":"2024-03-26T03:20:26.948074+0000","last_undegraded":"2024-03-26T03:20:26.948074+0000","last_fullsized":"2024-03-26T03:20:26.948074+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:45.148129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.148129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":642,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948098+0000","last_change":"2024-03-26T03:17:59.621631+0000","last_active":"2024-03-26T03:20:26.948098+0000","last_peered":"2024-03-26T03:20:26.948098+0000","last_clean":"2024-03-26T03:20:26.948098+0000","last_became_active":"2024-03-26T03:17:59.621344+0000","last_became_peered":"2024-03-26T03:17:59.621344+0000","last_unstale":"2024-03-26T03:20:26.948098+0000","last_undegraded":"2024-03-26T03:20:26.948098+0000","last_fullsized":"2024-03-26T03:20:26.948098+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:45.232258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:45.232258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"446'39","reported_seq":650,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330530+0000","last_change":"2024-03-26T03:16:56.991143+0000","last_active":"2024-03-26T03:20:26.330530+0000","last_peered":"2024-03-26T03:20:26.330530+0000","last_clean":"2024-03-26T03:20:26.330530+0000","last_became_active":"2024-03-26T03:14:37.273380+0000","last_became_peered":"2024-03-26T03:14:37.273380+0000","last_unstale":"2024-03-26T03:20:26.330530+0000","last_undegraded":"2024-03-26T03:20:26.330530+0000","last_fullsized":"2024-03-26T03:20:26.330530+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'31","last_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:16:56.991031+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":895,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:29.310073+0000","last_change":"2024-03-26T03:20:29.310098+0000","last_active":"2024-03-26T03:20:29.310073+0000","last_peered":"2024-03-26T03:20:29.310073+0000","last_clean":"2024-03-26T03:20:29.310073+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:20:29.310073+0000","last_undegraded":"2024-03-26T03:20:29.310073+0000","last_fullsized":"2024-03-26T03:20:29.310073+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:29.310000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:29.310000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:47.624344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":279,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947406+0000","last_change":"2024-03-26T03:20:12.719520+0000","last_active":"2024-03-26T03:20:26.947406+0000","last_peered":"2024-03-26T03:20:26.947406+0000","last_clean":"2024-03-26T03:20:26.947406+0000","last_became_active":"2024-03-26T03:20:12.718935+0000","last_became_peered":"2024-03-26T03:20:12.718935+0000","last_unstale":"2024-03-26T03:20:26.947406+0000","last_undegraded":"2024-03-26T03:20:26.947406+0000","last_fullsized":"2024-03-26T03:20:26.947406+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:32.002635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:17:32.002635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":558,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331281+0000","last_change":"2024-03-26T03:20:12.748256+0000","last_active":"2024-03-26T03:20:26.331281+0000","last_peered":"2024-03-26T03:20:26.331281+0000","last_clean":"2024-03-26T03:20:26.331281+0000","last_became_active":"2024-03-26T03:20:12.747304+0000","last_became_peered":"2024-03-26T03:20:12.747304+0000","last_unstale":"2024-03-26T03:20:26.331281+0000","last_undegraded":"2024-03-26T03:20:26.331281+0000","last_fullsized":"2024-03-26T03:20:26.331281+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:04.682583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:04.682583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:32.189002+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":905,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329784+0000","last_change":"2024-03-26T03:20:24.005374+0000","last_active":"2024-03-26T03:20:26.329784+0000","last_peered":"2024-03-26T03:20:26.329784+0000","last_clean":"2024-03-26T03:20:26.329784+0000","last_became_active":"2024-03-26T03:20:15.299184+0000","last_became_peered":"2024-03-26T03:20:15.299184+0000","last_unstale":"2024-03-26T03:20:26.329784+0000","last_undegraded":"2024-03-26T03:20:26.329784+0000","last_fullsized":"2024-03-26T03:20:26.329784+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:24.005238+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:20:24.005238+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:47.512405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"446'36","reported_seq":39,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.355672+0000","last_change":"2024-03-26T03:20:12.784169+0000","last_active":"2024-03-26T03:20:30.355672+0000","last_peered":"2024-03-26T03:20:30.355672+0000","last_clean":"2024-03-26T03:20:30.355672+0000","last_became_active":"2024-03-26T03:20:12.743983+0000","last_became_peered":"2024-03-26T03:20:12.743983+0000","last_unstale":"2024-03-26T03:20:30.355672+0000","last_undegraded":"2024-03-26T03:20:30.355672+0000","last_fullsized":"2024-03-26T03:20:30.355672+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:19:11.557859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:11.557859+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":924,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.919565+0000","last_change":"2024-03-26T03:20:27.919611+0000","last_active":"2024-03-26T03:20:27.919565+0000","last_peered":"2024-03-26T03:20:27.919565+0000","last_clean":"2024-03-26T03:20:27.919565+0000","last_became_active":"2024-03-26T03:14:37.266519+0000","last_became_peered":"2024-03-26T03:14:37.266519+0000","last_unstale":"2024-03-26T03:20:27.919565+0000","last_undegraded":"2024-03-26T03:20:27.919565+0000","last_fullsized":"2024-03-26T03:20:27.919565+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:27.919475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:20:27.919475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:37.272614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"428'11","reported_seq":1190,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208614+0000","last_change":"2024-03-26T03:20:00.250193+0000","last_active":"2024-03-26T03:20:27.208614+0000","last_peered":"2024-03-26T03:20:27.208614+0000","last_clean":"2024-03-26T03:20:27.208614+0000","last_became_active":"2024-03-26T03:17:56.569871+0000","last_became_peered":"2024-03-26T03:17:56.569871+0000","last_unstale":"2024-03-26T03:20:27.208614+0000","last_undegraded":"2024-03-26T03:20:27.208614+0000","last_fullsized":"2024-03-26T03:20:27.208614+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'11","last_scrub_stamp":"2024-03-26T03:20:00.250113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:20:00.250113+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:08.797417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":660,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329746+0000","last_change":"2024-03-26T03:19:17.489071+0000","last_active":"2024-03-26T03:20:26.329746+0000","last_peered":"2024-03-26T03:20:26.329746+0000","last_clean":"2024-03-26T03:20:26.329746+0000","last_became_active":"2024-03-26T03:16:50.206818+0000","last_became_peered":"2024-03-26T03:16:50.206818+0000","last_unstale":"2024-03-26T03:20:26.329746+0000","last_undegraded":"2024-03-26T03:20:26.329746+0000","last_fullsized":"2024-03-26T03:20:26.329746+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:19:17.488941+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:19:17.488941+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:27.176056+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":867,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947872+0000","last_change":"2024-03-26T03:17:59.611282+0000","last_active":"2024-03-26T03:20:26.947872+0000","last_peered":"2024-03-26T03:20:26.947872+0000","last_clean":"2024-03-26T03:20:26.947872+0000","last_became_active":"2024-03-26T03:17:59.610887+0000","last_became_peered":"2024-03-26T03:17:59.610887+0000","last_unstale":"2024-03-26T03:20:26.947872+0000","last_undegraded":"2024-03-26T03:20:26.947872+0000","last_fullsized":"2024-03-26T03:20:26.947872+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:16:29.803243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:16:29.803243+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"446'33","reported_seq":940,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.206723+0000","last_change":"2024-03-26T03:18:49.800384+0000","last_active":"2024-03-26T03:20:27.206723+0000","last_peered":"2024-03-26T03:20:27.206723+0000","last_clean":"2024-03-26T03:20:27.206723+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:20:27.206723+0000","last_undegraded":"2024-03-26T03:20:27.206723+0000","last_fullsized":"2024-03-26T03:20:27.206723+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'25","last_scrub_stamp":"2024-03-26T03:18:49.800256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:18:49.800256+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":535,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330858+0000","last_change":"2024-03-26T03:20:12.779609+0000","last_active":"2024-03-26T03:20:26.330858+0000","last_peered":"2024-03-26T03:20:26.330858+0000","last_clean":"2024-03-26T03:20:26.330858+0000","last_became_active":"2024-03-26T03:20:12.746319+0000","last_became_peered":"2024-03-26T03:20:12.746319+0000","last_unstale":"2024-03-26T03:20:26.330858+0000","last_undegraded":"2024-03-26T03:20:26.330858+0000","last_fullsized":"2024-03-26T03:20:26.330858+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:17:31.999158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:17:31.999158+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.363454+0000","last_change":"2024-03-26T03:20:13.911799+0000","last_active":"2024-03-26T03:20:30.363454+0000","last_peered":"2024-03-26T03:20:30.363454+0000","last_clean":"2024-03-26T03:20:30.363454+0000","last_became_active":"2024-03-26T03:20:12.733164+0000","last_became_peered":"2024-03-26T03:20:12.733164+0000","last_unstale":"2024-03-26T03:20:30.363454+0000","last_undegraded":"2024-03-26T03:20:30.363454+0000","last_fullsized":"2024-03-26T03:20:30.363454+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:13.911675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:20:13.911675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:27.594348+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":881,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.546045+0000","last_change":"2024-03-26T03:17:56.570706+0000","last_active":"2024-03-26T03:20:30.546045+0000","last_peered":"2024-03-26T03:20:30.546045+0000","last_clean":"2024-03-26T03:20:30.546045+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:20:30.546045+0000","last_undegraded":"2024-03-26T03:20:30.546045+0000","last_fullsized":"2024-03-26T03:20:30.546045+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:52.953482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:16:52.953482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":787,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330923+0000","last_change":"2024-03-26T03:20:15.337137+0000","last_active":"2024-03-26T03:20:26.330923+0000","last_peered":"2024-03-26T03:20:26.330923+0000","last_clean":"2024-03-26T03:20:26.330923+0000","last_became_active":"2024-03-26T03:20:15.336658+0000","last_became_peered":"2024-03-26T03:20:15.336658+0000","last_unstale":"2024-03-26T03:20:26.330923+0000","last_undegraded":"2024-03-26T03:20:26.330923+0000","last_fullsized":"2024-03-26T03:20:26.330923+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:04.509759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:04.509759+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:25.105477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"446'39","reported_seq":574,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.547005+0000","last_change":"2024-03-26T03:16:50.211157+0000","last_active":"2024-03-26T03:20:30.547005+0000","last_peered":"2024-03-26T03:20:30.547005+0000","last_clean":"2024-03-26T03:20:30.547005+0000","last_became_active":"2024-03-26T03:16:50.209508+0000","last_became_peered":"2024-03-26T03:16:50.209508+0000","last_unstale":"2024-03-26T03:20:30.547005+0000","last_undegraded":"2024-03-26T03:20:30.547005+0000","last_fullsized":"2024-03-26T03:20:30.547005+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'27","last_scrub_stamp":"2024-03-26T03:16:36.265181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:36.265181+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":909,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208094+0000","last_change":"2024-03-26T03:20:25.352642+0000","last_active":"2024-03-26T03:20:27.208094+0000","last_peered":"2024-03-26T03:20:27.208094+0000","last_clean":"2024-03-26T03:20:27.208094+0000","last_became_active":"2024-03-26T03:14:37.249066+0000","last_became_peered":"2024-03-26T03:14:37.249066+0000","last_unstale":"2024-03-26T03:20:27.208094+0000","last_undegraded":"2024-03-26T03:20:27.208094+0000","last_fullsized":"2024-03-26T03:20:27.208094+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:25.352522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:25.352522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:48.652423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":180,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208161+0000","last_change":"2024-03-26T03:20:18.304309+0000","last_active":"2024-03-26T03:20:27.208161+0000","last_peered":"2024-03-26T03:20:27.208161+0000","last_clean":"2024-03-26T03:20:27.208161+0000","last_became_active":"2024-03-26T03:20:12.733918+0000","last_became_peered":"2024-03-26T03:20:12.733918+0000","last_unstale":"2024-03-26T03:20:27.208161+0000","last_undegraded":"2024-03-26T03:20:27.208161+0000","last_fullsized":"2024-03-26T03:20:27.208161+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:18.304200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:20:18.304200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:47.501982+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":875,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:29.396654+0000","last_change":"2024-03-26T03:20:29.396678+0000","last_active":"2024-03-26T03:20:29.396654+0000","last_peered":"2024-03-26T03:20:29.396654+0000","last_clean":"2024-03-26T03:20:29.396654+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:20:29.396654+0000","last_undegraded":"2024-03-26T03:20:29.396654+0000","last_fullsized":"2024-03-26T03:20:29.396654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:29.396576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:20:29.396576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:54.400389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":681,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329532+0000","last_change":"2024-03-26T03:20:21.030219+0000","last_active":"2024-03-26T03:20:26.329532+0000","last_peered":"2024-03-26T03:20:26.329532+0000","last_clean":"2024-03-26T03:20:26.329532+0000","last_became_active":"2024-03-26T03:14:40.162583+0000","last_became_peered":"2024-03-26T03:14:40.162583+0000","last_unstale":"2024-03-26T03:20:26.329532+0000","last_undegraded":"2024-03-26T03:20:26.329532+0000","last_fullsized":"2024-03-26T03:20:26.329532+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:20:21.030059+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:20:21.030059+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:41.799477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"446'36","reported_seq":625,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331363+0000","last_change":"2024-03-26T03:16:48.957169+0000","last_active":"2024-03-26T03:20:26.331363+0000","last_peered":"2024-03-26T03:20:26.331363+0000","last_clean":"2024-03-26T03:20:26.331363+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:20:26.331363+0000","last_undegraded":"2024-03-26T03:20:26.331363+0000","last_fullsized":"2024-03-26T03:20:26.331363+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'30","last_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:16:48.957049+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":591,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331553+0000","last_change":"2024-03-26T03:20:10.554326+0000","last_active":"2024-03-26T03:20:26.331553+0000","last_peered":"2024-03-26T03:20:26.331553+0000","last_clean":"2024-03-26T03:20:26.331553+0000","last_became_active":"2024-03-26T03:14:37.275278+0000","last_became_peered":"2024-03-26T03:14:37.275278+0000","last_unstale":"2024-03-26T03:20:26.331553+0000","last_undegraded":"2024-03-26T03:20:26.331553+0000","last_fullsized":"2024-03-26T03:20:26.331553+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:10.554057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:10.554057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:24.105185+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":846,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948439+0000","last_change":"2024-03-26T03:14:36.271942+0000","last_active":"2024-03-26T03:20:26.948439+0000","last_peered":"2024-03-26T03:20:26.948439+0000","last_clean":"2024-03-26T03:20:26.948439+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:20:26.948439+0000","last_undegraded":"2024-03-26T03:20:26.948439+0000","last_fullsized":"2024-03-26T03:20:26.948439+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:14:36.271810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:14:36.271810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":546,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331121+0000","last_change":"2024-03-26T03:18:57.829715+0000","last_active":"2024-03-26T03:20:26.331121+0000","last_peered":"2024-03-26T03:20:26.331121+0000","last_clean":"2024-03-26T03:20:26.331121+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:20:26.331121+0000","last_undegraded":"2024-03-26T03:20:26.331121+0000","last_fullsized":"2024-03-26T03:20:26.331121+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:57.829604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:57.829604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"446'39","reported_seq":139,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947754+0000","last_change":"2024-03-26T03:20:12.786366+0000","last_active":"2024-03-26T03:20:26.947754+0000","last_peered":"2024-03-26T03:20:26.947754+0000","last_clean":"2024-03-26T03:20:26.947754+0000","last_became_active":"2024-03-26T03:20:12.732132+0000","last_became_peered":"2024-03-26T03:20:12.732132+0000","last_unstale":"2024-03-26T03:20:26.947754+0000","last_undegraded":"2024-03-26T03:20:26.947754+0000","last_fullsized":"2024-03-26T03:20:26.947754+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'33","last_scrub_stamp":"2024-03-26T03:17:57.160442+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:17:57.160442+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":277,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208064+0000","last_change":"2024-03-26T03:19:14.394760+0000","last_active":"2024-03-26T03:20:27.208064+0000","last_peered":"2024-03-26T03:20:27.208064+0000","last_clean":"2024-03-26T03:20:27.208064+0000","last_became_active":"2024-03-26T03:14:37.243678+0000","last_became_peered":"2024-03-26T03:14:37.243678+0000","last_unstale":"2024-03-26T03:20:27.208064+0000","last_undegraded":"2024-03-26T03:20:27.208064+0000","last_fullsized":"2024-03-26T03:20:27.208064+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:14.394668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.822728+0000","last_clean_scrub_stamp":"2024-03-26T03:19:14.394668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:33.267097+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":932,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329939+0000","last_change":"2024-03-26T03:20:12.732703+0000","last_active":"2024-03-26T03:20:26.329939+0000","last_peered":"2024-03-26T03:20:26.329939+0000","last_clean":"2024-03-26T03:20:26.329939+0000","last_became_active":"2024-03-26T03:20:12.732214+0000","last_became_peered":"2024-03-26T03:20:12.732214+0000","last_unstale":"2024-03-26T03:20:26.329939+0000","last_undegraded":"2024-03-26T03:20:26.329939+0000","last_fullsized":"2024-03-26T03:20:26.329939+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:08.060477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:20:08.060477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:15.963311+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.352713+0000","last_change":"2024-03-26T03:20:17.909291+0000","last_active":"2024-03-26T03:20:30.352713+0000","last_peered":"2024-03-26T03:20:30.352713+0000","last_clean":"2024-03-26T03:20:30.352713+0000","last_became_active":"2024-03-26T03:20:12.730912+0000","last_became_peered":"2024-03-26T03:20:12.730912+0000","last_unstale":"2024-03-26T03:20:30.352713+0000","last_undegraded":"2024-03-26T03:20:30.352713+0000","last_fullsized":"2024-03-26T03:20:30.352713+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:17.909140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:20:17.909140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:29.402450+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":621,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329809+0000","last_change":"2024-03-26T03:20:13.551782+0000","last_active":"2024-03-26T03:20:26.329809+0000","last_peered":"2024-03-26T03:20:26.329809+0000","last_clean":"2024-03-26T03:20:26.329809+0000","last_became_active":"2024-03-26T03:17:59.594759+0000","last_became_peered":"2024-03-26T03:17:59.594759+0000","last_unstale":"2024-03-26T03:20:26.329809+0000","last_undegraded":"2024-03-26T03:20:26.329809+0000","last_fullsized":"2024-03-26T03:20:26.329809+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:20:13.551684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:20:13.551684+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:40.231370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"446'15","reported_seq":99,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948100+0000","last_change":"2024-03-26T03:17:56.580380+0000","last_active":"2024-03-26T03:20:26.948100+0000","last_peered":"2024-03-26T03:20:26.948100+0000","last_clean":"2024-03-26T03:20:26.948100+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:20:26.948100+0000","last_undegraded":"2024-03-26T03:20:26.948100+0000","last_fullsized":"2024-03-26T03:20:26.948100+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'13","last_scrub_stamp":"2024-03-26T03:17:45.281375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:17:45.281375+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":888,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.208321+0000","last_change":"2024-03-26T03:20:12.734891+0000","last_active":"2024-03-26T03:20:27.208321+0000","last_peered":"2024-03-26T03:20:27.208321+0000","last_clean":"2024-03-26T03:20:27.208321+0000","last_became_active":"2024-03-26T03:20:12.734029+0000","last_became_peered":"2024-03-26T03:20:12.734029+0000","last_unstale":"2024-03-26T03:20:27.208321+0000","last_undegraded":"2024-03-26T03:20:27.208321+0000","last_fullsized":"2024-03-26T03:20:27.208321+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:05.285024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:20:05.285024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:25.762590+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":276,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948383+0000","last_change":"2024-03-26T03:20:12.719051+0000","last_active":"2024-03-26T03:20:26.948383+0000","last_peered":"2024-03-26T03:20:26.948383+0000","last_clean":"2024-03-26T03:20:26.948383+0000","last_became_active":"2024-03-26T03:20:12.718676+0000","last_became_peered":"2024-03-26T03:20:12.718676+0000","last_unstale":"2024-03-26T03:20:26.948383+0000","last_undegraded":"2024-03-26T03:20:26.948383+0000","last_fullsized":"2024-03-26T03:20:26.948383+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:51.182029+0000","last_clean_scrub_stamp":"2024-03-26T03:16:51.182029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":867,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.329718+0000","last_change":"2024-03-26T03:19:13.804053+0000","last_active":"2024-03-26T03:20:26.329718+0000","last_peered":"2024-03-26T03:20:26.329718+0000","last_clean":"2024-03-26T03:20:26.329718+0000","last_became_active":"2024-03-26T03:14:21.596806+0000","last_became_peered":"2024-03-26T03:14:21.596806+0000","last_unstale":"2024-03-26T03:20:26.329718+0000","last_undegraded":"2024-03-26T03:20:26.329718+0000","last_fullsized":"2024-03-26T03:20:26.329718+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:13.803940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:19:13.803940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:36.864830+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":674,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.545841+0000","last_change":"2024-03-26T03:17:59.610214+0000","last_active":"2024-03-26T03:20:30.545841+0000","last_peered":"2024-03-26T03:20:30.545841+0000","last_clean":"2024-03-26T03:20:30.545841+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:20:30.545841+0000","last_undegraded":"2024-03-26T03:20:30.545841+0000","last_fullsized":"2024-03-26T03:20:30.545841+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:17:51.516127+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:17:51.516127+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"446'27","reported_seq":919,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947405+0000","last_change":"2024-03-26T03:16:46.178208+0000","last_active":"2024-03-26T03:20:26.947405+0000","last_peered":"2024-03-26T03:20:26.947405+0000","last_clean":"2024-03-26T03:20:26.947405+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:20:26.947405+0000","last_undegraded":"2024-03-26T03:20:26.947405+0000","last_fullsized":"2024-03-26T03:20:26.947405+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'21","last_scrub_stamp":"2024-03-26T03:16:46.178099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:16:46.178099+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":562,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330446+0000","last_change":"2024-03-26T03:20:12.750678+0000","last_active":"2024-03-26T03:20:26.330446+0000","last_peered":"2024-03-26T03:20:26.330446+0000","last_clean":"2024-03-26T03:20:26.330446+0000","last_became_active":"2024-03-26T03:20:12.748421+0000","last_became_peered":"2024-03-26T03:20:12.748421+0000","last_unstale":"2024-03-26T03:20:26.330446+0000","last_undegraded":"2024-03-26T03:20:26.330446+0000","last_fullsized":"2024-03-26T03:20:26.330446+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:19:06.601677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:19:06.601677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:28.347943+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":879,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.544975+0000","last_change":"2024-03-26T03:15:20.809430+0000","last_active":"2024-03-26T03:20:30.544975+0000","last_peered":"2024-03-26T03:20:30.544975+0000","last_clean":"2024-03-26T03:20:30.544975+0000","last_became_active":"2024-03-26T03:14:37.242540+0000","last_became_peered":"2024-03-26T03:14:37.242540+0000","last_unstale":"2024-03-26T03:20:30.544975+0000","last_undegraded":"2024-03-26T03:20:30.544975+0000","last_fullsized":"2024-03-26T03:20:30.544975+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:15:20.809176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:15:20.809176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":873,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.948180+0000","last_change":"2024-03-26T03:17:43.076516+0000","last_active":"2024-03-26T03:20:26.948180+0000","last_peered":"2024-03-26T03:20:26.948180+0000","last_clean":"2024-03-26T03:20:26.948180+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:20:26.948180+0000","last_undegraded":"2024-03-26T03:20:26.948180+0000","last_fullsized":"2024-03-26T03:20:26.948180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.076334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.076334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":577,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:27.207826+0000","last_change":"2024-03-26T03:19:01.651338+0000","last_active":"2024-03-26T03:20:27.207826+0000","last_peered":"2024-03-26T03:20:27.207826+0000","last_clean":"2024-03-26T03:20:27.207826+0000","last_became_active":"2024-03-26T03:14:40.174111+0000","last_became_peered":"2024-03-26T03:14:40.174111+0000","last_unstale":"2024-03-26T03:20:27.207826+0000","last_undegraded":"2024-03-26T03:20:27.207826+0000","last_fullsized":"2024-03-26T03:20:27.207826+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:19:01.651232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:19:01.651232+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"446'64","reported_seq":692,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.330787+0000","last_change":"2024-03-26T03:20:12.794165+0000","last_active":"2024-03-26T03:20:26.330787+0000","last_peered":"2024-03-26T03:20:26.330787+0000","last_clean":"2024-03-26T03:20:26.330787+0000","last_became_active":"2024-03-26T03:20:12.748027+0000","last_became_peered":"2024-03-26T03:20:12.748027+0000","last_unstale":"2024-03-26T03:20:26.330787+0000","last_undegraded":"2024-03-26T03:20:26.330787+0000","last_fullsized":"2024-03-26T03:20:26.330787+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'64","last_scrub_stamp":"2024-03-26T03:19:09.592406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:19:09.592406+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:20:27.864063+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":33,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-03-26T03:20:30.354444+0000","last_change":"2024-03-26T03:20:30.043160+0000","last_active":"2024-03-26T03:20:30.354444+0000","last_peered":"2024-03-26T03:20:30.354444+0000","last_clean":"2024-03-26T03:20:30.354444+0000","last_became_active":"2024-03-26T03:20:12.734471+0000","last_became_peered":"2024-03-26T03:20:12.734471+0000","last_unstale":"2024-03-26T03:20:30.354444+0000","last_undegraded":"2024-03-26T03:20:30.354444+0000","last_fullsized":"2024-03-26T03:20:30.354444+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:20:30.043065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:30.043065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:21:49.047024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":88,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.947727+0000","last_change":"2024-03-26T03:17:56.572431+0000","last_active":"2024-03-26T03:20:26.947727+0000","last_peered":"2024-03-26T03:20:26.947727+0000","last_clean":"2024-03-26T03:20:26.947727+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:20:26.947727+0000","last_undegraded":"2024-03-26T03:20:26.947727+0000","last_fullsized":"2024-03-26T03:20:26.947727+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:17:43.615644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:17:43.615644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":547,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-03-26T03:20:26.331073+0000","last_change":"2024-03-26T03:18:58.798963+0000","last_active":"2024-03-26T03:20:26.331073+0000","last_peered":"2024-03-26T03:20:26.331073+0000","last_clean":"2024-03-26T03:20:26.331073+0000","last_became_active":"2024-03-26T03:14:37.265472+0000","last_became_peered":"2024-03-26T03:14:37.265472+0000","last_unstale":"2024-03-26T03:20:26.331073+0000","last_undegraded":"2024-03-26T03:20:26.331073+0000","last_fullsized":"2024-03-26T03:20:26.331073+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:18:58.798827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:18:58.798827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":3980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1245728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":2033792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1282,"num_read_kb":1277,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1825,"num_read_kb":1550,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":4488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1247,"ondisk_log_size":1247,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":249108103431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430475,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1116680,"kb_used_data":269184,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92628472,"statfs":{"total":95995035648,"available":94851555328,"internally_reserved":0,"allocated":275644416,"data_stored":275297737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":867826118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575140,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1119232,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92625920,"statfs":{"total":95995035648,"available":94848942080,"internally_reserved":0,"allocated":1040384,"data_stored":639165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816600,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92928552,"statfs":{"total":95995035648,"available":95158837248,"internally_reserved":0,"allocated":1064960,"data_stored":703212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411614,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924812,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820340,"statfs":{"total":95995035648,"available":95048028160,"internally_reserved":0,"allocated":987136,"data_stored":637962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175051,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477092,"kb_used_data":1052,"kb_used_omap":4,"kb_used_meta":476027,"kb_avail":93268060,"statfs":{"total":95995035648,"available":95506493440,"internally_reserved":0,"allocated":1077248,"data_stored":702586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":487452323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986453,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820408,"statfs":{"total":95995035648,"available":95048097792,"internally_reserved":0,"allocated":741376,"data_stored":374744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692113,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015072,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":1014078,"kb_avail":92730080,"statfs":{"total":95995035648,"available":94955601920,"internally_reserved":0,"allocated":1007616,"data_stored":637621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1038416325},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:20:37.278 INFO:tasks.thrashosds.thrasher:Setting 6.14 to [7, 3, 6] 2024-03-26T03:20:37.278 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.14', '7', '3', '6'] 2024-03-26T03:20:37.278 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 6.14 7 3 6 2024-03-26T03:20:37.554 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:37 smithi066 ceph-mon[29666]: pgmap v1020: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:20:37.554 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:37 smithi066 ceph-mon[37920]: pgmap v1020: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:20:37.599 INFO:teuthology.orchestra.run.smithi066.stdout:osd.0: {} 2024-03-26T03:20:37.613 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:37.623 INFO:teuthology.orchestra.run.smithi066.stdout:osd.1: {} 2024-03-26T03:20:37.651 INFO:teuthology.orchestra.run.smithi066.stdout:osd.2: {} 2024-03-26T03:20:37.658 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:37.673 INFO:teuthology.orchestra.run.smithi066.stdout:osd.3: {} 2024-03-26T03:20:37.694 INFO:teuthology.orchestra.run.smithi066.stdout:osd.4: {} 2024-03-26T03:20:37.715 INFO:teuthology.orchestra.run.smithi066.stdout:osd.5: {} 2024-03-26T03:20:37.735 INFO:teuthology.orchestra.run.smithi066.stdout:osd.6: {} 2024-03-26T03:20:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:37 smithi192 ceph-mon[33110]: pgmap v1020: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:20:37.757 INFO:teuthology.orchestra.run.smithi066.stdout:osd.7: {} 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:20:37.758 INFO:teuthology.orchestra.run.smithi066.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:20:38.563 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[37920]: osdmap e469: 8 total, 8 up, 7 in 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/168592499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[29666]: osdmap e469: 8 total, 8 up, 7 in 2024-03-26T03:20:38.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/168592499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:38.718 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:20:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:38 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:20:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:38 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:20:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:38 smithi192 ceph-mon[33110]: osdmap e469: 8 total, 8 up, 7 in 2024-03-26T03:20:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/168592499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:20:39.650 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:39 smithi066 ceph-mon[29666]: pgmap v1022: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:20:39.651 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:39 smithi066 ceph-mon[37920]: pgmap v1022: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:20:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:39 smithi192 ceph-mon[33110]: pgmap v1022: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:20:40.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:20:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:20:41.299 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:41.352 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:20:41.352 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:20:41.353 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:20:41.353 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:20:41.353 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:20:41.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:41.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:41.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:41.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-03-26T03:20:41.353 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 3]}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[29666]: pgmap v1023: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-03-26T03:20:41.354 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-03-26T03:20:41.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T03:20:41.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 3]}]: dispatch 2024-03-26T03:20:41.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]: dispatch 2024-03-26T03:20:41.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]: dispatch 2024-03-26T03:20:41.355 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:41 smithi066 ceph-mon[37920]: pgmap v1023: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:20:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 3]}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]: dispatch 2024-03-26T03:20:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:41 smithi192 ceph-mon[33110]: pgmap v1023: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:20:42.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-03-26T03:20:42.493 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 3]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[29666]: osdmap e470: 8 total, 8 up, 7 in 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 3]}]': finished 2024-03-26T03:20:42.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]': finished 2024-03-26T03:20:42.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-03-26T03:20:42.495 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:42 smithi066 ceph-mon[37920]: osdmap e470: 8 total, 8 up, 7 in 2024-03-26T03:20:42.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 5]}]': finished 2024-03-26T03:20:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-03-26T03:20:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-26T03:20:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.5", "id": [0, 3]}]': finished 2024-03-26T03:20:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 6]}]': finished 2024-03-26T03:20:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.19", "id": [3, 0]}]': finished 2024-03-26T03:20:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:42 smithi192 ceph-mon[33110]: osdmap e470: 8 total, 8 up, 7 in 2024-03-26T03:20:43.122 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:20:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:43 smithi066 ceph-mon[29666]: osdmap e471: 8 total, 8 up, 7 in 2024-03-26T03:20:43.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:43 smithi066 ceph-mon[29666]: pgmap v1026: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:43.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:43 smithi066 ceph-mon[37920]: osdmap e471: 8 total, 8 up, 7 in 2024-03-26T03:20:43.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:43 smithi066 ceph-mon[37920]: pgmap v1026: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:43 smithi192 ceph-mon[33110]: osdmap e471: 8 total, 8 up, 7 in 2024-03-26T03:20:43.867 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:43 smithi192 ceph-mon[33110]: pgmap v1026: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:20:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:20:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:20:44.254 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:44.276 INFO:teuthology.orchestra.run.smithi066.stderr:set 6.14 pg_upmap mapping to [7,3,6] 2024-03-26T03:20:44.494 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1794590465' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.14", "id": [7, 3, 6]}]: dispatch 2024-03-26T03:20:44.494 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1794590465' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.14", "id": [7, 3, 6]}]: dispatch 2024-03-26T03:20:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1794590465' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.14", "id": [7, 3, 6]}]: dispatch 2024-03-26T03:20:45.295 INFO:teuthology.orchestra.run.smithi066.stderr:noscrub is unset 2024-03-26T03:20:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1794590465' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.14", "id": [7, 3, 6]}]': finished 2024-03-26T03:20:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[29666]: osdmap e472: 8 total, 8 up, 7 in 2024-03-26T03:20:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[29666]: pgmap v1028: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1222747608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:20:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1794590465' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.14", "id": [7, 3, 6]}]': finished 2024-03-26T03:20:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[37920]: osdmap e472: 8 total, 8 up, 7 in 2024-03-26T03:20:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[37920]: pgmap v1028: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:45.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1222747608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:20:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1794590465' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.14", "id": [7, 3, 6]}]': finished 2024-03-26T03:20:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:45 smithi192 ceph-mon[33110]: osdmap e472: 8 total, 8 up, 7 in 2024-03-26T03:20:45.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:45 smithi192 ceph-mon[33110]: pgmap v1028: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:45.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1222747608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:20:46.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[29666]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-26T03:20:46.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:20:46.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1222747608' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:20:46.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[29666]: osdmap e473: 8 total, 8 up, 7 in 2024-03-26T03:20:46.576 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[37920]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-26T03:20:46.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:20:46.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1222747608' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:20:46.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:46 smithi066 ceph-mon[37920]: osdmap e473: 8 total, 8 up, 7 in 2024-03-26T03:20:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:46 smithi192 ceph-mon[33110]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-26T03:20:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:46 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:20:46.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:46 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1222747608' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:20:46.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:46 smithi192 ceph-mon[33110]: osdmap e473: 8 total, 8 up, 7 in 2024-03-26T03:20:47.302 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:20:47.570 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:47 smithi066 ceph-mon[29666]: pgmap v1030: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 977 B/s rd, 0 op/s; 62 B/s, 4 objects/s recovering 2024-03-26T03:20:47.570 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:47 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:20:47.571 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:47 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:20:47.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:47 smithi066 ceph-mon[37920]: pgmap v1030: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 977 B/s rd, 0 op/s; 62 B/s, 4 objects/s recovering 2024-03-26T03:20:47.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:47 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:20:47.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:47 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:20:47.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:47 smithi192 ceph-mon[33110]: pgmap v1030: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 977 B/s rd, 0 op/s; 62 B/s, 4 objects/s recovering 2024-03-26T03:20:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:47 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:20:47.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:47 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:20:47.858 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:47.862 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:20:47.862 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:20:47.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:20:47.862 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:20:47.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83057 11.c 11:33f61b63:::benchmark_data_smithi066_175179_object83056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.584670+0000", 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 172.27478600500001, 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.096656372, 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83057 2024-03-26T03:20:47.863 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.584670+0000", 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.584670+0000", 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.584671+0000", 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4200000000000005e-07 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.584701+0000", 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.012e-05 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.584709+0000", 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7910000000000005e-06 2024-03-26T03:20:47.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.584715+0000", 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5080000000000002e-06 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.586320+0000", 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0016051710000000001 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.586331+0000", 2024-03-26T03:20:47.866 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0791000000000001e-05 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560510+0000", 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97417944700000003 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560525+0000", 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4484999999999999e-05 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.674357+0000", 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1138324690000001 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.674480+0000", 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000122946 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.674586+0000", 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000105561 2024-03-26T03:20:47.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.676590+0000", 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020041210000000002 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.680949+0000", 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043589579999999996 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681010+0000", 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1062000000000006e-05 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681158+0000", 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014827199999999999 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681223+0000", 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4769000000000003e-05 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681326+0000", 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000103059 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83064 11.c 11:352bf786:::benchmark_data_smithi066_175179_object83063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.593598+0000", 2024-03-26T03:20:47.871 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 172.26585744400001, 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0880686850000001, 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83064 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.593598+0000", 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.593598+0000", 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.593600+0000", 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1090000000000001e-06 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.593625+0000", 2024-03-26T03:20:47.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4975999999999999e-05 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.593642+0000", 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6464e-05 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.593651+0000", 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8149999999999995e-06 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.594416+0000", 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00076554699999999999 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.594428+0000", 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1610000000000001e-05 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560553+0000", 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96612483000000005 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.875 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560561+0000", 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7090000000000008e-06 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.676649+0000", 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.116087729 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.876 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.877 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.676791+0000", 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014166499999999999 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.676901+0000", 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000110361 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681049+0000", 2024-03-26T03:20:47.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0041477559999999998 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681093+0000", 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4005000000000003e-05 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681345+0000", 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025239899999999999 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681461+0000", 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000115346 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681535+0000", 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4504000000000004e-05 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681667+0000", 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013186 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83114 11.c 11:33d4e87b:::benchmark_data_smithi066_175179_object83113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:20:47.881 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.628154+0000", 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 172.23130192100001, 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0537899730000002, 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83114 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.882 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628154+0000", 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628154+0000", 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628155+0000", 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5060000000000001e-06 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628175+0000", 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0035999999999999e-05 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628184+0000", 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1880000000000002e-06 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628189+0000", 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7810000000000002e-06 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628234+0000", 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.477e-05 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.628241+0000", 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.888e-06 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560643+0000", 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93240194600000004 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560653+0000", 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8889999999999997e-06 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.678324+0000", 2024-03-26T03:20:47.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.117670927 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.678459+0000", 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013500200000000001 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.678660+0000", 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020132099999999999 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681351+0000", 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002691124 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681412+0000", 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0575999999999997e-05 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681720+0000", 2024-03-26T03:20:47.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00030809700000000001 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681764+0000", 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4490999999999998e-05 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681828+0000", 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3479999999999995e-05 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.681944+0000", 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000115951 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83147 11.c 11:30e3b99e:::benchmark_data_smithi066_175179_object83146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.650110+0000", 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 172.20934592899999, 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.045129024, 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.891 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83147 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650110+0000", 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650110+0000", 2024-03-26T03:20:47.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650111+0000", 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.387e-06 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650131+0000", 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9695e-05 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650141+0000", 2024-03-26T03:20:47.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6989999999999998e-06 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650150+0000", 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6099999999999995e-06 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650221+0000", 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1107999999999997e-05 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.650245+0000", 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3550999999999999e-05 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560682+0000", 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91043665399999996 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560690+0000", 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9069999999999996e-06 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.682148+0000", 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1214577139999999 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.682302+0000", 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015342399999999999 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.682539+0000", 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00023703900000000001 2024-03-26T03:20:47.896 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.686979+0000", 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0044403819999999997 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.687034+0000", 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5136000000000002e-05 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694869+0000", 2024-03-26T03:20:47.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0078351370000000007 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695045+0000", 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017595299999999999 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695111+0000", 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5499000000000004e-05 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695239+0000", 2024-03-26T03:20:47.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012812899999999999 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83155 11.c 11:3edc40b1:::benchmark_data_smithi066_175179_object83154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.666534+0000", 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 172.19292199899999, 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.029291094, 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.899 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83155 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666534+0000", 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.900 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666534+0000", 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666535+0000", 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.426e-06 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666555+0000", 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9443999999999999e-05 2024-03-26T03:20:47.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666565+0000", 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.058e-05 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666572+0000", 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1509999999999998e-06 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666625+0000", 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2419000000000001e-05 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.666632+0000", 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9979999999999997e-06 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560721+0000", 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.89408876000000004 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.903 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.560730+0000", 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9749999999999996e-06 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684468+0000", 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1237388720000001 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684687+0000", 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021911700000000001 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684864+0000", 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017683900000000001 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694849+0000", 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0099846320000000002 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694878+0000", 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.8722999999999999e-05 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694983+0000", 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000105163 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695305+0000", 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00032172599999999999 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695603+0000", 2024-03-26T03:20:47.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000298738 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695825+0000", 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00022153099999999999 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83024 11.e 11:788c2b4b:::benchmark_data_smithi066_175179_object83023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e428)", 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:56.563448+0000", 2024-03-26T03:20:47.907 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 171.29600793700001, 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1206407140000001, 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83024 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563448+0000", 2024-03-26T03:20:47.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563448+0000", 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563450+0000", 2024-03-26T03:20:47.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1320000000000001e-06 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563503+0000", 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2818000000000002e-05 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563515+0000", 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2211e-05 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.910 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563525+0000", 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.041e-05 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572771+0000", 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0092455430000000002 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572780+0000", 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6030000000000005e-06 2024-03-26T03:20:47.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684017+0000", 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1112376180000001 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684089+0000", 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1378999999999995e-05 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.912 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83048 11.e 11:7155dac0:::benchmark_data_smithi066_175179_object83047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e428)", 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:56.563542+0000", 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 171.29591404300001, 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.126796659, 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83048 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.913 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563542+0000", 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563542+0000", 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563542+0000", 2024-03-26T03:20:47.914 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2799999999999995e-07 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563586+0000", 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.35e-05 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563594+0000", 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5769999999999996e-06 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563600+0000", 2024-03-26T03:20:47.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6649999999999997e-06 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572800+0000", 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0091998500000000007 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572807+0000", 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3010000000000001e-06 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684491+0000", 2024-03-26T03:20:47.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.111683405 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684687+0000", 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000196174 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,4,6", 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684981+0000", 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00029361099999999999 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.689973+0000", 2024-03-26T03:20:47.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049924130000000002 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.689990+0000", 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7473e-05 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690057+0000", 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6934000000000007e-05 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690159+0000", 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000101442 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690225+0000", 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6511000000000001e-05 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690338+0000", 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113075 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83051 11.e 11:7a2786eb:::benchmark_data_smithi066_175179_object83050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e428)", 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:56.563609+0000", 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 171.295846888, 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1317009229999999, 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83051 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.920 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563609+0000", 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563609+0000", 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563609+0000", 2024-03-26T03:20:47.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9500000000000002e-07 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563653+0000", 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3297999999999998e-05 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563660+0000", 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7870000000000004e-06 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563666+0000", 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5690000000000001e-06 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572830+0000", 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0091638529999999996 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572837+0000", 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3100000000000003e-06 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.687110+0000", 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.114272527 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.687258+0000", 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014846799999999999 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,4,6", 2024-03-26T03:20:47.924 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.687419+0000", 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016089699999999999 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690104+0000", 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00268512 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690355+0000", 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000250616 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.925 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690393+0000", 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8556999999999997e-05 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695123+0000", 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047296960000000002 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695185+0000", 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1394000000000002e-05 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.926 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695310+0000", 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012523600000000001 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83120 11.e 11:7b646efc:::benchmark_data_smithi066_175179_object83119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e428)", 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:56.563674+0000", 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 171.29578181700001, 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.132309397, 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.927 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83120 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563674+0000", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563674+0000", 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.928 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563675+0000", 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1099999999999995e-07 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563722+0000", 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7890999999999999e-05 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563729+0000", 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6819999999999997e-06 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.929 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563735+0000", 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3369999999999996e-06 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572862+0000", 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0091268849999999995 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572870+0000", 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7389999999999999e-06 2024-03-26T03:20:47.930 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690602+0000", 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1177321469999999 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690782+0000", 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017948199999999999 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.931 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,4,6", 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.690973+0000", 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000191728 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695472+0000", 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004498401 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695509+0000", 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7067000000000003e-05 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.932 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695664+0000", 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015550399999999999 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695786+0000", 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012119 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695845+0000", 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9639000000000003e-05 2024-03-26T03:20:47.933 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695983+0000", 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013809400000000001 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.934 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.935 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83129 11.e 11:721e1c15:::benchmark_data_smithi066_175179_object83128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e428)", 2024-03-26T03:20:47.935 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:56.563743+0000", 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 171.295712815, 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.146672707, 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83129 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.936 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563743+0000", 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563743+0000", 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563744+0000", 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9100000000000004e-07 2024-03-26T03:20:47.937 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563793+0000", 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9364999999999999e-05 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563800+0000", 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6640000000000001e-06 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563806+0000", 2024-03-26T03:20:47.938 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3060000000000003e-06 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572902+0000", 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0090959669999999999 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572909+0000", 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0500000000000003e-06 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.939 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.693326+0000", 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.12041671 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.693465+0000", 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000139061 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,4,6", 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.693592+0000", 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012775199999999999 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.940 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695683+0000", 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020907819999999998 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.695736+0000", 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2562000000000003e-05 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.700947+0000", 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.005210826 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.941 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.710225+0000", 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0092779260000000006 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.710286+0000", 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1433999999999995e-05 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.710416+0000", 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012971099999999999 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.942 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83145 11.e 11:79ff132f:::benchmark_data_smithi066_175179_object83144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e428)", 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:56.563813+0000", 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 171.295642259, 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.146893232, 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83145 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.943 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563813+0000", 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563813+0000", 2024-03-26T03:20:47.944 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563814+0000", 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3300000000000002e-07 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563860+0000", 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6241e-05 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.945 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563867+0000", 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8510000000000001e-06 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563873+0000", 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0079999999999999e-06 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.946 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572937+0000", 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0090632799999999999 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.572946+0000", 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1160000000000005e-06 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.696028+0000", 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.123082804 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.947 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.696167+0000", 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013824 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 3,4,6", 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.696322+0000", 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000155136 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.700962+0000", 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046398660000000003 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.948 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.701007+0000", 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5405999999999999e-05 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.709900+0000", 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0088930659999999998 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.710475+0000", 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00057464199999999999 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.949 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.710536+0000", 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1523e-05 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.710707+0000", 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017042000000000001 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.950 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179257 11.c 11:3bd2ce61:::benchmark_data_smithi066_175179_object47359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.365724+0000", 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.493731292000007, 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0900052840000001, 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179257 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.951 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365724+0000", 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365724+0000", 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365726+0000", 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8810000000000001e-06 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.952 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365732+0000", 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1630000000000004e-06 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365740+0000", 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9160000000000007e-06 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365748+0000", 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2080000000000003e-06 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.368151+0000", 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024030200000000001 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.368223+0000", 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2378000000000004e-05 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.368293+0000", 2024-03-26T03:20:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0426999999999995e-05 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.378651+0000", 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010357290999999999 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.378719+0000", 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8176999999999995e-05 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392630+0000", 2024-03-26T03:20:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013911315 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392639+0000", 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4700000000000002e-06 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.455641+0000", 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0630026889999999 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.455730+0000", 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8349000000000004e-05 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179258 11.c 11:378d687d:::benchmark_data_smithi066_175179_object47360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.365994+0000", 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.493461972000006, 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.09612602, 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179258 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365994+0000", 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365994+0000", 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.365995+0000", 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1999999999999999e-06 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.366000+0000", 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7990000000000001e-06 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.366010+0000", 2024-03-26T03:20:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0315e-05 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.366017+0000", 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1640000000000004e-06 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373244+0000", 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0072270750000000003 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.375803+0000", 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0025584499999999999 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.375956+0000", 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015360699999999999 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392666+0000", 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.016709643 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392673+0000", 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.725e-06 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456009+0000", 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.063336091 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for missing object", 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456140+0000", 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000130755 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462010+0000", 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0058703380000000001 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462120+0000", 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000109858 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179264 11.c 11:31c9259c:::benchmark_data_smithi066_175179_object47366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.373337+0000", 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.486118374, 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0893516189999999, 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179264 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373337+0000", 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373337+0000", 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373339+0000", 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.719e-06 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373345+0000", 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3679999999999998e-06 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373355+0000", 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7200000000000001e-06 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.373364+0000", 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0569999999999999e-06 2024-03-26T03:20:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.378755+0000", 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053908879999999999 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.378892+0000", 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013693399999999999 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4,5", 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.379020+0000", 2024-03-26T03:20:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000127773 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392699+0000", 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013678935 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392705+0000", 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5200000000000003e-06 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456198+0000", 2024-03-26T03:20:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0634924269999999 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for missing object", 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456284+0000", 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6173000000000003e-05 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462598+0000", 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0063144289999999999 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462689+0000", 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0675999999999997e-05 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179265 11.c 11:3452e687:::benchmark_data_smithi066_175179_object47367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.376207+0000", 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.483248802000006, 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0868468790000001, 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179265 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.376207+0000", 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.376207+0000", 2024-03-26T03:20:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.376208+0000", 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2839999999999999e-06 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.376212+0000", 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9249999999999997e-06 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.376220+0000", 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7160000000000003e-06 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.376228+0000", 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6590000000000007e-06 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392727+0000", 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.016499650000000001 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.392734+0000", 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5420000000000002e-06 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456324+0000", 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.063590313 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456461+0000", 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013675100000000001 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.456620+0000", 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000159455 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462114+0000", 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0054941180000000001 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462727+0000", 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00061279499999999996 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462874+0000", 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014678799999999999 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462941+0000", 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7310000000000004e-05 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463054+0000", 2024-03-26T03:20:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011257299999999999 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179290 11.c 11:3b10f549:::benchmark_data_smithi066_175179_object47392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.397985+0000", 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.461470632000001, 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.065277858, 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179290 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.397985+0000", 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.397985+0000", 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.397987+0000", 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4699999999999999e-06 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.397991+0000", 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5689999999999998e-06 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.398000+0000", 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6519999999999995e-06 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.398006+0000", 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2190000000000001e-06 2024-03-26T03:20:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.398090+0000", 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4270999999999993e-05 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.398099+0000", 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8179999999999993e-06 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.458410+0000", 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.060311088 2024-03-26T03:20:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.458548+0000", 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000137711 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.458706+0000", 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015765399999999999 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462119+0000", 2024-03-26T03:20:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.003413719 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.462784+0000", 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00066443899999999996 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463101+0000", 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00031679599999999999 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463154+0000", 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3705000000000002e-05 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463263+0000", 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010874699999999999 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179338 11.c 11:3d8d5825:::benchmark_data_smithi066_175179_object47440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.430027+0000", 2024-03-26T03:20:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.429428509000005, 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0395928720000001, 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179338 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430027+0000", 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430027+0000", 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430029+0000", 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4279999999999999e-06 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430033+0000", 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7219999999999999e-06 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430044+0000", 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0241000000000001e-05 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430051+0000", 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6960000000000005e-06 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430613+0000", 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00056123299999999996 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.430620+0000", 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8790000000000003e-06 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463517+0000", 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0328969189999999 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463630+0000", 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011263 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.463741+0000", 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000111506 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.469347+0000", 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0056050229999999998 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.469392+0000", 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5068999999999998e-05 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.469434+0000", 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2218000000000001e-05 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.469492+0000", 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7899000000000002e-05 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.469620+0000", 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012840900000000001 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179358 11.c 11:3df07788:::benchmark_data_smithi066_175179_object47460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-26T03:20:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.442571+0000", 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.416884738999997, 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0435168800000001, 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179358 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442571+0000", 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442571+0000", 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442572+0000", 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.423e-06 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442578+0000", 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5269999999999998e-06 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442622+0000", 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4119000000000002e-05 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442630+0000", 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4970000000000004e-06 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442755+0000", 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012571899999999999 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.442763+0000", 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.157e-06 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.469980+0000", 2024-03-26T03:20:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0272168100000001 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.470050+0000", 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9340999999999999e-05 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.470134+0000", 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4600999999999999e-05 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.473819+0000", 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036844960000000002 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.478022+0000", 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042032090000000003 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.485939+0000", 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.007917014 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.485988+0000", 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9081e-05 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.486088+0000", 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9885999999999999e-05 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179360 11.c 11:3a7caee9:::benchmark_data_smithi066_175179_object47462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.443081+0000", 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.416374677999997, 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0513240960000001, 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179360 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443081+0000", 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443081+0000", 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443082+0000", 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1599999999999999e-06 2024-03-26T03:20:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443087+0000", 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7299999999999996e-06 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443099+0000", 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2446000000000001e-05 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443107+0000", 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2049999999999996e-06 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443153+0000", 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6743999999999998e-05 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.443161+0000", 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.221e-06 2024-03-26T03:20:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.471863+0000", 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.02870262 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.471973+0000", 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000110042 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.472087+0000", 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113891 2024-03-26T03:20:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.475841+0000", 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0037543390000000002 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.478058+0000", 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022170129999999999 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.492923+0000", 2024-03-26T03:20:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.014864087 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.492974+0000", 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1762999999999999e-05 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.494405+0000", 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014308350000000001 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:179370 11.c 11:3c06f63f:::benchmark_data_smithi066_175179_object47472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:19:15.449763+0000", 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 92.409693189999999, 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.044912678, 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:20:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 179370 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449763+0000", 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449763+0000", 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449764+0000", 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5910000000000001e-06 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449769+0000", 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2190000000000003e-06 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449780+0000", 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0502e-05 2024-03-26T03:20:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449786+0000", 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7089999999999999e-06 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449848+0000", 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2780999999999994e-05 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:15.449857+0000", 2024-03-26T03:20:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3480000000000004e-06 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.473852+0000", 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.023995636 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.473976+0000", 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000123473 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.474096+0000", 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012012599999999999 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.478092+0000", 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0039964670000000001 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.480963+0000", 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028702010000000002 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.494493+0000", 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013530054999999999 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.494560+0000", 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7078999999999999e-05 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:19:16.494675+0000", 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000115491 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:20:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:20:48.013 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:20:48.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 6.a deep-scrub starts 2024-03-26T03:20:48.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 5.1c scrub starts 2024-03-26T03:20:48.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 6.a deep-scrub ok 2024-03-26T03:20:48.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 5.1c scrub ok 2024-03-26T03:20:48.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 5.13 deep-scrub starts 2024-03-26T03:20:48.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 5.13 deep-scrub ok 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 6.a deep-scrub starts 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 5.1c scrub starts 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 6.a deep-scrub ok 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 5.1c scrub ok 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 5.13 deep-scrub starts 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 5.13 deep-scrub ok 2024-03-26T03:20:48.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:20:48.457 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:48 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:20:48.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 6.a deep-scrub starts 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 5.1c scrub starts 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 6.a deep-scrub ok 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 5.1c scrub ok 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 5.13 deep-scrub starts 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 5.13 deep-scrub ok 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:20:48.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:48 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:20:49.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: 2.d scrub starts 2024-03-26T03:20:49.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: 2.d scrub ok 2024-03-26T03:20:49.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:20:49.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:20:49.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: 1.6 scrub starts 2024-03-26T03:20:49.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: 1.6 scrub ok 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: osdmap e474: 8 total, 8 up, 7 in 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[29666]: pgmap v1032: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 3 objects/s recovering 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: 2.d scrub starts 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: 2.d scrub ok 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: 1.6 scrub starts 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: 1.6 scrub ok 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: osdmap e474: 8 total, 8 up, 7 in 2024-03-26T03:20:49.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:49 smithi066 ceph-mon[37920]: pgmap v1032: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 3 objects/s recovering 2024-03-26T03:20:49.571 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:20:49.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: 2.d scrub starts 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: 2.d scrub ok 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: 1.6 scrub starts 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: 1.6 scrub ok 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: osdmap e474: 8 total, 8 up, 7 in 2024-03-26T03:20:49.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:49 smithi192 ceph-mon[33110]: pgmap v1032: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 3 objects/s recovering 2024-03-26T03:20:50.059 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:50.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:50 smithi192 ceph-mon[33110]: 4.9 deep-scrub starts 2024-03-26T03:20:50.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:50 smithi192 ceph-mon[33110]: 4.9 deep-scrub ok 2024-03-26T03:20:50.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:20:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:20:50.540 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:50 smithi066 ceph-mon[29666]: 4.9 deep-scrub starts 2024-03-26T03:20:50.540 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:50 smithi066 ceph-mon[29666]: 4.9 deep-scrub ok 2024-03-26T03:20:50.541 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:50 smithi066 ceph-mon[37920]: 4.9 deep-scrub starts 2024-03-26T03:20:50.541 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:50 smithi066 ceph-mon[37920]: 4.9 deep-scrub ok 2024-03-26T03:20:51.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:20:51.358 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[37920]: pgmap v1033: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 823 B/s rd, 0 op/s; 53 B/s, 3 objects/s recovering 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:20:51.359 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:51 smithi066 ceph-mon[29666]: pgmap v1033: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 823 B/s rd, 0 op/s; 53 B/s, 3 objects/s recovering 2024-03-26T03:20:51.497 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T03:20:51.497 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-26T03:20:51.497 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd in 7 2024-03-26T03:20:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:51 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:20:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:51 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:20:51.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:20:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:51 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:20:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:51 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:20:51.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:51 smithi192 ceph-mon[33110]: pgmap v1033: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 823 B/s rd, 0 op/s; 53 B/s, 3 objects/s recovering 2024-03-26T03:20:52.179 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:52.222 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:20:52.222 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:20:52.222 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 0 2024-03-26T03:20:52.223 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:20:52.299 INFO:teuthology.orchestra.run.smithi066.stderr:nodeep-scrub is unset 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[29666]: 1.1a scrub starts 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[29666]: 1.1a scrub ok 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4191871379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[29666]: 4.1f scrub starts 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[29666]: 4.1f scrub ok 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[37920]: 1.1a scrub starts 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[37920]: 1.1a scrub ok 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4191871379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[37920]: 4.1f scrub starts 2024-03-26T03:20:52.441 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:52 smithi066 ceph-mon[37920]: 4.1f scrub ok 2024-03-26T03:20:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:52 smithi192 ceph-mon[33110]: 1.1a scrub starts 2024-03-26T03:20:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:52 smithi192 ceph-mon[33110]: 1.1a scrub ok 2024-03-26T03:20:52.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:52 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4191871379' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:20:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:52 smithi192 ceph-mon[33110]: 4.1f scrub starts 2024-03-26T03:20:52.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:52 smithi192 ceph-mon[33110]: 4.1f scrub ok 2024-03-26T03:20:53.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4191871379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: osdmap e475: 8 total, 8 up, 7 in 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: pgmap v1035: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 15 B/s, 1 objects/s recovering 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: 4.14 scrub starts 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: 4.14 scrub ok 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4191871379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: osdmap e475: 8 total, 8 up, 7 in 2024-03-26T03:20:53.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: pgmap v1035: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 15 B/s, 1 objects/s recovering 2024-03-26T03:20:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: 4.14 scrub starts 2024-03-26T03:20:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: 4.14 scrub ok 2024-03-26T03:20:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:20:53.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:20:53.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4191871379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:20:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: osdmap e475: 8 total, 8 up, 7 in 2024-03-26T03:20:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: pgmap v1035: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 15 B/s, 1 objects/s recovering 2024-03-26T03:20:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: 4.14 scrub starts 2024-03-26T03:20:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: 4.14 scrub ok 2024-03-26T03:20:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:20:53.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:53 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:20:54.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:20:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:20:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:20:54.188 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:20:54.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:54 smithi066 ceph-mon[37920]: 1.d scrub starts 2024-03-26T03:20:54.564 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:54 smithi066 ceph-mon[37920]: 1.d scrub ok 2024-03-26T03:20:54.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:54 smithi066 ceph-mon[29666]: 1.d scrub starts 2024-03-26T03:20:54.564 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:54 smithi066 ceph-mon[29666]: 1.d scrub ok 2024-03-26T03:20:54.643 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:54 smithi192 ceph-mon[33110]: 1.d scrub starts 2024-03-26T03:20:54.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:54 smithi192 ceph-mon[33110]: 1.d scrub ok 2024-03-26T03:20:55.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:20:55.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:20:55.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: 5.8 scrub starts 2024-03-26T03:20:55.454 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: 5.8 scrub ok 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: pgmap v1036: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: 4.15 deep-scrub starts 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: 4.15 deep-scrub ok 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: 5.8 scrub starts 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: 5.8 scrub ok 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: pgmap v1036: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: 4.15 deep-scrub starts 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: 4.15 deep-scrub ok 2024-03-26T03:20:55.455 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:55.456 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:20:55.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: 5.8 scrub starts 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: 5.8 scrub ok 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: pgmap v1036: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: 4.15 deep-scrub starts 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: 4.15 deep-scrub ok 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:20:55.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:20:55.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:55 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29662]: 2024-03-26T03:20:55.545+0000 7f94a71dc700 -1 mon.a@0(leader).osd e475 definitely_dead 0 2024-03-26T03:20:56.336 INFO:teuthology.orchestra.run.smithi066.stderr:marked in osd.7. 2024-03-26T03:20:56.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1670873792' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T03:20:56.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1670873792' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T03:20:56.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1670873792' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T03:20:56.994 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:20:56.994 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:20:56.994 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:20:56.994 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:20:56.994 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:20:57.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1670873792' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-26T03:20:57.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[29666]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T03:20:57.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[29666]: pgmap v1038: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1670873792' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[37920]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[37920]: pgmap v1038: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:20:57.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:57 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:20:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:57 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1670873792' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-26T03:20:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:57 smithi192 ceph-mon[33110]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T03:20:57.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:57 smithi192 ceph-mon[33110]: pgmap v1038: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:20:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:57 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:20:57.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:57 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:20:57.755 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:20:58.324 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-26T03:20:58.324 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T03:20:58.324 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 31 2024-03-26T03:20:58.448 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:58.571 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: 5.12 scrub starts 2024-03-26T03:20:58.571 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: 5.12 scrub ok 2024-03-26T03:20:58.571 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T03:20:58.571 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: 4.11 scrub starts 2024-03-26T03:20:58.571 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: 4.11 scrub ok 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: 6.0 scrub starts 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[29666]: 6.0 scrub ok 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: 5.12 scrub starts 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: 5.12 scrub ok 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: 4.11 scrub starts 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: 4.11 scrub ok 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: 6.0 scrub starts 2024-03-26T03:20:58.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:58 smithi066 ceph-mon[37920]: 6.0 scrub ok 2024-03-26T03:20:58.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: 5.12 scrub starts 2024-03-26T03:20:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: 5.12 scrub ok 2024-03-26T03:20:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T03:20:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: 4.11 scrub starts 2024-03-26T03:20:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: 4.11 scrub ok 2024-03-26T03:20:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: 6.0 scrub starts 2024-03-26T03:20:58.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:58 smithi192 ceph-mon[33110]: 6.0 scrub ok 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: 4.c scrub starts 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: 4.c scrub ok 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: pgmap v1041: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: 2.19 scrub starts 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: 2.19 scrub ok 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[29666]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T03:20:59.581 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: 4.c scrub starts 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: 4.c scrub ok 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: pgmap v1041: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: 2.19 scrub starts 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: 2.19 scrub ok 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:20:59.582 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:20:59 smithi066 ceph-mon[37920]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T03:20:59.636 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:20:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:20:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:20:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T03:20:59.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: 4.c scrub starts 2024-03-26T03:20:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: 4.c scrub ok 2024-03-26T03:20:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: pgmap v1041: 162 pgs: 162 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:20:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: 2.19 scrub starts 2024-03-26T03:20:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: 2.19 scrub ok 2024-03-26T03:20:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:20:59.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:20:59 smithi192 ceph-mon[33110]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T03:21:00.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: 6.16 scrub starts 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: 6.16 scrub ok 2024-03-26T03:21:00.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:00 smithi192 ceph-mon[33110]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: 6.16 scrub starts 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: 6.16 scrub ok 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[29666]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T03:21:00.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: 6.16 scrub starts 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: 6.16 scrub ok 2024-03-26T03:21:00.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:00 smithi066 ceph-mon[37920]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T03:21:00.776 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137118 10.5 10:afbc2354:::benchmark_data_smithi066_148944_object12471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.188180+0000", 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.58537763099997, 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.265863966, 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.777 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137118 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188180+0000", 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188180+0000", 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.778 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188181+0000", 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2100000000000001e-06 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188185+0000", 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4789999999999998e-06 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188198+0000", 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2984e-05 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.779 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.188206+0000", 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9969999999999998e-06 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.203599+0000", 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.015392407 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.203733+0000", 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000134074 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.780 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,7", 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.203871+0000", 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000138541 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.213765+0000", 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0098930890000000007 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.213865+0000", 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100188 2024-03-26T03:21:00.781 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.213922+0000", 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7179999999999998e-05 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.248852+0000", 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.034929819000000001 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.248864+0000", 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2397e-05 2024-03-26T03:21:00.782 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.453940+0000", 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2050763520000001 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.454044+0000", 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000103249 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.783 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137142 10.5 10:a16c8a26:::benchmark_data_smithi066_148944_object12495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.215641+0000", 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.55791628999998, 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4465264339999999, 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137142 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.784 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215641+0000", 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215641+0000", 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215646+0000", 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3460000000000001e-06 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.785 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215652+0000", 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.891e-06 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215663+0000", 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0343000000000001e-05 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.215672+0000", 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1600000000000004e-06 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.786 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.249222+0000", 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.033549764000000003 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.249230+0000", 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6799999999999999e-06 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.455981+0000", 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2067507580000001 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.787 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.456156+0000", 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000175243 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.456608+0000", 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00045140399999999998 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465016+0000", 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0084080200000000004 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.788 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465111+0000", 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5457000000000004e-05 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465149+0000", 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8049000000000001e-05 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465360+0000", 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021100700000000001 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.789 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.790 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.660380+0000", 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19501934300000001 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.660419+0000", 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9712999999999997e-05 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.662168+0000", 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017482559999999999 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.791 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137155 10.8 10:1e4e6b57:::benchmark_data_smithi066_148944_object12508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.256026+0000", 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.517531308, 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3845877340000001, 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137155 2024-03-26T03:21:00.792 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256026+0000", 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256026+0000", 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256028+0000", 2024-03-26T03:21:00.793 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0250000000000001e-06 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256035+0000", 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.477e-06 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256047+0000", 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.236e-05 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.794 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256057+0000", 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.825e-06 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256130+0000", 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2993999999999999e-05 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.256148+0000", 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.859e-05 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.795 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230452+0000", 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97430318100000002 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230465+0000", 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3777999999999999e-05 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.321474+0000", 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0910083530000001 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.796 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.321615+0000", 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000141803 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.322034+0000", 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000418694 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331259+0000", 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0092250690000000007 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.797 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.337787+0000", 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0065275790000000004 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339170+0000", 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001382912 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640470+0000", 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.30130003300000002 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.798 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640511+0000", 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1180999999999998e-05 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640614+0000", 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010288 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137157 10.5 10:ac4af61a:::benchmark_data_smithi066_148944_object12510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.799 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.260244+0000", 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.51331319899998, 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.403910816, 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137157 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260244+0000", 2024-03-26T03:21:00.800 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260244+0000", 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260246+0000", 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9039999999999999e-06 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260253+0000", 2024-03-26T03:21:00.801 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7429999999999996e-06 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260266+0000", 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2982999999999999e-05 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260274+0000", 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.106e-06 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260343+0000", 2024-03-26T03:21:00.802 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8911999999999993e-05 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.260353+0000", 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6690000000000001e-06 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.462763+0000", 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.202410202 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.803 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.462909+0000", 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000146649 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.463028+0000", 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118212 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465296+0000", 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002267958 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.804 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465316+0000", 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0967000000000001e-05 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465412+0000", 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5011000000000002e-05 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470453+0000", 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0050418879999999996 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.805 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.662290+0000", 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19183647000000001 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.662359+0000", 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9095000000000002e-05 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.664155+0000", 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017960470000000001 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.806 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137160 10.8 10:1c0d5c53:::benchmark_data_smithi066_148944_object12513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.263557+0000", 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.510000203, 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.377314406, 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137160 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.807 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263557+0000", 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263557+0000", 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263559+0000", 2024-03-26T03:21:00.808 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.711e-06 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263564+0000", 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4380000000000003e-06 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263573+0000", 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.6300000000000004e-06 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263579+0000", 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3550000000000001e-06 2024-03-26T03:21:00.809 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263624+0000", 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4768000000000002e-05 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.263632+0000", 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0369999999999995e-06 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230498+0000", 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96686540200000004 2024-03-26T03:21:00.810 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230506+0000", 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9129999999999992e-06 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.327883+0000", 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.097376098 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.327983+0000", 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000100819 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.811 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.328068+0000", 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4331000000000003e-05 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332345+0000", 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0042777809999999996 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.337829+0000", 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.005483936 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.812 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339214+0000", 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001384461 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640670+0000", 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.30145598099999998 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640734+0000", 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3977000000000001e-05 2024-03-26T03:21:00.813 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640872+0000", 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000137768 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137161 10.5 10:a7b10033:::benchmark_data_smithi066_148944_object12514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.265129+0000", 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.50842797500002, 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4080903339999999, 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.814 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137161 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265129+0000", 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.815 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265129+0000", 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265131+0000", 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.883e-06 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265138+0000", 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5910000000000001e-06 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.816 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265155+0000", 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7037999999999998e-05 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265164+0000", 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0140000000000004e-06 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265216+0000", 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2194000000000002e-05 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.817 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.265227+0000", 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1316e-05 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465450+0000", 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.200222592 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465617+0000", 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016646799999999999 2024-03-26T03:21:00.818 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.465755+0000", 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013894299999999999 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470500+0000", 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047449329999999998 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470541+0000", 2024-03-26T03:21:00.819 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0467999999999999e-05 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475179+0000", 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046377140000000002 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475254+0000", 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4977999999999999e-05 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670899+0000", 2024-03-26T03:21:00.820 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.195645029 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670944+0000", 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5716000000000001e-05 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.673220+0000", 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0022754569999999998 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.821 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137169 10.8 10:102fb2c0:::benchmark_data_smithi066_148944_object12522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.277033+0000", 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.49652442600001, 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3641067549999999, 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137169 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.822 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277033+0000", 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277033+0000", 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277036+0000", 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.9730000000000002e-06 2024-03-26T03:21:00.823 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277046+0000", 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8179999999999999e-06 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277069+0000", 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2932e-05 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.824 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277080+0000", 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1245e-05 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277187+0000", 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000107459 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277206+0000", 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9026999999999999e-05 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.825 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230531+0000", 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95332462299999998 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230539+0000", 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2120000000000003e-06 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.329290+0000", 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.098750739 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.826 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.329363+0000", 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2788999999999997e-05 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.329439+0000", 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6247000000000004e-05 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332355+0000", 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002916189 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.827 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339249+0000", 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0068942120000000003 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.343995+0000", 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047458589999999998 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.640965+0000", 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29696974700000001 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.828 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641026+0000", 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1428999999999997e-05 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641140+0000", 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113255 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137172 10.5 10:ace37db4:::benchmark_data_smithi066_148944_object12525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.277612+0000", 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.49594583200002, 2024-03-26T03:21:00.829 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.398316208, 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137172 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277612+0000", 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.830 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277612+0000", 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277613+0000", 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.517e-06 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277619+0000", 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4239999999999998e-06 2024-03-26T03:21:00.831 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277633+0000", 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3534e-05 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277643+0000", 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9280000000000005e-06 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277748+0000", 2024-03-26T03:21:00.832 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000105078 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277759+0000", 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0529e-05 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.467847+0000", 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.190088931 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.467995+0000", 2024-03-26T03:21:00.833 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014706800000000001 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.468143+0000", 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000148209 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470590+0000", 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002447187 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.834 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475240+0000", 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0046503409999999997 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475258+0000", 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8178000000000001e-05 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475378+0000", 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011933400000000001 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.835 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.673285+0000", 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.19790735300000001 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.673325+0000", 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.0071999999999999e-05 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.675928+0000", 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026025250000000001 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.836 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137173 10.5 10:a3a4d821:::benchmark_data_smithi066_148944_object12526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.277668+0000", 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.49588929800001, 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4018042610000001, 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137173 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.837 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277668+0000", 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277668+0000", 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277669+0000", 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8299999999999996e-07 2024-03-26T03:21:00.838 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277673+0000", 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3100000000000002e-06 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277681+0000", 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9289999999999998e-06 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277688+0000", 2024-03-26T03:21:00.839 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6250000000000001e-06 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277784+0000", 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6658999999999997e-05 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.277791+0000", 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7959999999999998e-06 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470626+0000", 2024-03-26T03:21:00.840 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1928344019999999 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470770+0000", 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014413499999999999 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.470931+0000", 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000160917 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.841 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475263+0000", 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043327280000000001 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475317+0000", 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3677999999999997e-05 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475478+0000", 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000161201 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.842 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.843 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477904+0000", 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024256159999999998 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.676369+0000", 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.198464741 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.844 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.676435+0000", 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.601e-05 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.679472+0000", 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030377310000000001 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137178 10.5 10:a07e3bbe:::benchmark_data_smithi066_148944_object12531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.845 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.284294+0000", 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.48926324500002, 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.39794836, 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137178 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.846 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284294+0000", 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284294+0000", 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284296+0000", 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.798e-06 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.847 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284301+0000", 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.797e-06 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284309+0000", 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.7299999999999994e-06 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.284317+0000", 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4800000000000004e-06 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.848 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.303364+0000", 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.019046674999999999 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.303373+0000", 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5100000000000004e-06 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.472923+0000", 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1695500619999999 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.849 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.473065+0000", 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014189500000000001 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.473206+0000", 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014065700000000001 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475390+0000", 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002184189 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.850 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475435+0000", 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5034999999999998e-05 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477283+0000", 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018481120000000001 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477960+0000", 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00067646199999999996 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.851 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.679536+0000", 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.20157619900000001 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.679603+0000", 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7126999999999996e-05 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.682243+0000", 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026396319999999998 2024-03-26T03:21:00.852 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137190 10.8 10:195bf631:::benchmark_data_smithi066_148944_object12543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.297120+0000", 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.476437514, 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3442652210000001, 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.853 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137190 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297120+0000", 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297120+0000", 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.854 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297121+0000", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4589999999999999e-06 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297127+0000", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0009999999999998e-06 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297138+0000", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1158999999999999e-05 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.855 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297148+0000", 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5889999999999992e-06 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297206+0000", 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8326999999999999e-05 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.297213+0000", 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.8050000000000001e-06 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.856 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230563+0000", 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93334995799999998 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.230571+0000", 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1419999999999993e-06 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.330312+0000", 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.099740924 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.857 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.330386+0000", 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3553999999999994e-05 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.330464+0000", 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7879000000000004e-05 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334292+0000", 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038279049999999999 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.858 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339277+0000", 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0049859120000000003 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.344065+0000", 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047876869999999997 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641197+0000", 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29713157400000001 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.859 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641241+0000", 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4073999999999998e-05 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641385+0000", 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000144272 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137198 10.5 10:a2491f0f:::benchmark_data_smithi066_148944_object12551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:00.860 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:36.315521+0000", 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 384.45803662700001, 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.36946773, 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137198 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.861 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315521+0000", 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315521+0000", 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315523+0000", 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.829e-06 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.862 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315530+0000", 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1579999999999999e-06 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315545+0000", 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5211999999999999e-05 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315554+0000", 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3479999999999993e-06 2024-03-26T03:21:00.863 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315608+0000", 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3573999999999999e-05 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:36.315615+0000", 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6889999999999998e-06 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475505+0000", 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1598899730000001 2024-03-26T03:21:00.864 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475593+0000", 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8819000000000005e-05 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.475684+0000", 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0387e-05 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477331+0000", 2024-03-26T03:21:00.865 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001646822 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477355+0000", 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4907000000000001e-05 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.477692+0000", 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00033635099999999998 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.866 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.478062+0000", 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00036998899999999998 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.682301+0000", 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.204239217 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.682371+0000", 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0308000000000005e-05 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.867 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.684988+0000", 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026171469999999998 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137230 10.8 10:19c7df3e:::benchmark_data_smithi066_148944_object12583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.670559+0000", 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 383.10299862699998, 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97105015299999997, 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.868 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137230 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670559+0000", 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670559+0000", 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.869 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670560+0000", 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6700000000000001e-06 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670566+0000", 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.327e-06 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670576+0000", 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0475999999999999e-05 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.870 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670583+0000", 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6000000000000003e-06 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670649+0000", 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.5718999999999999e-05 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.670663+0000", 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3971999999999999e-05 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.871 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331415+0000", 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.660752322 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331490+0000", 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4967000000000002e-05 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.331579+0000", 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9543000000000006e-05 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.872 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336904+0000", 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0053242899999999998 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339335+0000", 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024316820000000001 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.344103+0000", 2024-03-26T03:21:00.873 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0047680020000000004 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641431+0000", 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29732734 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641483+0000", 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1867999999999999e-05 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.874 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641609+0000", 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000126375 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137239 10.8 10:1ce3fc92:::benchmark_data_smithi066_148944_object12592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.683080+0000", 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 383.090477448, 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95876216599999997, 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.875 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137239 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683080+0000", 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683080+0000", 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.876 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683082+0000", 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0600000000000002e-06 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683088+0000", 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3890000000000002e-06 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683104+0000", 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5648999999999999e-05 2024-03-26T03:21:00.877 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683113+0000", 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5790000000000004e-06 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683196+0000", 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3761e-05 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.683208+0000", 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1533e-05 2024-03-26T03:21:00.878 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332389+0000", 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.64918131899999998 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332464+0000", 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.4837000000000002e-05 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.332546+0000", 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1971999999999994e-05 2024-03-26T03:21:00.879 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336911+0000", 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0043649839999999997 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339365+0000", 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024541720000000001 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.346280+0000", 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069149889999999999 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.880 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641659+0000", 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29537875899999999 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641713+0000", 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3622000000000003e-05 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641842+0000", 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000129541 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.881 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137244 10.8 10:16d9ddfd:::benchmark_data_smithi066_148944_object12597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 383.08779184600002, 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95632148400000005, 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137244 2024-03-26T03:21:00.882 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.883 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685766+0000", 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3300000000000001e-07 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685769+0000", 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.066e-06 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685775+0000", 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3040000000000004e-06 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.884 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685780+0000", 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1059999999999999e-06 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685841+0000", 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1367999999999999e-05 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.685853+0000", 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1452e-05 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.885 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.333336+0000", 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.64748331100000001 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.333409+0000", 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2689999999999997e-05 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.333481+0000", 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.2821e-05 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.886 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336919+0000", 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0034379860000000001 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339397+0000", 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002477726 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.346327+0000", 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069297710000000004 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.887 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641902+0000", 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29557549100000002 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.641961+0000", 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8820000000000003e-05 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642087+0000", 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012583900000000001 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.888 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137255 10.8 10:1f88f654:::benchmark_data_smithi066_148944_object12608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.708111+0000", 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 383.06544601799999, 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93419730499999998, 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137255 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.889 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708111+0000", 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708111+0000", 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708113+0000", 2024-03-26T03:21:00.890 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.339e-06 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708118+0000", 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6910000000000002e-06 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708132+0000", 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3526000000000001e-05 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708141+0000", 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0040000000000005e-06 2024-03-26T03:21:00.891 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708195+0000", 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4114999999999999e-05 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.708205+0000", 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7079999999999992e-06 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334330+0000", 2024-03-26T03:21:00.892 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.62612499600000004 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334401+0000", 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1440999999999996e-05 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.334487+0000", 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5426999999999994e-05 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.893 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.336925+0000", 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0024385259999999999 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.339541+0000", 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0026162009999999999 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.346355+0000", 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0068138799999999996 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.894 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642132+0000", 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.29577706300000001 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642184+0000", 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1867000000000003e-05 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.642309+0000", 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000124521 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.895 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137258 10.8 10:1812d821:::benchmark_data_smithi066_148944_object12611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:37.730073+0000", 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 383.04348398299999, 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0739304270000001, 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137258 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.896 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730073+0000", 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730073+0000", 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730075+0000", 2024-03-26T03:21:00.897 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9089999999999998e-06 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730081+0000", 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.366e-06 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730092+0000", 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1022000000000001e-05 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.898 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730098+0000", 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.584e-06 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730152+0000", 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.3817999999999999e-05 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:37.730162+0000", 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5100000000000004e-06 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.899 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.335310+0000", 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.60514800899999999 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.335439+0000", 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012949700000000001 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.335583+0000", 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000143836 2024-03-26T03:21:00.900 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.337013+0000", 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0014299860000000001 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.343941+0000", 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0069278960000000002 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.347470+0000", 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0035294720000000001 2024-03-26T03:21:00.901 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.803783+0000", 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.45631306399999999 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.803853+0000", 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9276999999999994e-05 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.902 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.804004+0000", 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015118099999999999 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:137296 10.8 10:12c9bc13:::benchmark_data_smithi066_148944_object12649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:38.810787+0000", 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 381.96277019899998, 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.41837110700000002, 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.903 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 137296 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810787+0000", 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810787+0000", 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.904 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810789+0000", 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7260000000000001e-06 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810796+0000", 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6390000000000003e-06 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.905 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810808+0000", 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.259e-05 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810816+0000", 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7870000000000002e-06 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.810861+0000", 2024-03-26T03:21:00.906 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5003000000000003e-05 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.811020+0000", 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015900199999999999 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 4,5,7", 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.811155+0000", 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013474900000000001 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.907 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.813175+0000", 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020204369999999999 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.813385+0000", 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020939400000000001 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:38.814390+0000", 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00100573 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.908 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:39.228979+0000", 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.41458832400000001 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:39.229026+0000", 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7212000000000001e-05 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:39.229158+0000", 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000132514 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.909 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:138292 10.5 10:a14f9faa:::benchmark_data_smithi066_148944_object13645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:40.164725+0000", 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 380.608832087, 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.47151003000000002, 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138292 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.910 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164725+0000", 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164725+0000", 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164726+0000", 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1480000000000001e-06 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.911 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164731+0000", 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6210000000000003e-06 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164739+0000", 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0409999999999998e-06 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.164747+0000", 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9889999999999992e-06 2024-03-26T03:21:00.912 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.166114+0000", 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001366985 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op must wait for map", 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.166122+0000", 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7570000000000004e-06 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.187915+0000", 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.021792709 2024-03-26T03:21:00.913 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.188075+0000", 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000160008 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.188239+0000", 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016421 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.914 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198753+0000", 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010514122000000001 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198809+0000", 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6205999999999997e-05 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198847+0000", 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7579000000000001e-05 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.915 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.198856+0000", 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3530000000000001e-06 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.633328+0000", 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.434472203 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.633393+0000", 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4839999999999996e-05 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.916 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.636235+0000", 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028422590000000002 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:138344 10.5 10:a8c08bd4:::benchmark_data_smithi066_148944_object13697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:40.234748+0000", 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 380.53880981499998, 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.40442437799999997, 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:00.917 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138344 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234748+0000", 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.918 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234748+0000", 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234750+0000", 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2029999999999999e-06 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234757+0000", 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.742e-06 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.919 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234774+0000", 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7654000000000001e-05 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234783+0000", 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1200000000000008e-06 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.234845+0000", 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.1529e-05 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.920 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.235008+0000", 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016331399999999999 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 0,3,5,7", 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.235176+0000", 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016819599999999999 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.238849+0000", 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0036731490000000001 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.921 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.239003+0000", 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000153931 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.239070+0000", 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6387e-05 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.239322+0000", 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00025251000000000001 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.922 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.636301+0000", 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.39697872000000001 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.636371+0000", 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.9941e-05 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:40.639172+0000", 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002800982 2024-03-26T03:21:00.923 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.924 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.924 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.924 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:00.924 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:00.924 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:21:01.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 4.18 scrub starts 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 4.18 scrub ok 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: pgmap v1044: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 8.5 KiB/s rd, 68 KiB/s wr, 10 op/s; 114 KiB/s, 16 objects/s recovering 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:21:01.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 4.18 scrub starts 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 4.18 scrub ok 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: pgmap v1044: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 8.5 KiB/s rd, 68 KiB/s wr, 10 op/s; 114 KiB/s, 16 objects/s recovering 2024-03-26T03:21:01.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:21:01.611 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:01 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:21:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:21:01.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 4.18 scrub starts 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 4.18 scrub ok 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: pgmap v1044: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 8.5 KiB/s rd, 68 KiB/s wr, 10 op/s; 114 KiB/s, 16 objects/s recovering 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:21:01.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:01 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:21:02.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:21:02.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[29666]: 1.9 scrub starts 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[29666]: 1.9 scrub ok 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[29666]: 5.0 scrub starts 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[29666]: 5.0 scrub ok 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[37920]: 1.9 scrub starts 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[37920]: 1.9 scrub ok 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[37920]: 5.0 scrub starts 2024-03-26T03:21:02.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:02 smithi066 ceph-mon[37920]: 5.0 scrub ok 2024-03-26T03:21:02.665 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:02 smithi192 ceph-mon[33110]: 1.9 scrub starts 2024-03-26T03:21:02.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:02 smithi192 ceph-mon[33110]: 1.9 scrub ok 2024-03-26T03:21:02.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:02 smithi192 ceph-mon[33110]: 5.0 scrub starts 2024-03-26T03:21:02.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:02 smithi192 ceph-mon[33110]: 5.0 scrub ok 2024-03-26T03:21:03.360 INFO:teuthology.orchestra.run.smithi066.stderr:set pool 1 pgp_num to 31 2024-03-26T03:21:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:21:03.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: 1.10 scrub starts 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: 1.10 scrub ok 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: pgmap v1045: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 7.6 KiB/s rd, 53 KiB/s wr, 9 op/s; 88 KiB/s, 17 objects/s recovering 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4020237139' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: 1.10 scrub starts 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: 1.10 scrub ok 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:21:03.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: pgmap v1045: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 7.6 KiB/s rd, 53 KiB/s wr, 9 op/s; 88 KiB/s, 17 objects/s recovering 2024-03-26T03:21:03.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4020237139' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: 1.10 scrub starts 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: 1.10 scrub ok 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: pgmap v1045: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 7.6 KiB/s rd, 53 KiB/s wr, 9 op/s; 88 KiB/s, 17 objects/s recovering 2024-03-26T03:21:03.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4020237139' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-03-26T03:21:04.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:21:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:21:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:21:04.228 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-26T03:21:04.290 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:04.290 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:04.290 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:21:04.290 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:04.491 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:04.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: 1.d scrub starts 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: 1.d scrub ok 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: 1.7 scrub starts 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: 1.7 scrub ok 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4020237139' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: 4.1e scrub starts 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[29666]: 4.1e scrub ok 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: 1.d scrub starts 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: 1.d scrub ok 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: 1.7 scrub starts 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: 1.7 scrub ok 2024-03-26T03:21:04.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4020237139' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-03-26T03:21:04.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T03:21:04.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: 4.1e scrub starts 2024-03-26T03:21:04.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:04 smithi066 ceph-mon[37920]: 4.1e scrub ok 2024-03-26T03:21:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: 1.d scrub starts 2024-03-26T03:21:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: 1.d scrub ok 2024-03-26T03:21:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: 1.7 scrub starts 2024-03-26T03:21:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: 1.7 scrub ok 2024-03-26T03:21:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4020237139' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-03-26T03:21:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T03:21:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: 4.1e scrub starts 2024-03-26T03:21:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:04 smithi192 ceph-mon[33110]: 4.1e scrub ok 2024-03-26T03:21:04.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:21:05.160 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:05.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:05 smithi192 ceph-mon[33110]: 1.1c scrub starts 2024-03-26T03:21:05.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:05 smithi192 ceph-mon[33110]: 1.1c scrub ok 2024-03-26T03:21:05.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:05 smithi192 ceph-mon[33110]: 5.1 scrub starts 2024-03-26T03:21:05.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:05 smithi192 ceph-mon[33110]: 5.1 scrub ok 2024-03-26T03:21:05.583 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:05 smithi192 ceph-mon[33110]: pgmap v1047: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 6.5 KiB/s rd, 46 KiB/s wr, 7 op/s; 76 KiB/s, 14 objects/s recovering 2024-03-26T03:21:05.615 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[29666]: 1.1c scrub starts 2024-03-26T03:21:05.615 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[29666]: 1.1c scrub ok 2024-03-26T03:21:05.615 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[29666]: 5.1 scrub starts 2024-03-26T03:21:05.615 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[29666]: 5.1 scrub ok 2024-03-26T03:21:05.615 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[29666]: pgmap v1047: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 6.5 KiB/s rd, 46 KiB/s wr, 7 op/s; 76 KiB/s, 14 objects/s recovering 2024-03-26T03:21:05.616 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[37920]: 1.1c scrub starts 2024-03-26T03:21:05.616 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[37920]: 1.1c scrub ok 2024-03-26T03:21:05.616 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[37920]: 5.1 scrub starts 2024-03-26T03:21:05.616 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[37920]: 5.1 scrub ok 2024-03-26T03:21:05.616 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:05 smithi066 ceph-mon[37920]: pgmap v1047: 162 pgs: 14 peering, 148 active+clean; 808 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 6.5 KiB/s rd, 46 KiB/s wr, 7 op/s; 76 KiB/s, 14 objects/s recovering 2024-03-26T03:21:06.443 INFO:teuthology.orchestra.run.smithi066.stdout:mon.a: {} 2024-03-26T03:21:06.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:06 smithi066 ceph-mon[29666]: 2.e scrub starts 2024-03-26T03:21:06.444 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:06 smithi066 ceph-mon[29666]: 2.e scrub ok 2024-03-26T03:21:06.444 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:06 smithi066 ceph-mon[37920]: 2.e scrub starts 2024-03-26T03:21:06.444 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:06 smithi066 ceph-mon[37920]: 2.e scrub ok 2024-03-26T03:21:06.454 INFO:teuthology.orchestra.run.smithi066.stdout:mon.b: {} 2024-03-26T03:21:06.463 INFO:teuthology.orchestra.run.smithi066.stdout:mon.c: {} 2024-03-26T03:21:06.464 INFO:teuthology.orchestra.run.smithi066.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:21:06.464 INFO:teuthology.orchestra.run.smithi066.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:21:06.465 INFO:teuthology.orchestra.run.smithi066.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:21:06.541 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:06 smithi192 ceph-mon[33110]: 2.e scrub starts 2024-03-26T03:21:06.541 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:06 smithi192 ceph-mon[33110]: 2.e scrub ok 2024-03-26T03:21:07.081 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:07.082 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:07.082 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:21:07.082 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:21:07.082 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:07.191 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T03:21:07.449 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: 4.2 scrub starts 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: 4.2 scrub ok 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:21:07.571 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1588140596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[37920]: pgmap v1048: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 36 KiB/s, 12 objects/s recovering 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: 4.2 scrub starts 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: 4.2 scrub ok 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1588140596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:21:07.572 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:07 smithi066 ceph-mon[29666]: pgmap v1048: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 36 KiB/s, 12 objects/s recovering 2024-03-26T03:21:07.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: 4.2 scrub starts 2024-03-26T03:21:07.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: 4.2 scrub ok 2024-03-26T03:21:07.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:21:07.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:21:07.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:21:07.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:21:07.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1588140596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:21:07.691 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:07 smithi192 ceph-mon[33110]: pgmap v1048: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 36 KiB/s, 12 objects/s recovering 2024-03-26T03:21:07.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:21:07.962 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:08.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:08 smithi066 ceph-mon[29666]: 1.13 scrub starts 2024-03-26T03:21:08.569 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:08 smithi066 ceph-mon[29666]: 1.13 scrub ok 2024-03-26T03:21:08.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:08 smithi066 ceph-mon[37920]: 1.13 scrub starts 2024-03-26T03:21:08.570 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:08 smithi066 ceph-mon[37920]: 1.13 scrub ok 2024-03-26T03:21:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:08 smithi192 ceph-mon[33110]: 1.13 scrub starts 2024-03-26T03:21:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:08 smithi192 ceph-mon[33110]: 1.13 scrub ok 2024-03-26T03:21:09.390 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T03:21:09.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:21:09.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:21:09.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:21:09.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:21:09.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:21:09.472 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:21:09.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T03:21:09.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:21:09.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:21:09.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: pgmap v1050: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-26T03:21:09.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/90551618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:09.473 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:09 smithi192 ceph-mon[33110]: from='client.36092 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: pgmap v1050: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-26T03:21:09.711 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/90551618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[29666]: from='client.36092 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:21:09.712 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: pgmap v1050: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-26T03:21:09.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/90551618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:09.713 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:09 smithi066 ceph-mon[37920]: from='client.36092 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:09.719 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:21:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137145 10.1 10:88db8228:::benchmark_data_smithi066_148944_object12498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.237799+0000", 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 393.47972732300002, 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1472831160000001, 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137145 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237799+0000", 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237799+0000", 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237801+0000", 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.804e-06 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237808+0000", 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7880000000000001e-06 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237819+0000", 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1447000000000001e-05 2024-03-26T03:21:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.237828+0000", 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.422e-06 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.246680+0000", 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0088519389999999996 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.246688+0000", 2024-03-26T03:21:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8470000000000002e-06 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.348802+0000", 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.102113731 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.348940+0000", 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013755199999999999 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,2,7", 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.349324+0000", 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038390800000000002 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353114+0000", 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037899159999999999 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353135+0000", 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1654999999999998e-05 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353587+0000", 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000452163 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.353676+0000", 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8873000000000001e-05 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.384809+0000", 2024-03-26T03:21:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031132667999999999 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.384928+0000", 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001192 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.385082+0000", 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000154203 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137152 10.c 10:39382015:::benchmark_data_smithi066_148944_object12505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.235713+0000", 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 392.48181330599999, 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1013530659999999, 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137152 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235713+0000", 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235713+0000", 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235715+0000", 2024-03-26T03:21:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.663e-06 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235720+0000", 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.775e-06 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235730+0000", 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0030000000000001e-05 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235740+0000", 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0193999999999999e-05 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235903+0000", 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000163346 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235917+0000", 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3515e-05 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.316994+0000", 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.081077627 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for missing object", 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317110+0000", 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011518000000000001 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.336923+0000", 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019813408000000001 2024-03-26T03:21:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337066+0000", 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014332799999999999 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137181 10.c 10:30f0c64e:::benchmark_data_smithi066_148944_object12534:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.235757+0000", 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 392.481769235, 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.251399749, 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137181 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235757+0000", 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235757+0000", 2024-03-26T03:21:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235758+0000", 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2500000000000005e-07 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235761+0000", 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7429999999999998e-06 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235767+0000", 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3230000000000003e-06 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235775+0000", 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5569999999999998e-06 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235956+0000", 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018114699999999999 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235980+0000", 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4681999999999998e-05 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317365+0000", 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0813844930000001 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317499+0000", 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000133934 2024-03-26T03:21:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317899+0000", 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040043700000000002 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.328624+0000", 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010724559 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.328669+0000", 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5330999999999997e-05 2024-03-26T03:21:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334991+0000", 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063218470000000002 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339323+0000", 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043321030000000003 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.486979+0000", 2024-03-26T03:21:09.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14765573800000001 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487052+0000", 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3473000000000006e-05 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487157+0000", 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104757 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137196 10.c 10:35e1a6a7:::benchmark_data_smithi066_148944_object12549:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.235782+0000", 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 392.48174451099999, 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.252017725, 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137196 2024-03-26T03:21:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235782+0000", 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235782+0000", 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235783+0000", 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9299999999999998e-07 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235785+0000", 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.633e-06 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235791+0000", 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.235797+0000", 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8470000000000001e-06 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.236010+0000", 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000213396 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.236020+0000", 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8079999999999999e-06 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326110+0000", 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.090089571 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326258+0000", 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014866799999999999 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,6,7", 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326415+0000", 2024-03-26T03:21:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015718700000000001 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.328712+0000", 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022967980000000001 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.333008+0000", 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042961630000000004 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334998+0000", 2024-03-26T03:21:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019894999999999999 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339362+0000", 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043638030000000003 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487563+0000", 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14820169799999999 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487620+0000", 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6239000000000001e-05 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.487800+0000", 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018014699999999999 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158897 10.6 10:65a0f9c2:::benchmark_data_smithi066_148944_object34250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.525934+0000", 2024-03-26T03:21:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.19159292500001, 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.018215927, 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158897 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525934+0000", 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525934+0000", 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525935+0000", 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1170000000000001e-06 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525939+0000", 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4909999999999999e-06 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525947+0000", 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0709999999999995e-06 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525953+0000", 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.429e-06 2024-03-26T03:21:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528310+0000", 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023570219999999998 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528318+0000", 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5359999999999992e-06 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.532896+0000", 2024-03-26T03:21:09.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0045775850000001 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.533016+0000", 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120381 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.533101+0000", 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5241000000000006e-05 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537203+0000", 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041013949999999999 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540057+0000", 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002853642 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544078+0000", 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040211420000000001 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544112+0000", 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3914000000000001e-05 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544150+0000", 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7960999999999998e-05 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158898 10.6 10:6ab00462:::benchmark_data_smithi066_148944_object34251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.525959+0000", 2024-03-26T03:21:09.756 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.19156718800002, 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0183360560000001, 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158898 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525959+0000", 2024-03-26T03:21:09.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525959+0000", 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525960+0000", 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3599999999999999e-07 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525970+0000", 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0013000000000001e-05 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525975+0000", 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4229999999999996e-06 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.525980+0000", 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2109999999999997e-06 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528341+0000", 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002360441 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.528347+0000", 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6760000000000004e-06 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.535316+0000", 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0069691119999999 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.535414+0000", 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8239000000000006e-05 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.535495+0000", 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1291000000000005e-05 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540086+0000", 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045907839999999997 2024-03-26T03:21:09.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541353+0000", 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012669770000000001 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544219+0000", 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002865912 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544250+0000", 2024-03-26T03:21:09.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1639e-05 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544295+0000", 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4901999999999998e-05 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158904 10.6 10:686bb8c5:::benchmark_data_smithi066_148944_object34257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.532299+0000", 2024-03-26T03:21:09.763 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.18522796399998, 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0121124100000001, 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158904 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532299+0000", 2024-03-26T03:21:09.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532299+0000", 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532300+0000", 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.705e-06 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532306+0000", 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8189999999999997e-06 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532315+0000", 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9619999999999999e-06 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.532323+0000", 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6629999999999993e-06 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.533108+0000", 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00078506300000000002 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.533116+0000", 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0120000000000005e-06 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537242+0000", 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0041266049999999 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537334+0000", 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1242000000000005e-05 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.537403+0000", 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9321999999999998e-05 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540173+0000", 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002770242 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541359+0000", 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011858389999999999 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544354+0000", 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029949830000000001 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544386+0000", 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2308000000000003e-05 2024-03-26T03:21:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544411+0000", 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4644999999999999e-05 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158914 10.6 10:63d66798:::benchmark_data_smithi066_148944_object34267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.538143+0000", 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.17938330300001, 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0063660169999999, 2024-03-26T03:21:09.770 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158914 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538143+0000", 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538143+0000", 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538144+0000", 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538151+0000", 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8299999999999998e-06 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538159+0000", 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6079999999999998e-06 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.538165+0000", 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6309999999999997e-06 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.539129+0000", 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096368300000000001 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.539137+0000", 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0129999999999993e-06 2024-03-26T03:21:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.538712+0000", 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99957449899999995 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.538824+0000", 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112478 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.538942+0000", 2024-03-26T03:21:09.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118229 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.543952+0000", 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050095180000000001 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.543994+0000", 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2228000000000003e-05 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544446+0000", 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045226100000000003 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544475+0000", 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8370000000000001e-05 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544509+0000", 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4514000000000002e-05 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.779 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158922 10.6 10:69344914:::benchmark_data_smithi066_148944_object34275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.546660+0000", 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.17086650200002, 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.998002905, 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158922 2024-03-26T03:21:09.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546660+0000", 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546660+0000", 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546662+0000", 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.545e-06 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546667+0000", 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2669999999999997e-06 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546677+0000", 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0567999999999999e-05 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.546685+0000", 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9170000000000006e-06 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.547356+0000", 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067084800000000004 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.547364+0000", 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1990000000000006e-06 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540199+0000", 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99283470699999998 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540267+0000", 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7447999999999995e-05 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.540332+0000", 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5779999999999997e-05 2024-03-26T03:21:09.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544028+0000", 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036954750000000001 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544036+0000", 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8019999999999992e-06 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544600+0000", 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056420099999999996 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544633+0000", 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2948999999999999e-05 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544663+0000", 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0199e-05 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158945 10.6 10:6a979a68:::benchmark_data_smithi066_148944_object34298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.566506+0000", 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.15102023399999, 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97834264000000004, 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158945 2024-03-26T03:21:09.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566506+0000", 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566506+0000", 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566508+0000", 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.635e-06 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566513+0000", 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6829999999999999e-06 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566524+0000", 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0987e-05 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566529+0000", 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5589999999999996e-06 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566576+0000", 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6502999999999998e-05 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566582+0000", 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3330000000000002e-06 2024-03-26T03:21:09.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541387+0000", 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.974805221 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541452+0000", 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4992000000000004e-05 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.541573+0000", 2024-03-26T03:21:09.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120808 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544041+0000", 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002468231 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544566+0000", 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052513699999999996 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544750+0000", 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018393199999999999 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544815+0000", 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4380000000000004e-05 2024-03-26T03:21:09.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544849+0000", 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4239000000000002e-05 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158946 10.6 10:63718db7:::benchmark_data_smithi066_148944_object34299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.566744+0000", 2024-03-26T03:21:09.794 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.15078217899998, 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.979667344, 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158946 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566744+0000", 2024-03-26T03:21:09.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566744+0000", 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566745+0000", 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6599999999999996e-07 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566748+0000", 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0089999999999999e-06 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566753+0000", 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5730000000000001e-06 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566757+0000", 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9280000000000003e-06 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566786+0000", 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9879e-05 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.566791+0000", 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9520000000000002e-06 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.542682+0000", 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97589034500000005 2024-03-26T03:21:09.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.542781+0000", 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9197000000000006e-05 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.542869+0000", 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8534000000000003e-05 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544527+0000", 2024-03-26T03:21:09.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016575520000000001 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544701+0000", 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174272 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546312+0000", 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016107610000000001 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546371+0000", 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8791e-05 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546412+0000", 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0884999999999999e-05 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158956 10.6 10:6f00b1fc:::benchmark_data_smithi066_148944_object34309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.573836+0000", 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.14369040299999, 2024-03-26T03:21:09.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.977362116, 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158956 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573836+0000", 2024-03-26T03:21:09.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573836+0000", 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573838+0000", 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.719e-06 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573842+0000", 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2799999999999997e-06 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573850+0000", 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0239999999999997e-06 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573857+0000", 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8349999999999998e-06 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573910+0000", 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2846000000000002e-05 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.573918+0000", 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4120000000000001e-06 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544882+0000", 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97096349500000001 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.544970+0000", 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8603999999999994e-05 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.545040+0000", 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9832999999999994e-05 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550896+0000", 2024-03-26T03:21:09.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058562409999999999 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550935+0000", 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8210000000000002e-05 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.551073+0000", 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013858800000000001 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.551156+0000", 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2978000000000004e-05 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.551198+0000", 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2051000000000001e-05 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:158959 10.6 10:6f864bb5:::benchmark_data_smithi066_148944_object34312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:00.577141+0000", 2024-03-26T03:21:09.808 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 369.14038505899998, 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97805649400000005, 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 158959 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577141+0000", 2024-03-26T03:21:09.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577141+0000", 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577143+0000", 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1930000000000001e-06 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577147+0000", 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4830000000000001e-06 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577155+0000", 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3389999999999993e-06 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577163+0000", 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1210000000000001e-06 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577233+0000", 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0706999999999999e-05 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:00.577242+0000", 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4109999999999996e-06 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546459+0000", 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96921719200000001 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546574+0000", 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114706 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.546686+0000", 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112698 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550947+0000", 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042605810000000003 2024-03-26T03:21:09.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.550984+0000", 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6906999999999997e-05 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.555122+0000", 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041379909999999997 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.555154+0000", 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1736999999999998e-05 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.555198+0000", 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4428000000000002e-05 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:199030 10.4 10:291ad30d:::benchmark_data_smithi066_148944_object74383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-26T03:21:09.815 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:44.926131+0000", 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 324.79139576699998, 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97667217900000003, 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199030 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926131+0000", 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926131+0000", 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926132+0000", 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8400000000000003e-07 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926135+0000", 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5470000000000001e-06 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926145+0000", 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6199999999999994e-06 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926151+0000", 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.981e-06 2024-03-26T03:21:09.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926486+0000", 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000334965 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.926494+0000", 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5869999999999993e-06 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.895660+0000", 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96916596799999999 2024-03-26T03:21:09.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.895793+0000", 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013325199999999999 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.895932+0000", 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013806499999999999 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900704+0000", 2024-03-26T03:21:09.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047728720000000001 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902598+0000", 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018933330000000001 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902726+0000", 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012818199999999999 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902762+0000", 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6371999999999997e-05 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.902803+0000", 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0651000000000001e-05 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:199032 10.4 10:255721a4:::benchmark_data_smithi066_148944_object74385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:44.930307+0000", 2024-03-26T03:21:09.822 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 324.78721961999997, 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97661484300000001, 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199032 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930307+0000", 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930307+0000", 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930308+0000", 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3200000000000004e-07 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930311+0000", 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4879999999999999e-06 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930318+0000", 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8600000000000004e-06 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930324+0000", 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6840000000000001e-06 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930409+0000", 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4892e-05 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.930417+0000", 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8540000000000003e-06 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.898572+0000", 2024-03-26T03:21:09.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96815444900000003 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.898646+0000", 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4466000000000002e-05 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.898721+0000", 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4299000000000001e-05 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.904286+0000", 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00556581 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906712+0000", 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024256460000000001 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906821+0000", 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108491 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906868+0000", 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589000000000001e-05 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906922+0000", 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3582999999999998e-05 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.829 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:199042 10.4 10:2c3d90fc:::benchmark_data_smithi066_148944_object74395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:44.940858+0000", 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 324.77666843100002, 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.96623120299999998, 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 199042 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940858+0000", 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940858+0000", 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940859+0000", 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3799999999999999e-06 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940865+0000", 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0810000000000001e-06 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940874+0000", 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7820000000000005e-06 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940882+0000", 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6059999999999997e-06 2024-03-26T03:21:09.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940941+0000", 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9290999999999999e-05 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:44.940949+0000", 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9799999999999998e-06 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900739+0000", 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95978980899999999 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900823+0000", 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3926999999999998e-05 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:21:09.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.900898+0000", 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5498000000000001e-05 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.904293+0000", 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033949520000000001 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906762+0000", 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024681970000000001 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:09.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.906989+0000", 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022715999999999999 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.907045+0000", 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6215999999999998e-05 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:45.907089+0000", 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4323999999999998e-05 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.836 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:83024 11.e 11.d2d4311e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.550240+0000", 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 194.16728646199999, 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0111298360000001, 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:09.837 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83024 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550240+0000", 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550240+0000", 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550241+0000", 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.384e-06 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550288+0000", 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616999999999998e-05 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550299+0000", 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0472000000000001e-05 2024-03-26T03:21:09.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.550306+0000", 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3560000000000004e-06 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.561812+0000", 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011506003000000001 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.561820+0000", 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0730000000000005e-06 2024-03-26T03:21:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561358+0000", 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99953816900000003 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561370+0000", 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1762e-05 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.34346.0:4038 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94613743831168 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.551335+0000", 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 194.16619199900001, 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.023797048, 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.34346", 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/2004587607", 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4038 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.842 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551335+0000", 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551335+0000", 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551336+0000", 2024-03-26T03:21:09.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2619999999999999e-06 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551340+0000", 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9380000000000002e-06 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551347+0000", 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.661e-06 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.551353+0000", 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8479999999999997e-06 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.557762+0000", 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0064087220000000004 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.557770+0000", 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9540000000000004e-06 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.566358+0000", 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.00858767 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for missing object", 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.566514+0000", 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000156711 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.570613+0000", 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040989889999999999 2024-03-26T03:21:09.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for rw locks", 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.570718+0000", 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104448 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.571395+0000", 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067711099999999999 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.571649+0000", 2024-03-26T03:21:09.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025428899999999999 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.575132+0000", 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034824449999999998 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:83048 11.e 11.35baa8e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.564394+0000", 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 194.15313267400001, 2024-03-26T03:21:09.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99701060100000005, 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83048 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564394+0000", 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564394+0000", 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564396+0000", 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564438+0000", 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1622000000000002e-05 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564449+0000", 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1365e-05 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564457+0000", 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564513+0000", 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5358000000000001e-05 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.564521+0000", 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0360000000000006e-06 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561397+0000", 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99687634400000003 2024-03-26T03:21:09.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561404+0000", 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3119999999999996e-06 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:83051 11.e 11.d761e45e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:17:55.569213+0000", 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 194.14831387999999, 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99222091400000001, 2024-03-26T03:21:09.853 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83051 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569213+0000", 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569213+0000", 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569214+0000", 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.813e-06 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569274+0000", 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9602000000000003e-05 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569284+0000", 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0396e-05 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569290+0000", 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569333+0000", 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2273e-05 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:55.569338+0000", 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4e-06 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561426+0000", 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99208782399999995 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:17:56.561434+0000", 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5660000000000001e-06 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.857 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.858 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.858 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:09.858 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:09.858 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:10.081 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-26T03:21:10.334 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:10.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:21:10.706 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[37920]: from='client.36092 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[37920]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[29666]: from='client.36092 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[29666]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T03:21:10.707 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:10.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:21:10 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:21:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:10 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:21:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:10 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:21:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:10 smithi192 ceph-mon[33110]: from='client.36092 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:21:10.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:10 smithi192 ceph-mon[33110]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T03:21:10.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:10.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:21:11.040 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:11.487 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 4.16 scrub starts 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 4.16 scrub ok 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:21:11.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:11 smithi192 ceph-mon[33110]: pgmap v1052: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 4.16 scrub starts 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 4.16 scrub ok 2024-03-26T03:21:11.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[29666]: pgmap v1052: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 4.16 scrub starts 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 4.16 scrub ok 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:21:11.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:11 smithi066 ceph-mon[37920]: pgmap v1052: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2024-03-26T03:21:12.394 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-26T03:21:12.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: 4.1c scrub starts 2024-03-26T03:21:12.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: 4.1c scrub ok 2024-03-26T03:21:12.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: 1.14 scrub starts 2024-03-26T03:21:12.437 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: 1.14 scrub ok 2024-03-26T03:21:12.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: 6.c scrub starts 2024-03-26T03:21:12.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: 6.c scrub ok 2024-03-26T03:21:12.438 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:12 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/705820813' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:12.670 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:12.670 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:21:12.670 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:12.718 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: 4.1c scrub starts 2024-03-26T03:21:12.718 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: 4.1c scrub ok 2024-03-26T03:21:12.718 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: 1.14 scrub starts 2024-03-26T03:21:12.718 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: 1.14 scrub ok 2024-03-26T03:21:12.718 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: 6.c scrub starts 2024-03-26T03:21:12.718 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: 6.c scrub ok 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/705820813' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: 4.1c scrub starts 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: 4.1c scrub ok 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: 1.14 scrub starts 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: 1.14 scrub ok 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: 6.c scrub starts 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: 6.c scrub ok 2024-03-26T03:21:12.719 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:12 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/705820813' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:21:13.133 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T03:21:13.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:21:13.392 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:13.558 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:13.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/705820813' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:21:13.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-mon[29666]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T03:21:13.658 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-mon[29666]: pgmap v1054: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T03:21:13.658 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/705820813' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:21:13.658 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-mon[37920]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T03:21:13.659 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-mon[37920]: pgmap v1054: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T03:21:13.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:13 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/705820813' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:21:13.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:13 smithi192 ceph-mon[33110]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T03:21:13.680 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:13 smithi192 ceph-mon[33110]: pgmap v1054: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T03:21:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:21:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:21:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:21:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: 1.17 scrub starts 2024-03-26T03:21:14.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: 1.17 scrub ok 2024-03-26T03:21:14.760 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: 5.1a deep-scrub starts 2024-03-26T03:21:14.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: 5.1a deep-scrub ok 2024-03-26T03:21:14.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: 3.1 scrub starts 2024-03-26T03:21:14.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: 3.1 scrub ok 2024-03-26T03:21:14.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:21:14.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:21:14.761 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:14 smithi192 ceph-mon[33110]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T03:21:14.821 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: 1.17 scrub starts 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: 1.17 scrub ok 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: 5.1a deep-scrub starts 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: 5.1a deep-scrub ok 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: 3.1 scrub starts 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: 3.1 scrub ok 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[29666]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: 1.17 scrub starts 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: 1.17 scrub ok 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: 5.1a deep-scrub starts 2024-03-26T03:21:14.822 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: 5.1a deep-scrub ok 2024-03-26T03:21:14.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: 3.1 scrub starts 2024-03-26T03:21:14.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: 3.1 scrub ok 2024-03-26T03:21:14.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:21:14.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:21:14.823 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:14 smithi066 ceph-mon[37920]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T03:21:15.153 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:21:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:21:15.153 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:15.461 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T03:21:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:15 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:21:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:15 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:21:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:15 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:21:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:15 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:21:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:15 smithi192 ceph-mon[33110]: pgmap v1056: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:15.852 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:21:15.852 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:21:15.852 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:21:15.852 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:21:15.852 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[29666]: pgmap v1056: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:15.853 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:21:15.853 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:21:15.853 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:21:15.853 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:21:15.853 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:15 smithi066 ceph-mon[37920]: pgmap v1056: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:15.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:21:16.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-26T03:21:16.220 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:16.440 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:16.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:21:16.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:21:16.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2558315294' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:16.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: from='client.36104 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:16.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:21:16.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:21:16.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: from='client.36104 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:21:16.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T03:21:16.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: 5.14 scrub starts 2024-03-26T03:21:16.630 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:16 smithi192 ceph-mon[33110]: 5.14 scrub ok 2024-03-26T03:21:16.700 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:21:16.700 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:21:16.700 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2558315294' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:16.700 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: from='client.36104 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:16.700 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:21:16.700 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: from='client.36104 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: 5.14 scrub starts 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[29666]: 5.14 scrub ok 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2558315294' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: from='client.36104 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: from='client.36104 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:21:16.701 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T03:21:16.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: 5.14 scrub starts 2024-03-26T03:21:16.702 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:16 smithi066 ceph-mon[37920]: 5.14 scrub ok 2024-03-26T03:21:17.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: 2.0 scrub starts 2024-03-26T03:21:17.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: 2.0 scrub ok 2024-03-26T03:21:17.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: 5.b deep-scrub starts 2024-03-26T03:21:17.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: 5.b deep-scrub ok 2024-03-26T03:21:17.579 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: 4.e deep-scrub starts 2024-03-26T03:21:17.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: 4.e deep-scrub ok 2024-03-26T03:21:17.580 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:17 smithi192 ceph-mon[33110]: pgmap v1058: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:21:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: 2.0 scrub starts 2024-03-26T03:21:17.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: 2.0 scrub ok 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: 5.b deep-scrub starts 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: 5.b deep-scrub ok 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: 4.e deep-scrub starts 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: 4.e deep-scrub ok 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[29666]: pgmap v1058: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: 2.0 scrub starts 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: 2.0 scrub ok 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: 5.b deep-scrub starts 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: 5.b deep-scrub ok 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: 4.e deep-scrub starts 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: 4.e deep-scrub ok 2024-03-26T03:21:17.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:17 smithi066 ceph-mon[37920]: pgmap v1058: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15360.0:283807 9.f 9:f34c1168:::benchmark_data_smithi066_125102_object137506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-26T03:21:17.820 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:12:04.997387+0000", 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 552.820677972, 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16383914699999999, 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15360", 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/3209858867", 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 283807 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.997387+0000", 2024-03-26T03:21:17.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.997387+0000", 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.997388+0000", 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1689999999999999e-06 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.997393+0000", 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9969999999999998e-06 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.997400+0000", 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6359999999999994e-06 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.997407+0000", 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8360000000000003e-06 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.998726+0000", 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013188 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.998841+0000", 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114795 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:04.998957+0000", 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011647 2024-03-26T03:21:17.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.001685+0000", 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027277260000000002 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.161078+0000", 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.15939267700000001 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.161133+0000", 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5213000000000001e-05 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:12:05.161226+0000", 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2828e-05 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137219 10.9 10:9a86151a:::benchmark_data_smithi066_148944_object12572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:17.826 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.476796+0000", 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 400.341268152, 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.325674821, 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137219 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476796+0000", 2024-03-26T03:21:17.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476796+0000", 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476799+0000", 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3180000000000001e-06 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476805+0000", 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.852e-06 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476819+0000", 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4219999999999999e-05 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476826+0000", 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4070000000000004e-06 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476904+0000", 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7310000000000004e-05 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.476918+0000", 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4739000000000001e-05 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317836+0000", 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.84091820900000003 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.317954+0000", 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011777699999999999 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.318332+0000", 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037741599999999999 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334735+0000", 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016403395000000001 2024-03-26T03:21:17.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334777+0000", 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1491e-05 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339392+0000", 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046152709999999998 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.802218+0000", 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46282611299999998 2024-03-26T03:21:17.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.802337+0000", 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118745 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.802471+0000", 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013455800000000001 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.834 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137247 10.9 10:9cdb27b9:::benchmark_data_smithi066_148944_object12600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:17.834 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.688383+0000", 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 400.12968137399997, 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.218126987, 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137247 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688383+0000", 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688383+0000", 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688386+0000", 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5959999999999998e-06 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688397+0000", 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1019000000000001e-05 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688411+0000", 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4069e-05 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688419+0000", 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3510000000000002e-06 2024-03-26T03:21:17.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688494+0000", 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4760999999999998e-05 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.688503+0000", 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2070000000000001e-06 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334807+0000", 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64630341599999996 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.334935+0000", 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000128143 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335075+0000", 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013982900000000001 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339098+0000", 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00402339 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339379+0000", 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028078400000000001 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339400+0000", 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0862000000000001e-05 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906328+0000", 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56692821800000004 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906392+0000", 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671999999999993e-05 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906510+0000", 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011767 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.841 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137254 10.9 10:97a448bb:::benchmark_data_smithi066_148944_object12607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:37.706015+0000", 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 400.112049877, 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.200679982, 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.842 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137254 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706015+0000", 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706015+0000", 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706018+0000", 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3239999999999999e-06 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706024+0000", 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9549999999999997e-06 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706038+0000", 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3929e-05 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706054+0000", 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6469000000000001e-05 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706107+0000", 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2575999999999999e-05 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.706115+0000", 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1130000000000001e-06 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337003+0000", 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63088797600000002 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337124+0000", 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121208 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337251+0000", 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012716299999999999 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339404+0000", 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021529230000000002 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339851+0000", 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044708999999999998 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.339886+0000", 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4536000000000003e-05 2024-03-26T03:21:17.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906549+0000", 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56666309800000003 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906597+0000", 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7576e-05 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906695+0000", 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8046000000000003e-05 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137266 10.9 10:9c7c1621:::benchmark_data_smithi066_148944_object12619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.488103+0000", 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 399.32996152700002, 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55283626699999999, 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137266 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488103+0000", 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488103+0000", 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488106+0000", 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.134e-06 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488116+0000", 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4360000000000008e-06 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488134+0000", 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8873999999999999e-05 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488145+0000", 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0258000000000001e-05 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488213+0000", 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8443999999999997e-05 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488336+0000", 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012250999999999999 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488475+0000", 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013899800000000001 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.490550+0000", 2024-03-26T03:21:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020753260000000002 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.490837+0000", 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028720200000000002 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.491942+0000", 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001105071 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040760+0000", 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54881772900000003 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040823+0000", 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3122999999999999e-05 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040939+0000", 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116162 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.855 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137271 10.9 10:96c90966:::benchmark_data_smithi066_148944_object12624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.545904+0000", 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 399.27216077399999, 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50180483099999995, 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137271 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.856 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.545904+0000", 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.545904+0000", 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.545906+0000", 2024-03-26T03:21:17.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.545917+0000", 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.113e-05 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.545929+0000", 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2172e-05 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.545938+0000", 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9420000000000001e-06 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.546017+0000", 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9039e-05 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.546163+0000", 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014620200000000001 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.546305+0000", 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000142101 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.548250+0000", 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001944332 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.548318+0000", 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8428000000000003e-05 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.548556+0000", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023743399999999999 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047487+0000", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49893130800000002 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047584+0000", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7510000000000007e-05 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047709+0000", 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000124284 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137288 10.9 10:9d93d421:::benchmark_data_smithi066_148944_object12641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.655725+0000", 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 399.16233954, 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39220371399999998, 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.862 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137288 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655725+0000", 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655725+0000", 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655727+0000", 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5799999999999999e-06 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655732+0000", 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7230000000000001e-06 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655744+0000", 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1486e-05 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655750+0000", 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6320000000000002e-06 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655814+0000", 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3163000000000005e-05 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.655987+0000", 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017385600000000001 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656182+0000", 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000194897 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.658236+0000", 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002053862 2024-03-26T03:21:17.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.658532+0000", 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029566199999999998 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.659356+0000", 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00082368399999999998 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047747+0000", 2024-03-26T03:21:17.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.388391771 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047827+0000", 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9646000000000003e-05 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047929+0000", 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101752 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137299 10.9 10:92b4110a:::benchmark_data_smithi066_148944_object12652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.819058+0000", 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 398.99900685400002, 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23162239900000001, 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137299 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819058+0000", 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819058+0000", 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819060+0000", 2024-03-26T03:21:17.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819067+0000", 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6989999999999997e-06 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819081+0000", 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4367e-05 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819090+0000", 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7839999999999992e-06 2024-03-26T03:21:17.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819157+0000", 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7228000000000001e-05 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819307+0000", 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015040799999999999 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.819461+0000", 2024-03-26T03:21:17.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015413699999999999 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.821468+0000", 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020061419999999998 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.821511+0000", 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3492000000000003e-05 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.821812+0000", 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000301231 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050367+0000", 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22855493499999999 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050513+0000", 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014557400000000001 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050680+0000", 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001673 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:138306 10.9 10:9a927fdc:::benchmark_data_smithi066_148944_object13659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.187736+0000", 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 397.63032891400002, 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.278698689, 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.875 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 138306 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.187736+0000", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.187736+0000", 2024-03-26T03:21:17.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.187738+0000", 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0269999999999998e-06 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.187744+0000", 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.613e-06 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.187758+0000", 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3288000000000001e-05 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.187765+0000", 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5229999999999998e-06 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188055+0000", 2024-03-26T03:21:17.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028987499999999999 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188189+0000", 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013396099999999999 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188322+0000", 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013299300000000001 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.198078+0000", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0097557609999999999 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.198134+0000", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6266e-05 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.198939+0000", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080463400000000003 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.466260+0000", 2024-03-26T03:21:17.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.267321425 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.466322+0000", 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2298999999999994e-05 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.466434+0000", 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112024 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:138309 10.9 10:96d4bdf4:::benchmark_data_smithi066_148944_object13662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.189362+0000", 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 397.62870273800002, 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27724673100000002, 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 138309 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.882 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.189362+0000", 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.189362+0000", 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.189363+0000", 2024-03-26T03:21:17.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4959999999999999e-06 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.189369+0000", 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6840000000000001e-06 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.189379+0000", 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0406e-05 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.189388+0000", 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1669999999999999e-06 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.192561+0000", 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031731039999999999 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.192695+0000", 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013414500000000001 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.192827+0000", 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132597 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.198140+0000", 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0053128610000000003 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.199156+0000", 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010155920000000001 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.205196+0000", 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006040012 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.466471+0000", 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26127558699999998 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.466516+0000", 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.422e-05 2024-03-26T03:21:17.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.466609+0000", 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2860000000000002e-05 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:138339 10.9 10:9c153ed0:::benchmark_data_smithi066_148944_object13692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.227129+0000", 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 397.59093516600001, 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24221020100000001, 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.888 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.889 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 138339 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.227129+0000", 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.227129+0000", 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.227132+0000", 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.216e-06 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.227138+0000", 2024-03-26T03:21:17.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.764e-06 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.227151+0000", 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2585000000000001e-05 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.227160+0000", 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4010000000000003e-06 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.228794+0000", 2024-03-26T03:21:17.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016335309999999999 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.228940+0000", 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000146413 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.229089+0000", 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000148973 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.231462+0000", 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00237309 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.232912+0000", 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014497399999999999 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236043+0000", 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031304000000000002 2024-03-26T03:21:17.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.469163+0000", 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23312000299999999 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.469223+0000", 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0782000000000002e-05 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.469340+0000", 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116303 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:138346 10.9 10:9b9f6981:::benchmark_data_smithi066_148944_object13699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.236521+0000", 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 397.581543872, 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.232999971, 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 138346 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236521+0000", 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236521+0000", 2024-03-26T03:21:17.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236523+0000", 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.672e-06 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236530+0000", 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5560000000000004e-06 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236546+0000", 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5855999999999999e-05 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236554+0000", 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7159999999999992e-06 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.238014+0000", 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014599050000000001 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.238152+0000", 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137916 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.238288+0000", 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136126 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.240552+0000", 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022631079999999998 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.246148+0000", 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055963949999999997 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.247714+0000", 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001566006 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.469378+0000", 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22166435200000001 2024-03-26T03:21:17.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.469425+0000", 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6794e-05 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.469521+0000", 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5569000000000006e-05 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:157682 10.9 10:99c77610:::benchmark_data_smithi066_148944_object33035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:59.265342+0000", 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 378.55272247200003, 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21182130499999999, 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157682 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265342+0000", 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265342+0000", 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265344+0000", 2024-03-26T03:21:17.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1689999999999999e-06 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265351+0000", 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9879999999999998e-06 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265362+0000", 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0822000000000001e-05 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265371+0000", 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2879999999999998e-06 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265454+0000", 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2647999999999999e-05 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265608+0000", 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000154454 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.265744+0000", 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000135833 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.267575+0000", 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00183034 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.268145+0000", 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00057009499999999996 2024-03-26T03:21:17.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.270017+0000", 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001872662 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.476913+0000", 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20689576800000001 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.477042+0000", 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012930999999999999 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.477163+0000", 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120928 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:157687 10.9 10:9078db2f:::benchmark_data_smithi066_148944_object33040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:59.271210+0000", 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 378.54685468399998, 2024-03-26T03:21:17.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22034378099999999, 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157687 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271210+0000", 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271210+0000", 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271212+0000", 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.367e-06 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271221+0000", 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6319999999999997e-06 2024-03-26T03:21:17.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271235+0000", 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3868e-05 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271243+0000", 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8679999999999999e-06 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271310+0000", 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7422000000000006e-05 2024-03-26T03:21:17.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271457+0000", 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000146804 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4,7", 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.271555+0000", 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8348000000000001e-05 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.273415+0000", 2024-03-26T03:21:17.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018595809999999999 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.273750+0000", 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033521899999999999 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.273818+0000", 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7883e-05 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.491383+0000", 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.217565446 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.491442+0000", 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9030000000000002e-05 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:59.491554+0000", 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111313 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.915 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:207101 10.9 10:96cbf463:::benchmark_data_smithi066_148944_object82454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:54.027288+0000", 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 323.79077682000002, 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.201967427, 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.916 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207101 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.027288+0000", 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.027288+0000", 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.027290+0000", 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9640000000000002e-06 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.027304+0000", 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4262e-05 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.027314+0000", 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9070000000000002e-06 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.027323+0000", 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2909999999999996e-06 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.033055+0000", 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057316229999999999 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.033064+0000", 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0869999999999996e-06 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.218465+0000", 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18540125599999999 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.218579+0000", 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114073 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:21:17.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.218705+0000", 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012608399999999999 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.222863+0000", 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004157595 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.225491+0000", 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026279290000000002 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229145+0000", 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036545140000000002 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229209+0000", 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3851999999999998e-05 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229255+0000", 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5989999999999998e-05 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.922 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:207102 10.9 10:9497ed13:::benchmark_data_smithi066_148944_object82455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:54.028273+0000", 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 323.78979123099998, 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.201193657, 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.923 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207102 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.028273+0000", 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.028273+0000", 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.028275+0000", 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4479999999999999e-06 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.028280+0000", 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0000000000000004e-06 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.028346+0000", 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6110000000000002e-05 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.028392+0000", 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5775000000000001e-05 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.033539+0000", 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0051476220000000001 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.033547+0000", 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6149999999999999e-06 2024-03-26T03:21:17.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.220926+0000", 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18737912700000001 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.221058+0000", 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013164700000000001 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.221172+0000", 2024-03-26T03:21:17.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114045 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.225508+0000", 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004336187 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.225546+0000", 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229374+0000", 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003828335 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229418+0000", 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4109000000000001e-05 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229467+0000", 2024-03-26T03:21:17.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8656000000000003e-05 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:207113 10.9 10:93597a42:::benchmark_data_smithi066_148944_object82466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:54.047741+0000", 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 323.77032389999999, 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.186658245, 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207113 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047741+0000", 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047741+0000", 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047743+0000", 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047748+0000", 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0610000000000003e-06 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047759+0000", 2024-03-26T03:21:17.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1686e-05 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047768+0000", 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3000000000000002e-06 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047831+0000", 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3421000000000001e-05 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.047841+0000", 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0185e-05 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.222890+0000", 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17504883399999999 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.223008+0000", 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011753599999999999 2024-03-26T03:21:17.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.223115+0000", 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107308 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.225557+0000", 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002442282 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229297+0000", 2024-03-26T03:21:17.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003739657 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.234271+0000", 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049741469999999999 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.234353+0000", 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1991999999999997e-05 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.234399+0000", 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5806000000000002e-05 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:207115 10.9 10:98c9743b:::benchmark_data_smithi066_148944_object82468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:54.048033+0000", 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 323.77003199799998, 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.186522566, 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207115 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048033+0000", 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048033+0000", 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048034+0000", 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3e-06 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048039+0000", 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8919999999999999e-06 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048051+0000", 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2384999999999999e-05 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048059+0000", 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9629999999999998e-06 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048109+0000", 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9656e-05 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:17.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.048117+0000", 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9610000000000005e-06 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.224478+0000", 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17636162699999999 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.224541+0000", 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2769999999999997e-05 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:21:17.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.224606+0000", 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4716999999999997e-05 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229050+0000", 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044440419999999996 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.229335+0000", 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028555399999999998 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.234446+0000", 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0051103019999999997 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.234520+0000", 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4734000000000006e-05 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:54.234555+0000", 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4663000000000003e-05 2024-03-26T03:21:17.943 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:145408 11.5 11:ad6776bf:::benchmark_data_smithi066_175179_object13510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:50.551249+0000", 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 147.266815826, 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18586387300000001, 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:21:17.944 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 145408 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551249+0000", 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551249+0000", 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551250+0000", 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7430000000000001e-06 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551255+0000", 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9119999999999997e-06 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551267+0000", 2024-03-26T03:21:17.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1800000000000001e-05 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551275+0000", 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0050000000000004e-06 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551331+0000", 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5744e-05 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551457+0000", 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012588899999999999 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.551579+0000", 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000122452 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553327+0000", 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017472379999999999 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553553+0000", 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022609799999999999 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.736935+0000", 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18338231599999999 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.737039+0000", 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103717 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.737113+0000", 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3959000000000001e-05 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15978.0:145411 11.5 11:a3882c70:::benchmark_data_smithi066_175179_object13513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:18:50.553465+0000", 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 147.26459910099999, 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18376329599999999, 2024-03-26T03:21:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15978", 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 145411 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553465+0000", 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553465+0000", 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553467+0000", 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6360000000000001e-06 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553473+0000", 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5860000000000004e-06 2024-03-26T03:21:17.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553484+0000", 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1439e-05 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553492+0000", 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2090000000000005e-06 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553585+0000", 2024-03-26T03:21:17.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2949999999999996e-05 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553698+0000", 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112471 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,6", 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.553805+0000", 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010739999999999999 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.555524+0000", 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017184699999999999 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.558707+0000", 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003183127 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.737156+0000", 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.178449575 2024-03-26T03:21:17.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.737189+0000", 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2796999999999998e-05 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:18:50.737229+0000", 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635999999999998e-05 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:17.957 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:17.957 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:18.417 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-26T03:21:18.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:18 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:21:18.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:18 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:21:18.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:18 smithi192 ceph-mon[33110]: 6.8 scrub starts 2024-03-26T03:21:18.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:18 smithi192 ceph-mon[33110]: 6.8 scrub ok 2024-03-26T03:21:18.703 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:18 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/50521933' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:18.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:21:18.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:21:18.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[29666]: 6.8 scrub starts 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[29666]: 6.8 scrub ok 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/50521933' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[37920]: 6.8 scrub starts 2024-03-26T03:21:18.861 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[37920]: 6.8 scrub ok 2024-03-26T03:21:18.862 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:18 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/50521933' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:21:18.972 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:19.176 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T03:21:19.424 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:21:19.436 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:19.713 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:21:19.713 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:21:19.713 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:21:19.713 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/50521933' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:21:19.713 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T03:21:19.713 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: pgmap v1060: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: 6.f deep-scrub starts 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[29666]: 6.f deep-scrub ok 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/50521933' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: pgmap v1060: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:19.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: 6.f deep-scrub starts 2024-03-26T03:21:19.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:19 smithi066 ceph-mon[37920]: 6.f deep-scrub ok 2024-03-26T03:21:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:21:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/50521933' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: pgmap v1060: 162 pgs: 162 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: 6.f deep-scrub starts 2024-03-26T03:21:19.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:19 smithi192 ceph-mon[33110]: 6.f deep-scrub ok 2024-03-26T03:21:20.279 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:21:20 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:21:20.538 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:20.538 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:20.538 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:21:20.538 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:20.539 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:20 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:21:20.539 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:20 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:21:20.539 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:20 smithi192 ceph-mon[33110]: 6.b scrub starts 2024-03-26T03:21:20.539 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:20 smithi192 ceph-mon[33110]: 6.b scrub ok 2024-03-26T03:21:20.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:21:20.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:21:20.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[37920]: 6.b scrub starts 2024-03-26T03:21:20.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[37920]: 6.b scrub ok 2024-03-26T03:21:20.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:21:20.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:21:20.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[29666]: 6.b scrub starts 2024-03-26T03:21:20.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:20 smithi066 ceph-mon[29666]: 6.b scrub ok 2024-03-26T03:21:21.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:21:21.429 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T03:21:21.567 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:21.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:21:21.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: pgmap v1061: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 722 B/s rd, 0 op/s 2024-03-26T03:21:21.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:21 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1515189551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:21.787 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:21:21.787 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: pgmap v1061: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 722 B/s rd, 0 op/s 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1515189551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:21.788 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:21:21.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:21:21.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:21:21.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:21:21.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: pgmap v1061: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 722 B/s rd, 0 op/s 2024-03-26T03:21:21.789 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:21 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1515189551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:22.187 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-26T03:21:22.449 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: 1.b scrub starts 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: 1.b scrub ok 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: 4.1b scrub starts 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: 4.1b scrub ok 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1515189551' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:21:22.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:22 smithi192 ceph-mon[33110]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T03:21:22.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: 1.b scrub starts 2024-03-26T03:21:22.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: 1.b scrub ok 2024-03-26T03:21:22.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:21:22.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: 4.1b scrub starts 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: 4.1b scrub ok 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1515189551' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[29666]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: 1.b scrub starts 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: 1.b scrub ok 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: 4.1b scrub starts 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: 4.1b scrub ok 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1515189551' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:21:22.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:22 smithi066 ceph-mon[37920]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T03:21:23.159 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:23.159 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:23.160 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:21:23.160 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:21:23.160 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:23.698 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:21:23.698 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[29666]: pgmap v1063: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[29666]: 3.2 scrub starts 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[29666]: 3.2 scrub ok 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[37920]: pgmap v1063: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[37920]: 3.2 scrub starts 2024-03-26T03:21:23.699 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-mon[37920]: 3.2 scrub ok 2024-03-26T03:21:23.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:23 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:21:23.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:23 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:21:23.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:23 smithi192 ceph-mon[33110]: pgmap v1063: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:21:23.717 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:23 smithi192 ceph-mon[33110]: 3.2 scrub starts 2024-03-26T03:21:23.718 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:23 smithi192 ceph-mon[33110]: 3.2 scrub ok 2024-03-26T03:21:23.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:21:24.035 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:24.044 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:21:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:21:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:21:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:24 smithi192 ceph-mon[33110]: 2.15 scrub starts 2024-03-26T03:21:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:24 smithi192 ceph-mon[33110]: 2.15 scrub ok 2024-03-26T03:21:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:24 smithi192 ceph-mon[33110]: 6.14 scrub starts 2024-03-26T03:21:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:24 smithi192 ceph-mon[33110]: 6.14 scrub ok 2024-03-26T03:21:24.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[29666]: 2.15 scrub starts 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[29666]: 2.15 scrub ok 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[29666]: 6.14 scrub starts 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[29666]: 6.14 scrub ok 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[37920]: 2.15 scrub starts 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[37920]: 2.15 scrub ok 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[37920]: 6.14 scrub starts 2024-03-26T03:21:24.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:24 smithi066 ceph-mon[37920]: 6.14 scrub ok 2024-03-26T03:21:25.466 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159163 10.0 10:07efa237:::benchmark_data_smithi066_148944_object34516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.568736+0000", 2024-03-26T03:21:25.638 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.06636492600001, 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.024685979, 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159163 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568736+0000", 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568736+0000", 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568737+0000", 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.26e-06 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568742+0000", 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-03-26T03:21:25.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568756+0000", 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4408e-05 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568765+0000", 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4689999999999997e-06 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568828+0000", 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2966000000000007e-05 2024-03-26T03:21:25.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.568843+0000", 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5216e-05 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571792+0000", 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.002948945 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571804+0000", 2024-03-26T03:21:25.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.221e-05 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.586864+0000", 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.015060139 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.587027+0000", 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000162974 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.587388+0000", 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000361245 2024-03-26T03:21:25.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593072+0000", 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056834980000000004 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593094+0000", 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1801999999999999e-05 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593154+0000", 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0226000000000002e-05 2024-03-26T03:21:25.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593211+0000", 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6708e-05 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593287+0000", 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6550000000000004e-05 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593422+0000", 2024-03-26T03:21:25.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013466800000000001 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159168 10.0 10:057c738e:::benchmark_data_smithi066_148944_object34521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.573141+0000", 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.06196001799998, 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0208405119999999, 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159168 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573141+0000", 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573141+0000", 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573143+0000", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1490000000000001e-06 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573149+0000", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6350000000000003e-06 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573164+0000", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5319e-05 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573172+0000", 2024-03-26T03:21:25.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5360000000000004e-06 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573569+0000", 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039733100000000001 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.573579+0000", 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0019e-05 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571833+0000", 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.99825384900000003 2024-03-26T03:21:25.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571842+0000", 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6509999999999996e-06 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.590252+0000", 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0184094100000001 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.590353+0000", 2024-03-26T03:21:25.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010076799999999999 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.590460+0000", 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107865 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593438+0000", 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029778199999999999 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593501+0000", 2024-03-26T03:21:25.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2299999999999996e-05 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593741+0000", 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000239986 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593815+0000", 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4321999999999998e-05 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593868+0000", 2024-03-26T03:21:25.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2905999999999998e-05 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593981+0000", 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113646 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159183 10.0 10:0070b402:::benchmark_data_smithi066_148944_object34536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.582557+0000", 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.05254370799997, 2024-03-26T03:21:25.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0118551139999998, 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159183 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582557+0000", 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582557+0000", 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582560+0000", 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4700000000000001e-06 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582567+0000", 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8519999999999998e-06 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582581+0000", 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4518999999999999e-05 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582589+0000", 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8339999999999999e-06 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582653+0000", 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3851999999999998e-05 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.582664+0000", 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1088e-05 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571867+0000", 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98920343799999999 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571875+0000", 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8099999999999998e-06 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.591745+0000", 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0198699179999999 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.591829+0000", 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4005000000000006e-05 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.591907+0000", 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7583000000000006e-05 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.593674+0000", 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001767598 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594043+0000", 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036837300000000002 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594109+0000", 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6549000000000003e-05 2024-03-26T03:21:25.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594314+0000", 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000204565 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594349+0000", 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5070000000000001e-05 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594412+0000", 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3590000000000006e-05 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.660 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159193 10.0 10:0574c5af:::benchmark_data_smithi066_148944_object34546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.594587+0000", 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.04051358599997, 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0062639359999999, 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159193 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.661 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594587+0000", 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594587+0000", 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594589+0000", 2024-03-26T03:21:25.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3829999999999999e-06 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594594+0000", 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1680000000000003e-06 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594605+0000", 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1472e-05 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594614+0000", 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6119999999999999e-06 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594680+0000", 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5573000000000004e-05 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.594702+0000", 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2402e-05 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571899+0000", 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97719667600000004 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571907+0000", 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2649999999999996e-06 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594452+0000", 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0225446570000001 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594577+0000", 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125141 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.594711+0000", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134618 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600474+0000", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057629489999999998 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600602+0000", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012815099999999999 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600616+0000", 2024-03-26T03:21:25.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3191e-05 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600666+0000", 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0908000000000001e-05 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600727+0000", 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0872999999999998e-05 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600851+0000", 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012389700000000001 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159211 10.0 10:0157bc40:::benchmark_data_smithi066_148944_object34564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.614464+0000", 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.02063678399998, 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9955604789999999, 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159211 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614464+0000", 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614464+0000", 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614467+0000", 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4279999999999999e-06 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614478+0000", 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1469e-05 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614489+0000", 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1049999999999999e-05 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614497+0000", 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9990000000000008e-06 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614590+0000", 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2986999999999996e-05 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.614602+0000", 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1805e-05 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571940+0000", 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95733820400000003 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571949+0000", 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1959999999999997e-06 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.596335+0000", 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0243859980000001 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.596454+0000", 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119243 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.596573+0000", 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118182 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600533+0000", 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039598990000000002 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600620+0000", 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7175000000000005e-05 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600911+0000", 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029089099999999999 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.609788+0000", 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008877579 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.609859+0000", 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0611000000000005e-05 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610025+0000", 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016576300000000001 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159263 10.0 10:04aad00e:::benchmark_data_smithi066_148944_object34616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.657070+0000", 2024-03-26T03:21:25.676 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.97803045299997, 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9531957680000001, 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159263 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657070+0000", 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657070+0000", 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657071+0000", 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2099999999999996e-07 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657074+0000", 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.898e-06 2024-03-26T03:21:25.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657080+0000", 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6620000000000002e-06 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657086+0000", 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8900000000000004e-06 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657143+0000", 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7263999999999999e-05 2024-03-26T03:21:25.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.657152+0000", 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5000000000000005e-06 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571986+0000", 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91483341600000001 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.571994+0000", 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0569999999999993e-06 2024-03-26T03:21:25.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.598412+0000", 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.026417981 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.598534+0000", 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012180699999999999 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.598676+0000", 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000142664 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600928+0000", 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022519150000000002 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.600984+0000", 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5504999999999998e-05 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.601048+0000", 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4571000000000002e-05 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610083+0000", 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0090350069999999994 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610141+0000", 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7781000000000001e-05 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610266+0000", 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125229 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.683 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159266 10.0 10:03307f1d:::benchmark_data_smithi066_148944_object34619:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.662107+0000", 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.97299428600002, 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9524305749999999, 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159266 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.684 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662107+0000", 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662107+0000", 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662108+0000", 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0899999999999999e-06 2024-03-26T03:21:25.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662112+0000", 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5210000000000004e-06 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662121+0000", 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3180000000000007e-06 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662127+0000", 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7109999999999998e-06 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662187+0000", 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0081000000000003e-05 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.662197+0000", 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0027000000000001e-05 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572018+0000", 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90982019599999997 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572026+0000", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1750000000000005e-06 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.607627+0000", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0356016690000001 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.607753+0000", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012607300000000001 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.607891+0000", 2024-03-26T03:21:25.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013716800000000001 2024-03-26T03:21:25.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610329+0000", 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: 1.1d scrub starts 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: 1.1d scrub ok 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: pgmap v1064: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2645078819' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:25.689 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: from='client.36122 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:25.690 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024379060000000001 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610339+0000", 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0549e-05 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614276+0000", 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039369419999999997 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614339+0000", 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3368999999999996e-05 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614401+0000", 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1693999999999996e-05 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614537+0000", 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136086 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159278 10.0 10:08f1fbe6:::benchmark_data_smithi066_148944_object34631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.671412+0000", 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.96368844699998, 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.947903535, 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159278 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671412+0000", 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671412+0000", 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671414+0000", 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4330000000000001e-06 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671419+0000", 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2660000000000001e-06 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671430+0000", 2024-03-26T03:21:25.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0395e-05 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671438+0000", 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7739999999999993e-06 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671514+0000", 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5190000000000003e-05 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.671525+0000", 2024-03-26T03:21:25.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1328999999999999e-05 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572065+0000", 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.90054052500000004 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572074+0000", 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-06 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610454+0000", 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0383793560000001 2024-03-26T03:21:25.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610535+0000", 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1655000000000003e-05 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.610662+0000", 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012715200000000001 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614550+0000", 2024-03-26T03:21:25.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038874399999999998 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614594+0000", 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4543999999999999e-05 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.618986+0000", 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043918339999999998 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619134+0000", 2024-03-26T03:21:25.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014828700000000001 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619195+0000", 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0328000000000002e-05 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619316+0000", 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000121227 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159281 10.0 10:0da9889e:::benchmark_data_smithi066_148944_object34634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.675662+0000", 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.95943936399999, 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.944017356, 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159281 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675662+0000", 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675662+0000", 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675663+0000", 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2419999999999999e-06 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675668+0000", 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0350000000000001e-06 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675678+0000", 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8609999999999993e-06 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675685+0000", 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5229999999999998e-06 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675774+0000", 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8348000000000002e-05 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.675783+0000", 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4499999999999993e-06 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572099+0000", 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.89631563800000003 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572107+0000", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5080000000000005e-06 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.612698+0000", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0405907619999999 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.612815+0000", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116841 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.612973+0000", 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015818000000000001 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.616650+0000", 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036775190000000002 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619061+0000", 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024105720000000001 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619366+0000", 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030508299999999999 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619521+0000", 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015539199999999999 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619575+0000", 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3526000000000002e-05 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619679+0000", 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010387600000000001 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159300 10.0 10:0d9da093:::benchmark_data_smithi066_148944_object34653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.696572+0000", 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.938529281, 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.923426979, 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159300 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696572+0000", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696572+0000", 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696573+0000", 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7829999999999999e-06 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696579+0000", 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6389999999999997e-06 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696589+0000", 2024-03-26T03:21:25.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6739999999999992e-06 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696597+0000", 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3259999999999996e-06 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696650+0000", 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3443000000000003e-05 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.696658+0000", 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3139999999999998e-06 2024-03-26T03:21:25.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572176+0000", 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87551864400000001 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572186+0000", 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8279999999999998e-06 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614632+0000", 2024-03-26T03:21:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.042446032 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614749+0000", 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117083 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.614902+0000", 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015228799999999999 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619379+0000", 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044775839999999997 2024-03-26T03:21:25.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619465+0000", 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5970000000000005e-05 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619717+0000", 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025213900000000002 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619831+0000", 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113278 2024-03-26T03:21:25.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619886+0000", 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5619999999999999e-05 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619999+0000", 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112334 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.715 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:159309 10.0 10:08341859:::benchmark_data_smithi066_148944_object34662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:01.709099+0000", 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.92600161000001, 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.917722997, 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159309 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709099+0000", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709099+0000", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709100+0000", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.212e-06 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709105+0000", 2024-03-26T03:21:25.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.967e-06 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709114+0000", 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1030000000000002e-06 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709121+0000", 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2749999999999998e-06 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709181+0000", 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0652000000000001e-05 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:01.709191+0000", 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0890000000000006e-06 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572211+0000", 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86302047800000004 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:02.572219+0000", 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9279999999999993e-06 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.616805+0000", 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0445862749999999 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.617013+0000", 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000207744 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.617128+0000", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115396 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619385+0000", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022570530000000002 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.619784+0000", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039864300000000001 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.620061+0000", 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027677399999999998 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626596+0000", 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006534832 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626656+0000", 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.071e-05 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:03.626822+0000", 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000165866 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191613 10.b 10:d1d430eb:::benchmark_data_smithi066_148944_object66966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.773494+0000", 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.86160696600001, 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0939851890000001, 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191613 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773494+0000", 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773494+0000", 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773494+0000", 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1600000000000001e-07 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773497+0000", 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7180000000000001e-06 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773502+0000", 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2059999999999998e-06 2024-03-26T03:21:25.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773507+0000", 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.13e-06 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773576+0000", 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8010999999999996e-05 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773692+0000", 2024-03-26T03:21:25.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011673800000000001 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.773799+0000", 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106469 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.776189+0000", 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023901870000000002 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.782904+0000", 2024-03-26T03:21:25.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0067153320000000001 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.782918+0000", 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3721e-05 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786039+0000", 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003120942 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786055+0000", 2024-03-26T03:21:25.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5684000000000001e-05 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.867387+0000", 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0813322350000001 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.867479+0000", 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2299999999999994e-05 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191663 10.b 10:db3e72d5:::benchmark_data_smithi066_148944_object67016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.847235+0000", 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.78786595999998, 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0289745969999999, 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191663 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847235+0000", 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847235+0000", 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847236+0000", 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3060000000000001e-06 2024-03-26T03:21:25.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847240+0000", 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1620000000000001e-06 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847249+0000", 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5590000000000006e-06 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847254+0000", 2024-03-26T03:21:25.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2599999999999996e-06 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847304+0000", 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9289999999999997e-05 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.847311+0000", 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3980000000000002e-06 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786095+0000", 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93878407500000005 2024-03-26T03:21:25.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786104+0000", 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8470000000000002e-06 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.867705+0000", 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.081600761 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.867781+0000", 2024-03-26T03:21:25.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6656999999999995e-05 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.868132+0000", 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035118500000000001 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.875866+0000", 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0077338509999999999 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.875882+0000", 2024-03-26T03:21:25.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5458999999999999e-05 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.875927+0000", 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5210999999999998e-05 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.875982+0000", 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4834999999999999e-05 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876059+0000", 2024-03-26T03:21:25.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6907000000000001e-05 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876210+0000", 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000150834 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191692 10.b 10:dd320ea5:::benchmark_data_smithi066_148944_object67045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.871384+0000", 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.763716714, 2024-03-26T03:21:25.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0091335510000001, 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191692 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871384+0000", 2024-03-26T03:21:25.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871384+0000", 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871386+0000", 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.533e-06 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871391+0000", 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.118e-06 2024-03-26T03:21:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871409+0000", 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8014999999999999e-05 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871416+0000", 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.34e-06 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871489+0000", 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2649000000000002e-05 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.871499+0000", 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0076e-05 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786134+0000", 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91463466199999999 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786142+0000", 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8189999999999998e-06 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.872098+0000", 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0859560500000001 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.872230+0000", 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132015 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.872373+0000", 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014290200000000001 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.875888+0000", 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003514391 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876297+0000", 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000408923 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876354+0000", 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7312000000000003e-05 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880333+0000", 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039788590000000004 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880395+0000", 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2012000000000002e-05 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880518+0000", 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012287499999999999 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.744 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191697 10.b 10:dd41d791:::benchmark_data_smithi066_148944_object67050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.877669+0000", 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.75743151400002, 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0074741390000002, 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191697 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877669+0000", 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877669+0000", 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877671+0000", 2024-03-26T03:21:25.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1680000000000001e-06 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877675+0000", 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1019999999999999e-06 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877684+0000", 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7450000000000007e-06 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877690+0000", 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6500000000000001e-06 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877735+0000", 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4530999999999998e-05 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.877743+0000", 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4200000000000007e-06 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786171+0000", 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.908427863 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786179+0000", 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.943e-06 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.874129+0000", 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0879503050000001 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.874261+0000", 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013164900000000001 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.874404+0000", 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000142953 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876247+0000", 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018431859999999999 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880577+0000", 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004330095 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880639+0000", 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1532000000000007e-05 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.884810+0000", 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041715490000000001 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.884923+0000", 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011255399999999999 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885144+0000", 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022089399999999999 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191707 10.b 10:d3499b92:::benchmark_data_smithi066_148944_object67060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.883574+0000", 2024-03-26T03:21:25.752 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.75152683900001, 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.002075863, 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191707 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883574+0000", 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883574+0000", 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883576+0000", 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.734e-06 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883581+0000", 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2499999999999997e-06 2024-03-26T03:21:25.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883594+0000", 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3142999999999999e-05 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883602+0000", 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6279999999999996e-06 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883661+0000", 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8693999999999997e-05 2024-03-26T03:21:25.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.883670+0000", 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7329999999999998e-06 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786207+0000", 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.902537056 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786215+0000", 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9070000000000007e-06 2024-03-26T03:21:25.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876402+0000", 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0901868480000001 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876474+0000", 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2057000000000005e-05 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.876573+0000", 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9170000000000001e-05 2024-03-26T03:21:25.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880653+0000", 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040800109999999997 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880683+0000", 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9674000000000001e-05 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880726+0000", 2024-03-26T03:21:25.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2616e-05 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885411+0000", 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046856190000000002 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885491+0000", 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9599999999999997e-05 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885650+0000", 2024-03-26T03:21:25.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000159123 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191764 10.b 10:db8e53ab:::benchmark_data_smithi066_148944_object67117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.934019+0000", 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.70108167500001, 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9520802150000001, 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.760 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191764 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934019+0000", 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934019+0000", 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934021+0000", 2024-03-26T03:21:25.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.694e-06 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934031+0000", 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6330000000000008e-06 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934046+0000", 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5209e-05 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934054+0000", 2024-03-26T03:21:25.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2570000000000007e-06 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934115+0000", 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1363000000000002e-05 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.934125+0000", 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0003999999999999e-05 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786243+0000", 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.85211766200000005 2024-03-26T03:21:25.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786251+0000", 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9519999999999994e-06 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.878304+0000", 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0920527760000001 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.878384+0000", 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0364000000000005e-05 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.878462+0000", 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7854000000000004e-05 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880662+0000", 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021998819999999998 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880766+0000", 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104398 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880827+0000", 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1039999999999998e-05 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885910+0000", 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050822949999999997 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885972+0000", 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2053999999999999e-05 2024-03-26T03:21:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.886099+0000", 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127778 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191787 10.b 10:d7ee6479:::benchmark_data_smithi066_148944_object67140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.957576+0000", 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.677524608, 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9328720640000001, 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191787 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957576+0000", 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957576+0000", 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957578+0000", 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.452e-06 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957583+0000", 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5969999999999999e-06 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957598+0000", 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4759999999999999e-05 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957605+0000", 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.345e-06 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957667+0000", 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1684000000000001e-05 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.957677+0000", 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0064999999999999e-05 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786282+0000", 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.82860502300000005 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786397+0000", 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114406 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880872+0000", 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944758590000001 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.880999+0000", 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012604399999999999 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.881134+0000", 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013569600000000001 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885668+0000", 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045334090000000004 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885748+0000", 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0112999999999997e-05 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885833+0000", 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369000000000002e-05 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.890245+0000", 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004412366 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.890311+0000", 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5838000000000002e-05 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.890448+0000", 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137038 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.774 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191806 10.b 10:db07b84b:::benchmark_data_smithi066_148944_object67159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.975183+0000", 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.659917678, 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9196836020000001, 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.775 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191806 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975183+0000", 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975183+0000", 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975185+0000", 2024-03-26T03:21:25.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.395e-06 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975190+0000", 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.091e-06 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975204+0000", 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4158000000000001e-05 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975212+0000", 2024-03-26T03:21:25.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1140000000000006e-06 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975273+0000", 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0983999999999998e-05 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.975283+0000", 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7950000000000002e-06 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786440+0000", 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.81115747100000002 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786450+0000", 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6460000000000005e-06 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.882941+0000", 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0964915369999999 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.883078+0000", 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136539 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.883228+0000", 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014973699999999999 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.885681+0000", 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024534470000000001 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.886152+0000", 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047053299999999998 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.890503+0000", 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004351092 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.894667+0000", 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00416403 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.894727+0000", 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0423000000000001e-05 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.894867+0000", 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013961 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:191816 10.b 10:da2ce06c:::benchmark_data_smithi066_148944_object67169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:15:35.984600+0000", 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 349.65050115100001, 2024-03-26T03:21:25.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9148800070000001, 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 191816 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984600+0000", 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984600+0000", 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984601+0000", 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1999999999999998e-07 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984605+0000", 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8709999999999999e-06 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984614+0000", 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3079999999999997e-06 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984622+0000", 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0190000000000006e-06 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984690+0000", 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8610999999999998e-05 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:35.984702+0000", 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1044e-05 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786479+0000", 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80177767899999997 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:36.786487+0000", 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6629999999999993e-06 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.886186+0000", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0996990719999999 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.886310+0000", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000124189 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2,4", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.886429+0000", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011863500000000001 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:25.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.890464+0000", 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040352560000000001 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.894925+0000", 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004461217 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.899217+0000", 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00429197 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:25.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.899303+0000", 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6224999999999995e-05 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.899357+0000", 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3531e-05 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:15:37.899480+0000", 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012279700000000001 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:25.789 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: 1.1d scrub starts 2024-03-26T03:21:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: 1.1d scrub ok 2024-03-26T03:21:25.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: pgmap v1064: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2645078819' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: from='client.36122 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: 1.1d scrub starts 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: 1.1d scrub ok 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: pgmap v1064: 162 pgs: 1 active+clean+scrubbing, 161 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2645078819' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:25.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: from='client.36122 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:21:25.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:26.298 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T03:21:26.568 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: from='client.36122 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:21:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T03:21:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:21:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:21:26.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: 5.10 scrub starts 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: 5.10 scrub ok 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[29666]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: from='client.36122 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: 5.10 scrub starts 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: 5.10 scrub ok 2024-03-26T03:21:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:26 smithi066 ceph-mon[37920]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T03:21:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: from='client.36122 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:21:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T03:21:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:21:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:21:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: 5.10 scrub starts 2024-03-26T03:21:26.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: 5.10 scrub ok 2024-03-26T03:21:26.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:26 smithi192 ceph-mon[33110]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T03:21:26.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:21:27.081 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:27 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:21:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:27 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:21:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:27 smithi192 ceph-mon[33110]: pgmap v1067: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T03:21:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:27 smithi192 ceph-mon[33110]: 6.5 scrub starts 2024-03-26T03:21:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:27 smithi192 ceph-mon[33110]: 6.5 scrub ok 2024-03-26T03:21:27.751 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:21:27.751 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:21:27.751 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[29666]: pgmap v1067: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T03:21:27.751 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[29666]: 6.5 scrub starts 2024-03-26T03:21:27.751 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[29666]: 6.5 scrub ok 2024-03-26T03:21:27.752 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:21:27.752 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:21:27.752 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[37920]: pgmap v1067: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T03:21:27.752 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[37920]: 6.5 scrub starts 2024-03-26T03:21:27.752 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:27 smithi066 ceph-mon[37920]: 6.5 scrub ok 2024-03-26T03:21:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 6.d scrub starts 2024-03-26T03:21:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 6.d scrub ok 2024-03-26T03:21:28.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 1.8 deep-scrub starts 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 1.8 deep-scrub ok 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: Health check failed: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 6.1f scrub starts 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 6.1f scrub ok 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 6.d scrub starts 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 6.d scrub ok 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 1.8 deep-scrub starts 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 1.8 deep-scrub ok 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: Health check failed: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 6.1f scrub starts 2024-03-26T03:21:28.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 6.1f scrub ok 2024-03-26T03:21:28.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:21:28.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:28 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:21:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 6.d scrub starts 2024-03-26T03:21:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 6.d scrub ok 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 1.8 deep-scrub starts 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 1.8 deep-scrub ok 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: Health check failed: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 6.1f scrub starts 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 6.1f scrub ok 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:21:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:28 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:21:28.764 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:28.764 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:28.764 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-03-26T03:21:28.764 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:29.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:21:29.488 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T03:21:29.624 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: 4.1a scrub starts 2024-03-26T03:21:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: 4.1a scrub ok 2024-03-26T03:21:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: pgmap v1068: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T03:21:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1743016856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: from='client.36128 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: 1.0 scrub starts 2024-03-26T03:21:29.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:29 smithi192 ceph-mon[33110]: 1.0 scrub ok 2024-03-26T03:21:29.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: 4.1a scrub starts 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: 4.1a scrub ok 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: pgmap v1068: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1743016856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: from='client.36128 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: 1.0 scrub starts 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[29666]: 1.0 scrub ok 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: 4.1a scrub starts 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: 4.1a scrub ok 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: pgmap v1068: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1743016856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:29.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: from='client.36128 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:29.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: 1.0 scrub starts 2024-03-26T03:21:29.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:29 smithi066 ceph-mon[37920]: 1.0 scrub ok 2024-03-26T03:21:30.228 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-26T03:21:30.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:21:30.491 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: 2.6 scrub starts 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: 2.6 scrub ok 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: from='client.36128 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:21:30.736 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:30 smithi192 ceph-mon[33110]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: 2.6 scrub starts 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: 2.6 scrub ok 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: from='client.36128 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[29666]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T03:21:30.737 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: 2.6 scrub starts 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: 2.6 scrub ok 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: from='client.36128 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:21:30.738 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:30 smithi066 ceph-mon[37920]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T03:21:31.305 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-03-26T03:21:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-03-26T03:21:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-03-26T03:21:31.305 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:31.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:31 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:21:31.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:31 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:21:31.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:31.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:31 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:21:31.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:31 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:21:31.619 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:31 smithi192 ceph-mon[33110]: pgmap v1070: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-26T03:21:31.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:21:31.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:21:31.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:31.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:21:31.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:21:31.761 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[37920]: pgmap v1070: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-26T03:21:31.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:21:31.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:21:31.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:31.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:21:31.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:21:31.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:31 smithi066 ceph-mon[29666]: pgmap v1070: 162 pgs: 13 peering, 149 active+clean; 711 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-26T03:21:31.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:21:32.160 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:21:32.605 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:32 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:21:32.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:32 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:21:32.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:32 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:21:32.606 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:32 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:21:32.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:21:32.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:32 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:21:33.503 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: 5.17 scrub starts 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: 5.17 scrub ok 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: pgmap v1071: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2017150961' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:33.772 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:33 smithi192 ceph-mon[33110]: from='client.36134 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:33.781 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-26T03:21:33.781 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137152 10.c 10:39382015:::benchmark_data_smithi066_148944_object12505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.228227+0000", 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 417.551666023, 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0128037700000001, 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:33.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137152 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228227+0000", 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228227+0000", 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228229+0000", 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.606e-06 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228239+0000", 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0379e-05 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228258+0000", 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8518000000000001e-05 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228269+0000", 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1591e-05 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228326+0000", 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6226e-05 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228495+0000", 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016949099999999999 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.228655+0000", 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160063 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.233055+0000", 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044002210000000002 2024-03-26T03:21:33.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.239260+0000", 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0062048579999999997 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.264903+0000", 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025642841 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.264916+0000", 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3325e-05 2024-03-26T03:21:33.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241014+0000", 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97609788799999997 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241031+0000", 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6762999999999999e-05 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137181 10.c 10.72630f0c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.286355+0000", 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 417.493537962, 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95471075100000002, 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137181 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286355+0000", 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286355+0000", 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286357+0000", 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1940000000000001e-06 2024-03-26T03:21:33.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286364+0000", 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6200000000000001e-06 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286379+0000", 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5282999999999999e-05 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286387+0000", 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5789999999999998e-06 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286447+0000", 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0154e-05 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.286458+0000", 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0808000000000001e-05 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241057+0000", 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95459924399999996 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241066+0000", 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8689999999999993e-06 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25522.0:137196 10.c 10.e56587ac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:36.308988+0000", 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 417.47090543000002, 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93210979800000004, 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25522", 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137196 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.794 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308988+0000", 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308988+0000", 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308990+0000", 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.351e-06 2024-03-26T03:21:33.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.308995+0000", 2024-03-26T03:21:33.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1710000000000001e-06 2024-03-26T03:21:33.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309010+0000", 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4812e-05 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309017+0000", 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1119999999999998e-06 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309078+0000", 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0445999999999997e-05 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:36.309088+0000", 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0752e-05 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241089+0000", 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.93200086800000004 2024-03-26T03:21:33.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:37.241098+0000", 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2859999999999999e-06 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137219 10.9 e343/342 10:9a86151a:::benchmark_data_smithi066_148944_object12572:head v 343'8542, mlcod=0'0)", 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.326483+0000", 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.45341032599998, 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47544255400000002, 2024-03-26T03:21:33.799 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326483+0000", 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326483+0000", 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326485+0000", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.73e-06 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326493+0000", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7339999999999997e-06 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326503+0000", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.006e-05 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326510+0000", 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8299999999999998e-06 2024-03-26T03:21:33.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326583+0000", 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2498999999999998e-05 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.326927+0000", 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034401400000000002 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801882+0000", 2024-03-26T03:21:33.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47495507399999998 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801925+0000", 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3612999999999997e-05 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.803 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137247 10.9 e343/342 10:9cdb27b9:::benchmark_data_smithi066_148944_object12600:head v 343'8543, mlcod=0'0)", 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.335355+0000", 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.44453792799999, 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57069690200000001, 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335355+0000", 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335355+0000", 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335357+0000", 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2860000000000001e-06 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335364+0000", 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3100000000000003e-06 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335373+0000", 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3009999999999995e-06 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335380+0000", 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4110000000000001e-06 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.352564+0000", 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017184041000000001 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.352586+0000", 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2751999999999998e-05 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.905930+0000", 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55334367399999995 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906052+0000", 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000122127 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.807 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137258 10.8 e343/342 10:1812d821:::benchmark_data_smithi066_148944_object12611:head v 343'8573, mlcod=0'0)", 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.335708+0000", 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.44418478599999, 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46613804800000003, 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335708+0000", 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335708+0000", 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335710+0000", 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0860000000000001e-06 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335719+0000", 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7110000000000007e-06 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335728+0000", 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0380000000000005e-06 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.335735+0000", 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.537398+0000", 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20166337500000001 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.537427+0000", 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.853e-05 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801809+0000", 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26438231699999998 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801847+0000", 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7389999999999999e-05 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137254 10.9 e343/342 10:97a448bb:::benchmark_data_smithi066_148944_object12607:head v 343'8544, mlcod=0'0)", 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.337366+0000", 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.442527138, 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.568755389, 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337366+0000", 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337366+0000", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337367+0000", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.221e-06 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337375+0000", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3440000000000003e-06 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337389+0000", 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4479e-05 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.337397+0000", 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5639999999999999e-06 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.356826+0000", 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019429168 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.357212+0000", 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038587099999999998 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906056+0000", 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54884401500000002 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906122+0000", 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5727000000000003e-05 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137266 10.9 e343/342 10:9c7c1621:::benchmark_data_smithi066_148944_object12619:head v 343'8545, mlcod=0'0)", 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.488743+0000", 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.291150276, 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55176513699999996, 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488743+0000", 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488743+0000", 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488745+0000", 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.826e-06 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488753+0000", 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6629999999999999e-06 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488768+0000", 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4498000000000001e-05 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488776+0000", 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3909999999999998e-06 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708819+0000", 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.220042882 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708835+0000", 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5818e-05 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040459+0000", 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33162439900000001 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040508+0000", 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8659999999999998e-05 2024-03-26T03:21:33.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137265 10.1 e343/341 10:8eba7ad0:::benchmark_data_smithi066_148944_object12618:head v 343'8713, mlcod=342'8711)", 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.488970+0000", 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.290922986, 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55538486899999995, 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488970+0000", 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488970+0000", 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488972+0000", 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3990000000000001e-06 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488978+0000", 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6460000000000004e-06 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488992+0000", 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3434e-05 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.488999+0000", 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8410000000000002e-06 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801839+0000", 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.31284087700000002 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801863+0000", 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3876999999999999e-05 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.044306+0000", 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24244289799999999 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.044355+0000", 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8897000000000003e-05 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137271 10.9 e343/342 10:96c90966:::benchmark_data_smithi066_148944_object12624:head v 343'8546, mlcod=0'0)", 2024-03-26T03:21:33.824 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.547439+0000", 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.23245419300002, 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49976598, 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547439+0000", 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547439+0000", 2024-03-26T03:21:33.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547441+0000", 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547451+0000", 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6339999999999996e-06 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547467+0000", 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6708e-05 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.547478+0000", 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0818e-05 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801958+0000", 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25448008500000002 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801995+0000", 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6918000000000001e-05 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.046398+0000", 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.244402386 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047205+0000", 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000807386 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.828 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137274 10.8 e343/342 10:15ded6e2:::benchmark_data_smithi066_148944_object12627:head v 343'8575, mlcod=343'8571)", 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.642726+0000", 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.13716767900002, 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39793152999999998, 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642726+0000", 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642726+0000", 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642727+0000", 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.42e-06 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642734+0000", 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3100000000000003e-06 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642746+0000", 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2091e-05 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.642754+0000", 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6860000000000006e-06 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708753+0000", 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.065998860000000006 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.708781+0000", 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7880000000000001e-05 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040583+0000", 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33180187300000002 2024-03-26T03:21:33.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040657+0000", 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4410000000000001e-05 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137279 10.1 e343/341 10:8adca3c1:::benchmark_data_smithi066_148944_object12632:head v 343'8714, mlcod=342'8711)", 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.645279+0000", 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.13461391599998, 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40479705599999999, 2024-03-26T03:21:33.833 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645279+0000", 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645279+0000", 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645282+0000", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645291+0000", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6870000000000006e-06 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645308+0000", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6716000000000001e-05 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.645317+0000", 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3549999999999994e-06 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.905951+0000", 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26063393499999998 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.905984+0000", 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2678999999999997e-05 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.049973+0000", 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14398904600000001 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050076+0000", 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103428 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.837 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137281 10.1 e343/341 10:8598039e:::benchmark_data_smithi066_148944_object12634:head v 343'8715, mlcod=342'8711)", 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.648536+0000", 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.13135728499998, 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.401635768, 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648536+0000", 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648536+0000", 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648538+0000", 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7349999999999999e-06 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648547+0000", 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0799999999999995e-06 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648575+0000", 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8367000000000001e-05 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.648584+0000", 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2070000000000001e-06 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040465+0000", 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39188030800000001 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040490+0000", 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5731000000000001e-05 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050116+0000", 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0096260900000000003 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050172+0000", 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5250000000000001e-05 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137282 10.1 e343/341 10:86db17fd:::benchmark_data_smithi066_148944_object12635:head v 343'8716, mlcod=342'8711)", 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.650848+0000", 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.12904505699998, 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39937403100000002, 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650848+0000", 2024-03-26T03:21:33.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650848+0000", 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650850+0000", 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9539999999999998e-06 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650859+0000", 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1290000000000002e-06 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650877+0000", 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7227999999999998e-05 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.650885+0000", 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6009999999999995e-06 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042328+0000", 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39144269500000001 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042348+0000", 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050174+0000", 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078259000000000002 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.050222+0000", 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8161999999999997e-05 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137283 10.8 e343/342 10:18321bbd:::benchmark_data_smithi066_148944_object12636:head v 343'8576, mlcod=343'8571)", 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.652601+0000", 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.12729256900002, 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.390188335, 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.846 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652601+0000", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652601+0000", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652602+0000", 2024-03-26T03:21:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5629999999999999e-06 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652611+0000", 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.149e-06 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652627+0000", 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5875e-05 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.652635+0000", 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1040000000000007e-06 2024-03-26T03:21:33.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801872+0000", 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.149237114 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.801890+0000", 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7972999999999999e-05 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042720+0000", 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.240829876 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.042789+0000", 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8681000000000002e-05 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: 5.17 scrub starts 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: 5.17 scrub ok 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: pgmap v1071: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2017150961' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[29666]: from='client.36134 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:33.851 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:21:33.852 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:21:33.852 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: 5.17 scrub starts 2024-03-26T03:21:33.852 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: 5.17 scrub ok 2024-03-26T03:21:33.852 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: pgmap v1071: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-26T03:21:33.852 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2017150961' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:33.852 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-mon[37920]: from='client.36134 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137284 10.1 e343/341 10:83f5b3bd:::benchmark_data_smithi066_148944_object12637:head v 343'8717, mlcod=342'8711)", 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.654011+0000", 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.12588259799998, 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40273788399999999, 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.852 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654011+0000", 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654011+0000", 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654012+0000", 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4789999999999999e-06 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654021+0000", 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5320000000000006e-06 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654042+0000", 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1755000000000001e-05 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.654050+0000", 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3339999999999996e-06 2024-03-26T03:21:33.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.045994+0000", 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39194385999999998 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.046016+0000", 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.232e-05 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.056702+0000", 2024-03-26T03:21:33.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010686322999999999 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.056749+0000", 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6281e-05 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137288 10.9 e343/342 10:9d93d421:::benchmark_data_smithi066_148944_object12641:head v 343'8547, mlcod=0'0)", 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.656335+0000", 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 415.12355804800001, 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39090719400000001, 2024-03-26T03:21:33.856 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656335+0000", 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656335+0000", 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656337+0000", 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.494e-06 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656345+0000", 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6109999999999994e-06 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656360+0000", 2024-03-26T03:21:33.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4423999999999999e-05 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.656368+0000", 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0029999999999994e-06 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906207+0000", 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24983883000000001 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.906231+0000", 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4131e-05 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047210+0000", 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14097958099999999 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.047242+0000", 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2119999999999997e-05 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.860 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:137296 10.8 e343/342 10:12c9bc13:::benchmark_data_smithi066_148944_object12649:head v 343'8578, mlcod=343'8572)", 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:38.811312+0000", 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 414.96858076000001, 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.417516, 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811312+0000", 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811312+0000", 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811314+0000", 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811325+0000", 2024-03-26T03:21:33.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0745e-05 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811340+0000", 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4419e-05 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:38.811348+0000", 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5760000000000006e-06 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040799+0000", 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22945073699999999 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.040824+0000", 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5324999999999999e-05 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.228786+0000", 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.187961355 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:39.228828+0000", 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2845000000000001e-05 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:138292 10.5 e345/341 10:a14f9faa:::benchmark_data_smithi066_148944_object13645:head v 345'8656, mlcod=0'0)", 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.188450+0000", 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 413.59144301200001, 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43925914700000002, 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188450+0000", 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188450+0000", 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188451+0000", 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7300000000000004e-07 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188458+0000", 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4869999999999999e-06 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188466+0000", 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1510000000000004e-06 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188472+0000", 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5359999999999998e-06 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188773+0000", 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000300635 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.188795+0000", 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1716e-05 2024-03-26T03:21:33.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627678+0000", 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43888337999999999 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627709+0000", 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1269000000000002e-05 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25522.0:138344 10.5 e345/341 10:a8c08bd4:::benchmark_data_smithi066_148944_object13697:head v 345'8657, mlcod=0'0)", 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-03-26T03:14:40.236829+0000", 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 413.54306452399999, 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39092764000000002, 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236829+0000", 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236829+0000", 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236831+0000", 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0540000000000002e-06 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236841+0000", 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0234000000000001e-05 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236860+0000", 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9051000000000001e-05 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.236869+0000", 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2890000000000003e-06 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.237100+0000", 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023038600000000001 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.237128+0000", 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8132000000000001e-05 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627731+0000", 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39060298199999999 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-03-26T03:14:40.627756+0000", 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5511999999999999e-05 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-26T03:21:33.874 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-26T03:21:33.874 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-26T03:21:34.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:21:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:21:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:21:34.264 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T03:21:34.532 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:34.584 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:21:34.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: 1.16 scrub starts 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: 1.16 scrub ok 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: 6.1b scrub starts 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: 6.1b scrub ok 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 11 pgs peering) 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: Cluster is now healthy 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: from='client.36134 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[37920]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: 1.16 scrub starts 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: 1.16 scrub ok 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: 6.1b scrub starts 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: 6.1b scrub ok 2024-03-26T03:21:34.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 11 pgs peering) 2024-03-26T03:21:34.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: Cluster is now healthy 2024-03-26T03:21:34.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: from='client.36134 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:21:34.810 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:34 smithi066 ceph-mon[29666]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T03:21:34.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: 1.16 scrub starts 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: 1.16 scrub ok 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: 6.1b scrub starts 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: 6.1b scrub ok 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 11 pgs peering) 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: Cluster is now healthy 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: from='client.36134 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:21:34.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:34 smithi192 ceph-mon[33110]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T03:21:35.777 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[29666]: 5.a scrub starts 2024-03-26T03:21:35.777 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[29666]: 5.a scrub ok 2024-03-26T03:21:35.777 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[29666]: 6.17 scrub starts 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[29666]: pgmap v1073: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[29666]: 2.1c deep-scrub starts 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[29666]: 2.1c deep-scrub ok 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[37920]: 5.a scrub starts 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[37920]: 5.a scrub ok 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[37920]: 6.17 scrub starts 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[37920]: pgmap v1073: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[37920]: 2.1c deep-scrub starts 2024-03-26T03:21:35.778 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:35 smithi066 ceph-mon[37920]: 2.1c deep-scrub ok 2024-03-26T03:21:35.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:35 smithi192 ceph-mon[33110]: 5.a scrub starts 2024-03-26T03:21:35.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:35 smithi192 ceph-mon[33110]: 5.a scrub ok 2024-03-26T03:21:35.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:35 smithi192 ceph-mon[33110]: 6.17 scrub starts 2024-03-26T03:21:35.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:35 smithi192 ceph-mon[33110]: pgmap v1073: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:35.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:35 smithi192 ceph-mon[33110]: 2.1c deep-scrub starts 2024-03-26T03:21:35.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:35 smithi192 ceph-mon[33110]: 2.1c deep-scrub ok 2024-03-26T03:21:36.530 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 6.17 scrub ok 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 1.f scrub starts 2024-03-26T03:21:36.774 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[29666]: 1.f scrub ok 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 6.17 scrub ok 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 1.f scrub starts 2024-03-26T03:21:36.775 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:36 smithi066 ceph-mon[37920]: 1.f scrub ok 2024-03-26T03:21:36.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 6.17 scrub ok 2024-03-26T03:21:36.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:21:36.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:21:36.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:21:36.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:21:36.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 1.f scrub starts 2024-03-26T03:21:36.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:36 smithi192 ceph-mon[33110]: 1.f scrub ok 2024-03-26T03:21:37.516 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T03:21:37.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[37920]: 2.13 scrub starts 2024-03-26T03:21:37.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[37920]: 2.13 scrub ok 2024-03-26T03:21:37.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[37920]: pgmap v1074: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/14081561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[37920]: 4.a scrub starts 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[37920]: 4.a scrub ok 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[29666]: 2.13 scrub starts 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[29666]: 2.13 scrub ok 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[29666]: pgmap v1074: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/14081561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[29666]: 4.a scrub starts 2024-03-26T03:21:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:37 smithi066 ceph-mon[29666]: 4.a scrub ok 2024-03-26T03:21:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:37 smithi192 ceph-mon[33110]: 2.13 scrub starts 2024-03-26T03:21:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:37 smithi192 ceph-mon[33110]: 2.13 scrub ok 2024-03-26T03:21:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:37 smithi192 ceph-mon[33110]: pgmap v1074: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:37.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:37 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/14081561' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:21:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:37 smithi192 ceph-mon[33110]: 4.a scrub starts 2024-03-26T03:21:37.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:37 smithi192 ceph-mon[33110]: 4.a scrub ok 2024-03-26T03:21:38.514 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 5.d scrub starts 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 5.d scrub ok 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 4.19 scrub starts 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 4.19 scrub ok 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 1.5 deep-scrub starts 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 1.5 deep-scrub ok 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/14081561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 6.1c scrub starts 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[29666]: 6.1c scrub ok 2024-03-26T03:21:38.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 5.d scrub starts 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 5.d scrub ok 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 4.19 scrub starts 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 4.19 scrub ok 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 1.5 deep-scrub starts 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 1.5 deep-scrub ok 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/14081561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 6.1c scrub starts 2024-03-26T03:21:38.764 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:38 smithi066 ceph-mon[37920]: 6.1c scrub ok 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 5.d scrub starts 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 5.d scrub ok 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 4.19 scrub starts 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 4.19 scrub ok 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 1.5 deep-scrub starts 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 1.5 deep-scrub ok 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/14081561' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 6.1c scrub starts 2024-03-26T03:21:38.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:38 smithi192 ceph-mon[33110]: 6.1c scrub ok 2024-03-26T03:21:39.011 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:21:39.012 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:21:39.012 INFO:teuthology.orchestra.run.smithi066.stdout: "num_ops": 0 2024-03-26T03:21:39.012 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:21:39.020 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:39.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:21:39.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:21:39.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: 2.1e scrub starts 2024-03-26T03:21:39.661 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: 2.1e scrub ok 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: pgmap v1076: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: 1.18 scrub starts 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[29666]: 1.18 scrub ok 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: 2.1e scrub starts 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: 2.1e scrub ok 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: pgmap v1076: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: 1.18 scrub starts 2024-03-26T03:21:39.662 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:39 smithi066 ceph-mon[37920]: 1.18 scrub ok 2024-03-26T03:21:39.688 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:21:39.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:21:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:21:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: 2.1e scrub starts 2024-03-26T03:21:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: 2.1e scrub ok 2024-03-26T03:21:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: pgmap v1076: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: 1.18 scrub starts 2024-03-26T03:21:39.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:39 smithi192 ceph-mon[33110]: 1.18 scrub ok 2024-03-26T03:21:40.532 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:21:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:21:40.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:40 smithi066 ceph-mon[29666]: 6.2 deep-scrub starts 2024-03-26T03:21:40.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:40 smithi066 ceph-mon[29666]: 6.2 deep-scrub ok 2024-03-26T03:21:40.802 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:40.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:40 smithi066 ceph-mon[37920]: 6.2 deep-scrub starts 2024-03-26T03:21:40.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:40 smithi066 ceph-mon[37920]: 6.2 deep-scrub ok 2024-03-26T03:21:40.802 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:40.809 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:40.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:40 smithi192 ceph-mon[33110]: 6.2 deep-scrub starts 2024-03-26T03:21:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:40 smithi192 ceph-mon[33110]: 6.2 deep-scrub ok 2024-03-26T03:21:40.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[29666]: 2.1f deep-scrub starts 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[29666]: 2.1f deep-scrub ok 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[29666]: pgmap v1077: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[37920]: 2.1f deep-scrub starts 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[37920]: 2.1f deep-scrub ok 2024-03-26T03:21:41.768 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[37920]: pgmap v1077: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:41.769 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:41 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-26T03:21:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:41 smithi192 ceph-mon[33110]: 2.1f deep-scrub starts 2024-03-26T03:21:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:41 smithi192 ceph-mon[33110]: 2.1f deep-scrub ok 2024-03-26T03:21:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:41 smithi192 ceph-mon[33110]: pgmap v1077: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:21:41.889 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:41 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-03-26T03:21:42.547 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-26T03:21:42.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-26T03:21:42.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T03:21:42.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/750976425' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:21:42.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:21:42.716 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: 3.0 scrub starts 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[29666]: 3.0 scrub ok 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/750976425' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: 3.0 scrub starts 2024-03-26T03:21:42.717 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:42 smithi066 ceph-mon[37920]: 3.0 scrub ok 2024-03-26T03:21:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-03-26T03:21:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T03:21:42.908 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/750976425' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:21:42.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:21:42.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:21:42.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: 3.0 scrub starts 2024-03-26T03:21:42.909 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:42 smithi192 ceph-mon[33110]: 3.0 scrub ok 2024-03-26T03:21:43.465 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T03:21:43.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: pgmap v1079: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:43.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/750976425' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:21:43.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T03:21:43.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: 2.a scrub starts 2024-03-26T03:21:43.714 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: 2.a scrub ok 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: 6.3 deep-scrub starts 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[37920]: 6.3 deep-scrub ok 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: pgmap v1079: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/750976425' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: 2.a scrub starts 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: 2.a scrub ok 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: 6.3 deep-scrub starts 2024-03-26T03:21:43.715 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-mon[29666]: 6.3 deep-scrub ok 2024-03-26T03:21:43.737 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:21:43.737 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [], 2024-03-26T03:21:43.737 INFO:teuthology.orchestra.run.smithi066.stdout: "complaint_time": 30, 2024-03-26T03:21:43.738 INFO:teuthology.orchestra.run.smithi066.stdout: "num_blocked_ops": 0 2024-03-26T03:21:43.738 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:21:43.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: pgmap v1079: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:43.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/750976425' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:21:43.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T03:21:43.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: 2.a scrub starts 2024-03-26T03:21:43.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: 2.a scrub ok 2024-03-26T03:21:43.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: 6.3 deep-scrub starts 2024-03-26T03:21:43.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:43 smithi192 ceph-mon[33110]: 6.3 deep-scrub ok 2024-03-26T03:21:44.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:21:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:21:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:21:44.456 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:21:44.469 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: 2.5 scrub starts 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: 2.5 scrub ok 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: 1.e scrub starts 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: 1.e scrub ok 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: 1.8 scrub starts 2024-03-26T03:21:44.730 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[29666]: 1.8 scrub ok 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: 2.5 scrub starts 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: 2.5 scrub ok 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: 1.e scrub starts 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: 1.e scrub ok 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: 1.8 scrub starts 2024-03-26T03:21:44.731 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:44 smithi066 ceph-mon[37920]: 1.8 scrub ok 2024-03-26T03:21:44.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: 2.5 scrub starts 2024-03-26T03:21:44.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: 2.5 scrub ok 2024-03-26T03:21:44.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: 1.e scrub starts 2024-03-26T03:21:44.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: 1.e scrub ok 2024-03-26T03:21:44.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T03:21:44.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: 1.8 scrub starts 2024-03-26T03:21:44.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:44 smithi192 ceph-mon[33110]: 1.8 scrub ok 2024-03-26T03:21:45.225 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:45.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:45 smithi066 ceph-mon[37920]: pgmap v1082: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:45 smithi066 ceph-mon[37920]: 1.a scrub starts 2024-03-26T03:21:45.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:45 smithi066 ceph-mon[37920]: 1.a scrub ok 2024-03-26T03:21:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:45 smithi066 ceph-mon[29666]: pgmap v1082: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:45 smithi066 ceph-mon[29666]: 1.a scrub starts 2024-03-26T03:21:45.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:45 smithi066 ceph-mon[29666]: 1.a scrub ok 2024-03-26T03:21:45.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:45 smithi192 ceph-mon[33110]: pgmap v1082: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:45.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:45 smithi192 ceph-mon[33110]: 1.a scrub starts 2024-03-26T03:21:45.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:45 smithi192 ceph-mon[33110]: 1.a scrub ok 2024-03-26T03:21:46.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[37920]: 6.13 scrub starts 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[37920]: 6.13 scrub ok 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[37920]: 1.1 scrub starts 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[37920]: 1.1 scrub ok 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[37920]: 2.8 scrub starts 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[37920]: 2.8 scrub ok 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[29666]: 6.13 scrub starts 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[29666]: 6.13 scrub ok 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[29666]: 1.1 scrub starts 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[29666]: 1.1 scrub ok 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[29666]: 2.8 scrub starts 2024-03-26T03:21:46.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:46 smithi066 ceph-mon[29666]: 2.8 scrub ok 2024-03-26T03:21:46.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:46 smithi192 ceph-mon[33110]: 6.13 scrub starts 2024-03-26T03:21:46.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:46 smithi192 ceph-mon[33110]: 6.13 scrub ok 2024-03-26T03:21:46.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:46 smithi192 ceph-mon[33110]: 1.1 scrub starts 2024-03-26T03:21:46.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:46 smithi192 ceph-mon[33110]: 1.1 scrub ok 2024-03-26T03:21:46.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:46 smithi192 ceph-mon[33110]: 2.8 scrub starts 2024-03-26T03:21:46.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:46 smithi192 ceph-mon[33110]: 2.8 scrub ok 2024-03-26T03:21:47.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:21:47.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:21:47.825 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 6.6 scrub starts 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 6.6 scrub ok 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 5.e scrub starts 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 5.e scrub ok 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: pgmap v1083: 162 pgs: 162 active+clean; 635 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 6.0 KiB/s rd, 36 KiB/s wr, 7 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 1.10 scrub starts 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 1.10 scrub ok 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 2.1d scrub starts 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[29666]: 2.1d scrub ok 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:21:47.826 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 6.6 scrub starts 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 6.6 scrub ok 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 5.e scrub starts 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 5.e scrub ok 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: pgmap v1083: 162 pgs: 162 active+clean; 635 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 6.0 KiB/s rd, 36 KiB/s wr, 7 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 1.10 scrub starts 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 1.10 scrub ok 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 2.1d scrub starts 2024-03-26T03:21:47.827 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:47 smithi066 ceph-mon[37920]: 2.1d scrub ok 2024-03-26T03:21:47.953 INFO:teuthology.orchestra.run.smithi066.stdout:{ 2024-03-26T03:21:47.953 INFO:teuthology.orchestra.run.smithi066.stdout: "size": 20, 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 600, 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "ops": [ 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:124423 10.3 10.bdd83133 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e331)", 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:20.565202+0000", 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 447.38612877600002, 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0004394190000001, 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:21:47.954 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 124423 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565202+0000", 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565202+0000", 2024-03-26T03:21:47.955 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565204+0000", 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.6249999999999999e-06 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565276+0000", 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1457000000000004e-05 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.956 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565286+0000", 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0421000000000001e-05 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565297+0000", 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1032999999999999e-05 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578757+0000", 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013459891999999999 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.957 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578765+0000", 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4100000000000008e-06 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565630+0000", 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98686430599999997 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565641+0000", 2024-03-26T03:21:47.958 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1275e-05 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:124424 10.3 10.9f3a05a3 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e331)", 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:14:20.565322+0000", 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 447.38600874600002, 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0003774160000001, 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "reached pg", 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.959 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 124424 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565322+0000", 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565322+0000", 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.960 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565323+0000", 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0100000000000003e-07 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565373+0000", 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0773999999999999e-05 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565385+0000", 2024-03-26T03:21:47.961 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1430000000000001e-05 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.565392+0000", 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.52e-06 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578791+0000", 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.013398558 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.962 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:20.578797+0000", 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2029999999999997e-06 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565691+0000", 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.986893927 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:14:21.565699+0000", 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1030000000000002e-06 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.963 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198816 10.3 10:c18f3cf7:::benchmark_data_smithi066_148944_object74169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.912953+0000", 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 364.03837724800002, 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99607143899999995, 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.964 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198816 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912953+0000", 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912953+0000", 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.965 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912956+0000", 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.4339999999999999e-06 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912961+0000", 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7570000000000001e-06 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912975+0000", 2024-03-26T03:21:47.966 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4042e-05 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.912983+0000", 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3769999999999998e-06 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.913057+0000", 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.3811000000000002e-05 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.913066+0000", 2024-03-26T03:21:47.967 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.5920000000000007e-06 2024-03-26T03:21:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.968 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.895282+0000", 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98221624799999996 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.895410+0000", 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012716499999999999 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.895706+0000", 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00029644900000000001 2024-03-26T03:21:47.969 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904010+0000", 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0083038139999999996 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904057+0000", 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7370999999999998e-05 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.908933+0000", 2024-03-26T03:21:47.970 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004875379 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.908988+0000", 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5380000000000002e-05 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909025+0000", 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6619999999999998e-05 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.971 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198822 10.3 10:c95efd72:::benchmark_data_smithi066_148944_object74175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.920307+0000", 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 364.03102295799999, 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98900118000000004, 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.972 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198822 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.973 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920307+0000", 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920307+0000", 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.974 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920309+0000", 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920314+0000", 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6290000000000001e-06 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920323+0000", 2024-03-26T03:21:47.975 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.8859999999999993e-06 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920329+0000", 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.4320000000000004e-06 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920387+0000", 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7651000000000001e-05 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.976 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.920395+0000", 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3510000000000002e-06 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904091+0000", 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.983696126 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904231+0000", 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013951499999999999 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.977 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.904352+0000", 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012101 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.908917+0000", 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0045653120000000002 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909121+0000", 2024-03-26T03:21:47.978 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00020395100000000001 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909200+0000", 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.9015000000000002e-05 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909259+0000", 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8956000000000003e-05 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:47.979 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909309+0000", 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9746000000000001e-05 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198826 10.3 10:cbb6a365:::benchmark_data_smithi066_148944_object74179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.922987+0000", 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 364.02834337299998, 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98656156500000003, 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:47.980 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198826 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922987+0000", 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922987+0000", 2024-03-26T03:21:47.981 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922989+0000", 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6270000000000001e-06 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.922994+0000", 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7690000000000002e-06 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.982 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923004+0000", 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9180000000000006e-06 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923012+0000", 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.7510000000000008e-06 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923946+0000", 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00093349999999999998 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.983 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.923956+0000", 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0709e-05 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.905833+0000", 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.981876741 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.905933+0000", 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9852999999999994e-05 2024-03-26T03:21:47.984 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.985 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.906046+0000", 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000112899 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909072+0000", 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0030257830000000002 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909442+0000", 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00037073599999999998 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909458+0000", 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5773999999999998e-05 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909506+0000", 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7905000000000003e-05 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909549+0000", 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.2599999999999999e-05 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.988 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198829 10.3 10:cb612e94:::benchmark_data_smithi066_148944_object74182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.924268+0000", 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 364.02706197100002, 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.99669433600000001, 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198829 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.989 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924268+0000", 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924268+0000", 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.990 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924270+0000", 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.176e-06 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924284+0000", 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4351e-05 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924292+0000", 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4840000000000004e-06 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.991 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.924299+0000", 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.0480000000000001e-06 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.927171+0000", 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028710929999999999 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.927180+0000", 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1689999999999998e-06 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.992 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.907599+0000", 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98041955000000003 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.907696+0000", 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.6552000000000007e-05 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.907762+0000", 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.6438000000000003e-05 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.993 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.912805+0000", 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0050422770000000004 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.912851+0000", 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6381999999999997e-05 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.994 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920853+0000", 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0080025549999999997 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920913+0000", 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9673999999999998e-05 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.920963+0000", 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9586999999999998e-05 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.995 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198856 10.3 10:ca535049:::benchmark_data_smithi066_148944_object74209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.951098+0000", 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 364.00023278399999, 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97358442499999998, 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:47.996 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198856 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951098+0000", 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951098+0000", 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:47.997 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951099+0000", 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9120000000000001e-06 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951105+0000", 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8359999999999997e-06 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:47.998 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951120+0000", 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4851000000000001e-05 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951128+0000", 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8420000000000005e-06 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951192+0000", 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3709999999999998e-05 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:47.999 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.951201+0000", 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7070000000000004e-06 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909662+0000", 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.95846033399999997 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909784+0000", 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012201699999999999 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:48.000 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.909932+0000", 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014834599999999999 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.915982+0000", 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0060503120000000004 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923190+0000", 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0072076320000000003 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.001 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924590+0000", 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0013998330000000001 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924639+0000", 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8955000000000001e-05 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924682+0000", 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.3137999999999999e-05 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.002 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198874 10.3 10:c46ca9df:::benchmark_data_smithi066_148944_object74227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.964061+0000", 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 363.98726916499999, 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96077497499999998, 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:48.003 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198874 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964061+0000", 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964061+0000", 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.004 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964063+0000", 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.125e-06 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964070+0000", 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7950000000000002e-06 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964083+0000", 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3063e-05 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.005 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964092+0000", 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0620000000000007e-06 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964171+0000", 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.8604999999999997e-05 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.964181+0000", 2024-03-26T03:21:48.006 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.9119999999999993e-06 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.911186+0000", 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.94700484399999996 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.911293+0000", 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000107865 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:48.007 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.911405+0000", 2024-03-26T03:21:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011191 2024-03-26T03:21:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.008 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923197+0000", 2024-03-26T03:21:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.011791585 2024-03-26T03:21:48.009 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924720+0000", 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001523429 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924758+0000", 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7191999999999999e-05 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.010 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924802+0000", 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4503999999999999e-05 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924836+0000", 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.4084000000000001e-05 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198895 10.3 10:c9006454:::benchmark_data_smithi066_148944_object74248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:43.990762+0000", 2024-03-26T03:21:48.011 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 363.96056785799999, 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93426770699999995, 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198895 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990762+0000", 2024-03-26T03:21:48.012 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990762+0000", 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990765+0000", 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.1449999999999998e-06 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990770+0000", 2024-03-26T03:21:48.013 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.8359999999999997e-06 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990784+0000", 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3998e-05 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990793+0000", 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3909999999999998e-06 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.014 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990853+0000", 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0543e-05 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:43.990862+0000", 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0089999999999996e-06 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.912898+0000", 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.922035194 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.015 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.913027+0000", 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00012908 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.913163+0000", 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013621799999999999 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923202+0000", 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.010039078 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.016 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924872+0000", 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001669554 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924947+0000", 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5858000000000005e-05 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.924993+0000", 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.5778000000000001e-05 2024-03-26T03:21:48.017 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925030+0000", 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7024999999999999e-05 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.25522.0:198918 10.3 10:cc0a2071:::benchmark_data_smithi066_148944_object74271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:15:44.009004+0000", 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 363.94232616800002, 2024-03-26T03:21:48.018 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91617897400000003, 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.25522", 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1205634018", 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 198918 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009004+0000", 2024-03-26T03:21:48.019 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009004+0000", 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009006+0000", 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2510000000000001e-06 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.020 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009013+0000", 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.041e-06 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009028+0000", 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4673e-05 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009038+0000", 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0336999999999999e-05 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.021 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009098+0000", 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.9861999999999997e-05 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.009109+0000", 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0465000000000001e-05 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.914465+0000", 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.90535576699999998 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.022 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.914578+0000", 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000113625 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.914697+0000", 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000118663 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.923207+0000", 2024-03-26T03:21:48.023 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0085102230000000008 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925068+0000", 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018610460000000001 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925106+0000", 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.7373999999999998e-05 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925147+0000", 2024-03-26T03:21:48.024 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.1285000000000002e-05 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:15:44.925183+0000", 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6362000000000002e-05 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83059 11.b 11:d81c22b5:::benchmark_data_smithi066_175179_object83058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.591733+0000", 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 232.35959716799999, 2024-03-26T03:21:48.025 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.100162675, 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83059 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.026 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591733+0000", 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591733+0000", 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591736+0000", 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.469e-06 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.027 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591764+0000", 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.7869999999999999e-05 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591782+0000", 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.8533999999999999e-05 2024-03-26T03:21:48.028 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 6.6 scrub starts 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 6.6 scrub ok 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 5.e scrub starts 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 5.e scrub ok 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: pgmap v1083: 162 pgs: 162 active+clean; 635 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 6.0 KiB/s rd, 36 KiB/s wr, 7 op/s; 26 KiB/s, 0 objects/s recovering 2024-03-26T03:21:48.029 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 1.10 scrub starts 2024-03-26T03:21:48.030 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 1.10 scrub ok 2024-03-26T03:21:48.030 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 2.1d scrub starts 2024-03-26T03:21:48.030 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:47 smithi192 ceph-mon[33110]: 2.1d scrub ok 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591791+0000", 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3810000000000005e-06 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591900+0000", 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000108557 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.031 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.591914+0000", 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.3818000000000001e-05 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563758+0000", 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97184428 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563771+0000", 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2422999999999999e-05 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.032 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.682417+0000", 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.118645989 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.682572+0000", 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000155542 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.682738+0000", 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016616600000000001 2024-03-26T03:21:48.033 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684827+0000", 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0020892520000000002 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684877+0000", 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9419999999999998e-05 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.034 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.691623+0000", 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0067459060000000003 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.691690+0000", 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7318999999999997e-05 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.691754+0000", 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.3808999999999998e-05 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.035 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.691896+0000", 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014194000000000001 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83072 11.b 11:d3fb2af7:::benchmark_data_smithi066_175179_object83071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.598936+0000", 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 232.35239415999999, 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0933250760000002, 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.036 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83072 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.598936+0000", 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.598936+0000", 2024-03-26T03:21:48.037 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.598938+0000", 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.615e-06 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.598962+0000", 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.3703999999999999e-05 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.038 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.598971+0000", 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3850000000000008e-06 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.598978+0000", 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.1319999999999996e-06 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.599097+0000", 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000119014 2024-03-26T03:21:48.039 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.599109+0000", 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1745e-05 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563798+0000", 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96468940000000003 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563807+0000", 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.0040000000000005e-06 2024-03-26T03:21:48.040 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.684946+0000", 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.121138803 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.685096+0000", 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015007700000000001 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:48.041 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.685259+0000", 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00016279599999999999 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.691630+0000", 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0063707160000000002 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.691957+0000", 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00032691300000000002 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.042 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692004+0000", 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7472999999999998e-05 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692059+0000", 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.5068000000000002e-05 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692121+0000", 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2323999999999995e-05 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.043 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692261+0000", 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00013990699999999999 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83121 11.b 11:d6d044ed:::benchmark_data_smithi066_175179_object83120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.633614+0000", 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 232.31771621799999, 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0805044389999998, 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.044 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83121 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633614+0000", 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.045 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633614+0000", 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633616+0000", 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.804e-06 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633639+0000", 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2952999999999999e-05 2024-03-26T03:21:48.046 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633646+0000", 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.593e-06 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633652+0000", 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2719999999999997e-06 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633720+0000", 2024-03-26T03:21:48.047 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7858e-05 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.633732+0000", 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2089000000000001e-05 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563834+0000", 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.93010253700000001 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.048 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563843+0000", 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3639999999999999e-06 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.688818+0000", 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1249756870000001 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.688964+0000", 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00014606800000000001 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.049 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.689123+0000", 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015891899999999999 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692322+0000", 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0031991989999999998 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692370+0000", 2024-03-26T03:21:48.050 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.761e-05 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.696880+0000", 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0045099320000000004 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.713954+0000", 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.017074431000000001 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.051 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714006+0000", 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1153000000000003e-05 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714119+0000", 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011297 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83144 11.b 11:d5e8c222:::benchmark_data_smithi066_175179_object83143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:48.052 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.648708+0000", 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 232.302622794, 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0656799220000002, 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83144 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.053 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648708+0000", 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648708+0000", 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648709+0000", 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.5740000000000001e-06 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.054 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648732+0000", 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.2745e-05 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648746+0000", 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.4012999999999999e-05 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648754+0000", 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.6109999999999996e-06 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.055 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648814+0000", 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.0900000000000003e-05 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.648824+0000", 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.7280000000000007e-06 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563869+0000", 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.915045357 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.056 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563878+0000", 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2970000000000004e-06 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692405+0000", 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.128526929 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692731+0000", 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00032665100000000001 2024-03-26T03:21:48.057 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.692947+0000", 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00021541099999999999 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.696953+0000", 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0040062609999999997 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.058 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714128+0000", 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.017174989000000002 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714178+0000", 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.9611999999999999e-05 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714216+0000", 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.8195000000000003e-05 2024-03-26T03:21:48.059 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714266+0000", 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0501999999999999e-05 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714387+0000", 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000121147 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.060 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:83148 11.b 11:d536f4b7:::benchmark_data_smithi066_175179_object83147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e427)", 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:17:55.651536+0000", 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 232.299794042, 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0631130070000001, 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 83148 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.061 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651536+0000", 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651536+0000", 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651538+0000", 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.6160000000000001e-06 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.062 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651558+0000", 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.9647e-05 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651568+0000", 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.0512e-05 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651576+0000", 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2560000000000002e-06 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.063 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651628+0000", 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1270999999999998e-05 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:55.651634+0000", 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.849e-06 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563919+0000", 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91228417399999995 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.064 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:56.563927+0000", 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.1769999999999998e-06 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694681+0000", 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.1307544110000001 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694828+0000", 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000146508 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.065 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.694986+0000", 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015786199999999999 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.713012+0000", 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.018026872999999999 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714134+0000", 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0011217670000000001 2024-03-26T03:21:48.066 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714427+0000", 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00029274200000000002 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714478+0000", 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0643000000000003e-05 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714535+0000", 2024-03-26T03:21:48.067 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.7052000000000002e-05 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:17:57.714649+0000", 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000114647 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:138171 11.b 11:dc5b46d7:::benchmark_data_smithi066_175179_object6273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:18:43.540270+0000", 2024-03-26T03:21:48.068 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 184.41106033099999, 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.98281798099999995, 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138171 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540270+0000", 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.069 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540270+0000", 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540271+0000", 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.5099999999999998e-07 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540275+0000", 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.6500000000000002e-06 2024-03-26T03:21:48.070 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540281+0000", 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.657e-06 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540288+0000", 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7259999999999996e-06 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.071 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540350+0000", 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.2324999999999997e-05 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.540359+0000", 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.1419999999999999e-06 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.518767+0000", 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97840737 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.072 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.518920+0000", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00015317999999999999 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.519096+0000", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00017602300000000001 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.522921+0000", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0038250160000000001 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.073 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.522930+0000", 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.312e-06 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.522981+0000", 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.0537000000000001e-05 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523040+0000", 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.914e-05 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.074 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523088+0000", 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8052e-05 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:138178 11.b 11:db4398f8:::benchmark_data_smithi066_175179_object6280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:18:43.544977+0000", 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 184.40635358599999, 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.979799329, 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.075 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138178 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.544977+0000", 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.544977+0000", 2024-03-26T03:21:48.076 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.544977+0000", 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.4400000000000001e-07 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.544980+0000", 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.52e-06 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.544985+0000", 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.417e-06 2024-03-26T03:21:48.077 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.544990+0000", 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.327e-06 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.546151+0000", 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001160634 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.546159+0000", 2024-03-26T03:21:48.078 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.2360000000000004e-06 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.521141+0000", 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.97498122799999998 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.521257+0000", 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00011692 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:21:48.079 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.521341+0000", 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.3829e-05 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523137+0000", 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0017959880000000001 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523241+0000", 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010390700000000001 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.080 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.524665+0000", 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.001423852 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.524721+0000", 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.6159999999999998e-05 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.524776+0000", 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4867000000000002e-05 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.081 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:138201 11.b 11:dee41134:::benchmark_data_smithi066_175179_object6303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:18:43.561260+0000", 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 184.390070012, 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96995721700000004, 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.082 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138201 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561260+0000", 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561260+0000", 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.083 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561262+0000", 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 2.0109999999999999e-06 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561267+0000", 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1340000000000003e-06 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561279+0000", 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.119e-05 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.084 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561287+0000", 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0199999999999994e-06 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561339+0000", 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2064000000000002e-05 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.561346+0000", 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5630000000000003e-06 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.085 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523280+0000", 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.96193375199999998 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523397+0000", 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117042 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.523514+0000", 2024-03-26T03:21:48.086 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000117333 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.526397+0000", 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0028828980000000001 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.526431+0000", 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.3368000000000003e-05 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531072+0000", 2024-03-26T03:21:48.087 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.004641236 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531171+0000", 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.8554999999999994e-05 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531218+0000", 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.7051000000000001e-05 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.088 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:138230 11.b 11:dee2b3bc:::benchmark_data_smithi066_175179_object6332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:18:43.613529+0000", 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 184.337801332, 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.917847089, 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138230 2024-03-26T03:21:48.089 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613529+0000", 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613529+0000", 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.090 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613531+0000", 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.508e-06 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613536+0000", 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.4389999999999999e-06 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613547+0000", 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.078e-05 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.091 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613554+0000", 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 7.5399999999999998e-06 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613917+0000", 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00036229200000000001 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.613926+0000", 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.9570000000000008e-06 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.092 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.524812+0000", 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91088632300000005 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.524892+0000", 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.0207e-05 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.524984+0000", 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.2303e-05 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.093 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.529447+0000", 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0044629320000000002 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.529465+0000", 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.7218999999999999e-05 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531285+0000", 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0018201739999999999 2024-03-26T03:21:48.094 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531336+0000", 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.1765999999999999e-05 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531376+0000", 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 3.9648999999999999e-05 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "description": "osd_op(client.15978.0:138240 11.b 11:d39c0177:::benchmark_data_smithi066_175179_object6342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-03-26T03:21:48.095 INFO:teuthology.orchestra.run.smithi066.stdout: "initiated_at": "2024-03-26T03:18:43.616761+0000", 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "age": 184.33456930400001, 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.91483719600000002, 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "continuous": false, 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "type_data": { 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "client_info": { 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "client": "client.15978", 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "client_addr": "v1:172.21.15.66:0/1372365207", 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "tid": 138240 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "events": [ 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "initiated", 2024-03-26T03:21:48.096 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616761+0000", 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "header_read", 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616761+0000", 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "throttled", 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616762+0000", 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.288e-06 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "all_read", 2024-03-26T03:21:48.097 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616767+0000", 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8659999999999997e-06 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "dispatched", 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616777+0000", 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 9.3260000000000002e-06 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "queued_for_pg", 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616783+0000", 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 6.7739999999999999e-06 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.098 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616836+0000", 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 5.2414999999999999e-05 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for peered", 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:43.616844+0000", 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 8.4579999999999993e-06 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "reached_pg", 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.526472+0000", 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.90962739000000004 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.099 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "started", 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.526579+0000", 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.00010783 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.526691+0000", 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.000111488 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.529505+0000", 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.002814029 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.100 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "op_commit", 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.529517+0000", 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 1.2104e-05 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531503+0000", 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 0.0019859840000000001 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "commit_sent", 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531552+0000", 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.8887999999999997e-05 2024-03-26T03:21:48.101 INFO:teuthology.orchestra.run.smithi066.stdout: }, 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: { 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "event": "done", 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "time": "2024-03-26T03:18:44.531598+0000", 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: "duration": 4.6356000000000001e-05 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: } 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout: ] 2024-03-26T03:21:48.102 INFO:teuthology.orchestra.run.smithi066.stdout:} 2024-03-26T03:21:48.577 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T03:21:48.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: 2.1b scrub starts 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: 2.1b scrub ok 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3350266244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: 1.1b deep-scrub starts 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: 1.1b deep-scrub ok 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: 6.4 scrub starts 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[29666]: 6.4 scrub ok 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: 2.1b scrub starts 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: 2.1b scrub ok 2024-03-26T03:21:48.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3350266244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:48.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:48.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: 1.1b deep-scrub starts 2024-03-26T03:21:48.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: 1.1b deep-scrub ok 2024-03-26T03:21:48.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: 6.4 scrub starts 2024-03-26T03:21:48.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:48 smithi066 ceph-mon[37920]: 6.4 scrub ok 2024-03-26T03:21:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: 2.1b scrub starts 2024-03-26T03:21:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: 2.1b scrub ok 2024-03-26T03:21:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3350266244' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:48.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: 1.1b deep-scrub starts 2024-03-26T03:21:48.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: 1.1b deep-scrub ok 2024-03-26T03:21:48.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: 6.4 scrub starts 2024-03-26T03:21:48.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:48 smithi192 ceph-mon[33110]: 6.4 scrub ok 2024-03-26T03:21:49.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-26T03:21:49.855 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:49.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: 6.18 scrub starts 2024-03-26T03:21:49.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: 6.18 scrub ok 2024-03-26T03:21:49.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: 2.1 scrub starts 2024-03-26T03:21:49.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: 2.1 scrub ok 2024-03-26T03:21:49.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: pgmap v1084: 162 pgs: 162 active+clean; 635 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 31 KiB/s wr, 5 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: 5.4 deep-scrub starts 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[37920]: 5.4 deep-scrub ok 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: 6.18 scrub starts 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: 6.18 scrub ok 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: 2.1 scrub starts 2024-03-26T03:21:49.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: 2.1 scrub ok 2024-03-26T03:21:49.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: pgmap v1084: 162 pgs: 162 active+clean; 635 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 31 KiB/s wr, 5 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-26T03:21:49.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:21:49.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T03:21:49.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: 5.4 deep-scrub starts 2024-03-26T03:21:49.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:49 smithi066 ceph-mon[29666]: 5.4 deep-scrub ok 2024-03-26T03:21:49.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: 6.18 scrub starts 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: 6.18 scrub ok 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: 2.1 scrub starts 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: 2.1 scrub ok 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: pgmap v1084: 162 pgs: 162 active+clean; 635 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 4.5 KiB/s rd, 31 KiB/s wr, 5 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: 5.4 deep-scrub starts 2024-03-26T03:21:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:49 smithi192 ceph-mon[33110]: 5.4 deep-scrub ok 2024-03-26T03:21:50.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:21:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:21:50.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:50 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:21:50.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:50 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:21:50.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:50 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:21:50.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:50 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:21:50.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:50 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:21:51.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:51.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: 1.c scrub starts 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: 1.c scrub ok 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: pgmap v1086: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 643 B/s rd, 0 op/s 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: 5.16 deep-scrub starts 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: 5.16 deep-scrub ok 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: 5.9 scrub starts 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[37920]: 5.9 scrub ok 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: 1.c scrub starts 2024-03-26T03:21:51.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: 1.c scrub ok 2024-03-26T03:21:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: pgmap v1086: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 643 B/s rd, 0 op/s 2024-03-26T03:21:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T03:21:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: 5.16 deep-scrub starts 2024-03-26T03:21:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: 5.16 deep-scrub ok 2024-03-26T03:21:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: 5.9 scrub starts 2024-03-26T03:21:51.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:51 smithi066 ceph-mon[29666]: 5.9 scrub ok 2024-03-26T03:21:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: 1.c scrub starts 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: 1.c scrub ok 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: pgmap v1086: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 643 B/s rd, 0 op/s 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: 5.16 deep-scrub starts 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: 5.16 deep-scrub ok 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: 5.9 scrub starts 2024-03-26T03:21:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:51 smithi192 ceph-mon[33110]: 5.9 scrub ok 2024-03-26T03:21:52.593 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-26T03:21:52.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3734294264' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: 2.9 scrub starts 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[29666]: 2.9 scrub ok 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:21:52.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:21:52.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T03:21:52.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3734294264' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:52.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: 2.9 scrub starts 2024-03-26T03:21:52.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:52 smithi066 ceph-mon[37920]: 2.9 scrub ok 2024-03-26T03:21:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:21:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:21:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:21:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:21:52.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T03:21:52.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3734294264' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T03:21:52.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: 2.9 scrub starts 2024-03-26T03:21:52.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:52 smithi192 ceph-mon[33110]: 2.9 scrub ok 2024-03-26T03:21:53.295 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T03:21:53.559 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:53.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:21:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: 4.10 scrub starts 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: 4.10 scrub ok 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: pgmap v1089: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3734294264' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: 4.10 scrub starts 2024-03-26T03:21:53.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: 4.10 scrub ok 2024-03-26T03:21:53.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: pgmap v1089: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-03-26T03:21:53.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3734294264' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:21:53.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T03:21:53.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:21:53.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:53 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:21:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: 4.10 scrub starts 2024-03-26T03:21:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: 4.10 scrub ok 2024-03-26T03:21:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: pgmap v1089: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-03-26T03:21:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3734294264' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T03:21:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T03:21:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:21:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:53 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:21:54.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[37920]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[37920]: 6.a scrub starts 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[37920]: 6.a scrub ok 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[37920]: 1.2 scrub starts 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[37920]: 1.2 scrub ok 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[29666]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[29666]: 6.a scrub starts 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[29666]: 6.a scrub ok 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[29666]: 1.2 scrub starts 2024-03-26T03:21:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:54 smithi066 ceph-mon[29666]: 1.2 scrub ok 2024-03-26T03:21:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:54 smithi192 ceph-mon[33110]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T03:21:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:54 smithi192 ceph-mon[33110]: 6.a scrub starts 2024-03-26T03:21:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:54 smithi192 ceph-mon[33110]: 6.a scrub ok 2024-03-26T03:21:54.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:54 smithi192 ceph-mon[33110]: 1.2 scrub starts 2024-03-26T03:21:54.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:54 smithi192 ceph-mon[33110]: 1.2 scrub ok 2024-03-26T03:21:55.611 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T03:21:55.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:55 smithi192 ceph-mon[33110]: pgmap v1092: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:55.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:55 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4126354404' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:21:56.076 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:55 smithi066 ceph-mon[29666]: pgmap v1092: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:56.076 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:56.077 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:55 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4126354404' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:21:56.077 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:55 smithi066 ceph-mon[37920]: pgmap v1092: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:21:56.077 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:21:56.077 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:55 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4126354404' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:21:56.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-26T03:21:56.701 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:21:56.823 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[29666]: 6.10 scrub starts 2024-03-26T03:21:56.823 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[29666]: 6.10 scrub ok 2024-03-26T03:21:56.823 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/4126354404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[29666]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[29666]: 6.1e scrub starts 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[29666]: 6.1e scrub ok 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[37920]: 6.10 scrub starts 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[37920]: 6.10 scrub ok 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/4126354404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[37920]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[37920]: 6.1e scrub starts 2024-03-26T03:21:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:56 smithi066 ceph-mon[37920]: 6.1e scrub ok 2024-03-26T03:21:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:56 smithi192 ceph-mon[33110]: 6.10 scrub starts 2024-03-26T03:21:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:56 smithi192 ceph-mon[33110]: 6.10 scrub ok 2024-03-26T03:21:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:56 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/4126354404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:21:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:56 smithi192 ceph-mon[33110]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T03:21:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:56 smithi192 ceph-mon[33110]: 6.1e scrub starts 2024-03-26T03:21:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:56 smithi192 ceph-mon[33110]: 6.1e scrub ok 2024-03-26T03:21:57.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:57 smithi066 ceph-mon[37920]: pgmap v1094: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:21:57.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:57 smithi066 ceph-mon[29666]: pgmap v1094: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:21:57.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:57 smithi192 ceph-mon[33110]: pgmap v1094: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:21:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:58 smithi192 ceph-mon[33110]: 1.15 scrub starts 2024-03-26T03:21:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:58 smithi192 ceph-mon[33110]: 1.15 scrub ok 2024-03-26T03:21:59.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:58 smithi066 ceph-mon[29666]: 1.15 scrub starts 2024-03-26T03:21:59.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:58 smithi066 ceph-mon[29666]: 1.15 scrub ok 2024-03-26T03:21:59.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:58 smithi066 ceph-mon[37920]: 1.15 scrub starts 2024-03-26T03:21:59.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:58 smithi066 ceph-mon[37920]: 1.15 scrub ok 2024-03-26T03:21:59.641 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-26T03:21:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:59 smithi192 ceph-mon[33110]: pgmap v1095: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:21:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:59 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3924771537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:21:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:59 smithi192 ceph-mon[33110]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:21:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:59 smithi192 ceph-mon[33110]: 2.d scrub starts 2024-03-26T03:21:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:21:59 smithi192 ceph-mon[33110]: 2.d scrub ok 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[29666]: pgmap v1095: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3924771537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[29666]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[29666]: 2.d scrub starts 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[29666]: 2.d scrub ok 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[37920]: pgmap v1095: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3924771537' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:22:00.054 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[37920]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:22:00.055 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[37920]: 2.d scrub starts 2024-03-26T03:22:00.055 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:21:59 smithi066 ceph-mon[37920]: 2.d scrub ok 2024-03-26T03:22:00.464 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T03:22:00.643 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:22:00.734 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: 4.b scrub starts 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: 4.b scrub ok 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: 4.4 scrub starts 2024-03-26T03:22:00.857 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: 4.4 scrub ok 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[37920]: pgmap v1097: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: 4.b scrub starts 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: 4.b scrub ok 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:22:00.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: 4.4 scrub starts 2024-03-26T03:22:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: 4.4 scrub ok 2024-03-26T03:22:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:22:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:00.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:00 smithi066 ceph-mon[29666]: pgmap v1097: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-03-26T03:22:00.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: 4.b scrub starts 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: 4.b scrub ok 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: 4.4 scrub starts 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: 4.4 scrub ok 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:00.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:00 smithi192 ceph-mon[33110]: pgmap v1097: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-03-26T03:22:01.926 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[29666]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T03:22:01.927 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:22:01.927 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:22:01.927 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:22:01.927 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[37920]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T03:22:01.927 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:22:01.927 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:22:01.928 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:22:01.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:01 smithi192 ceph-mon[33110]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T03:22:01.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:22:01.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:22:01.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:22:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:02 smithi192 ceph-mon[33110]: 5.3 scrub starts 2024-03-26T03:22:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:02 smithi192 ceph-mon[33110]: 5.3 scrub ok 2024-03-26T03:22:02.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:02 smithi192 ceph-mon[33110]: pgmap v1099: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:22:03.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:02 smithi066 ceph-mon[29666]: 5.3 scrub starts 2024-03-26T03:22:03.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:02 smithi066 ceph-mon[29666]: 5.3 scrub ok 2024-03-26T03:22:03.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:02 smithi066 ceph-mon[29666]: pgmap v1099: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:22:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:02 smithi066 ceph-mon[37920]: 5.3 scrub starts 2024-03-26T03:22:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:02 smithi066 ceph-mon[37920]: 5.3 scrub ok 2024-03-26T03:22:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:02 smithi066 ceph-mon[37920]: pgmap v1099: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:22:03.666 INFO:teuthology.orchestra.run.smithi066.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T03:22:03.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:03 smithi192 ceph-mon[33110]: 5.11 deep-scrub starts 2024-03-26T03:22:03.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:03 smithi192 ceph-mon[33110]: 5.11 deep-scrub ok 2024-03-26T03:22:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:03 smithi192 ceph-mon[33110]: 1.19 scrub starts 2024-03-26T03:22:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:03 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/896821963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:22:03.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:03 smithi192 ceph-mon[33110]: 1.19 scrub ok 2024-03-26T03:22:04.017 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[29666]: 5.11 deep-scrub starts 2024-03-26T03:22:04.017 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[29666]: 5.11 deep-scrub ok 2024-03-26T03:22:04.017 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[29666]: 1.19 scrub starts 2024-03-26T03:22:04.017 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/896821963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:22:04.017 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[29666]: 1.19 scrub ok 2024-03-26T03:22:04.018 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:22:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:22:04.018 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[37920]: 5.11 deep-scrub starts 2024-03-26T03:22:04.018 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[37920]: 5.11 deep-scrub ok 2024-03-26T03:22:04.018 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[37920]: 1.19 scrub starts 2024-03-26T03:22:04.018 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/896821963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:22:04.018 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:03 smithi066 ceph-mon[37920]: 1.19 scrub ok 2024-03-26T03:22:04.409 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-26T03:22:04.670 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:04.965 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:04 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/896821963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:22:04.965 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:04 smithi066 ceph-mon[29666]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T03:22:04.965 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:04 smithi066 ceph-mon[29666]: pgmap v1101: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:04.966 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:04 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/896821963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:22:04.966 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:04 smithi066 ceph-mon[37920]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T03:22:04.966 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:04 smithi066 ceph-mon[37920]: pgmap v1101: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:04 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/896821963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:22:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:04 smithi192 ceph-mon[33110]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T03:22:04.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:04 smithi192 ceph-mon[33110]: pgmap v1101: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:06.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[29666]: 5.12 scrub starts 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[29666]: 5.12 scrub ok 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[29666]: 5.8 deep-scrub starts 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[29666]: 5.8 deep-scrub ok 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[29666]: 5.5 scrub starts 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[29666]: 5.5 scrub ok 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[37920]: 5.12 scrub starts 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[37920]: 5.12 scrub ok 2024-03-26T03:22:06.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[37920]: 5.8 deep-scrub starts 2024-03-26T03:22:06.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[37920]: 5.8 deep-scrub ok 2024-03-26T03:22:06.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[37920]: 5.5 scrub starts 2024-03-26T03:22:06.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:06 smithi066 ceph-mon[37920]: 5.5 scrub ok 2024-03-26T03:22:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:06 smithi192 ceph-mon[33110]: 5.12 scrub starts 2024-03-26T03:22:06.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:06 smithi192 ceph-mon[33110]: 5.12 scrub ok 2024-03-26T03:22:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:06 smithi192 ceph-mon[33110]: 5.8 deep-scrub starts 2024-03-26T03:22:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:06 smithi192 ceph-mon[33110]: 5.8 deep-scrub ok 2024-03-26T03:22:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:06 smithi192 ceph-mon[33110]: 5.5 scrub starts 2024-03-26T03:22:06.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:06 smithi192 ceph-mon[33110]: 5.5 scrub ok 2024-03-26T03:22:07.304 INFO:teuthology.orchestra.run.smithi066.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[29666]: 2.12 scrub starts 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[29666]: 2.12 scrub ok 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[29666]: pgmap v1102: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2738837784' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[37920]: 2.12 scrub starts 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[37920]: 2.12 scrub ok 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[37920]: pgmap v1102: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:22:07.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:07 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2738837784' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:22:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:07 smithi192 ceph-mon[33110]: 2.12 scrub starts 2024-03-26T03:22:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:07 smithi192 ceph-mon[33110]: 2.12 scrub ok 2024-03-26T03:22:07.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:07 smithi192 ceph-mon[33110]: pgmap v1102: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T03:22:07.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:07 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2738837784' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:22:08.013 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-26T03:22:08.014 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-26T03:22:08.014 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-26T03:22:08.014 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-26T03:22:08.014 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T03:22:08.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T03:22:08.280 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:08.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2738837784' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:22:08.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[29666]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T03:22:08.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[29666]: 1.1e deep-scrub starts 2024-03-26T03:22:08.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[29666]: 2.c scrub starts 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[29666]: 1.1e deep-scrub ok 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[29666]: 2.c scrub ok 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2738837784' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[37920]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[37920]: 1.1e deep-scrub starts 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[37920]: 2.c scrub starts 2024-03-26T03:22:08.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[37920]: 1.1e deep-scrub ok 2024-03-26T03:22:08.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:08 smithi066 ceph-mon[37920]: 2.c scrub ok 2024-03-26T03:22:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:08 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2738837784' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:22:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:08 smithi192 ceph-mon[33110]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T03:22:08.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:08 smithi192 ceph-mon[33110]: 1.1e deep-scrub starts 2024-03-26T03:22:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:08 smithi192 ceph-mon[33110]: 2.c scrub starts 2024-03-26T03:22:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:08 smithi192 ceph-mon[33110]: 1.1e deep-scrub ok 2024-03-26T03:22:08.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:08 smithi192 ceph-mon[33110]: 2.c scrub ok 2024-03-26T03:22:09.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[37920]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[37920]: pgmap v1105: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[37920]: 6.11 scrub starts 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[37920]: 6.11 scrub ok 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[29666]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T03:22:09.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[29666]: pgmap v1105: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:09.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[29666]: 6.11 scrub starts 2024-03-26T03:22:09.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:09 smithi066 ceph-mon[29666]: 6.11 scrub ok 2024-03-26T03:22:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:09 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:22:09.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:09 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:22:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:09 smithi192 ceph-mon[33110]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T03:22:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:09 smithi192 ceph-mon[33110]: pgmap v1105: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:09 smithi192 ceph-mon[33110]: 6.11 scrub starts 2024-03-26T03:22:09.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:09 smithi192 ceph-mon[33110]: 6.11 scrub ok 2024-03-26T03:22:10.400 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:22:10.400 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":507,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","created":"2024-03-26T02:53:28.887014+0000","modified":"2024-03-26T03:22:08.309697+0000","last_up_change":"2024-03-26T03:14:20.518826+0000","last_in_change":"2024-03-26T03:20:56.311329+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:56:43.176262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"481","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"303","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.809999942779541,"score_stable":1.809999942779541,"optimal_score":1,"raw_score_acting":1.809999942779541,"raw_score_stable":1.809999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T02:58:55.077781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T02:58:55.118069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:58:56.987641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:58:58.940665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:59:01.006117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90c9bf50-0b39-4fd1-9f00-b02800f07bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":478,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6801","nonce":1028658447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6802","nonce":1028658447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6804","nonce":1028658447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6803","nonce":1028658447}]},"public_addr":"172.21.15.66:6801/1028658447","cluster_addr":"172.21.15.66:6802/1028658447","heartbeat_back_addr":"172.21.15.66:6804/1028658447","heartbeat_front_addr":"172.21.15.66:6803/1028658447","state":["exists","up"]},{"osd":1,"uuid":"b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":478,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6805","nonce":3170415936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6806","nonce":3170415936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6808","nonce":3170415936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6807","nonce":3170415936}]},"public_addr":"172.21.15.66:6805/3170415936","cluster_addr":"172.21.15.66:6806/3170415936","heartbeat_back_addr":"172.21.15.66:6808/3170415936","heartbeat_front_addr":"172.21.15.66:6807/3170415936","state":["exists","up"]},{"osd":2,"uuid":"76639ea3-00c7-4693-9eb2-3421f234283b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":494,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6809","nonce":2520642381}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6810","nonce":2520642381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6812","nonce":2520642381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6811","nonce":2520642381}]},"public_addr":"172.21.15.66:6809/2520642381","cluster_addr":"172.21.15.66:6810/2520642381","heartbeat_back_addr":"172.21.15.66:6812/2520642381","heartbeat_front_addr":"172.21.15.66:6811/2520642381","state":["exists","up"]},{"osd":3,"uuid":"2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":197,"up_from":331,"up_thru":479,"down_at":198,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6813","nonce":52636497}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6814","nonce":52636497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6816","nonce":52636497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6815","nonce":52636497}]},"public_addr":"172.21.15.66:6813/52636497","cluster_addr":"172.21.15.66:6814/52636497","heartbeat_back_addr":"172.21.15.66:6816/52636497","heartbeat_front_addr":"172.21.15.66:6815/52636497","state":["exists","up"]},{"osd":4,"uuid":"4fae794b-e89c-4931-9ed3-0cc32b445883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":1766416869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":1766416869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":1766416869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":1766416869}]},"public_addr":"172.21.15.192:6800/1766416869","cluster_addr":"172.21.15.192:6801/1766416869","heartbeat_back_addr":"172.21.15.192:6803/1766416869","heartbeat_front_addr":"172.21.15.192:6802/1766416869","state":["exists","up"]},{"osd":5,"uuid":"04b178c3-9be8-466c-822c-0acfca8a92b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":495,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":2546913262}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":2546913262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":2546913262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":2546913262}]},"public_addr":"172.21.15.192:6804/2546913262","cluster_addr":"172.21.15.192:6805/2546913262","heartbeat_back_addr":"172.21.15.192:6807/2546913262","heartbeat_front_addr":"172.21.15.192:6806/2546913262","state":["exists","up"]},{"osd":6,"uuid":"fc63b52a-e4fe-4396-bf54-2e38696aedcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":4039268292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":4039268292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":4039268292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":4039268292}]},"public_addr":"172.21.15.192:6808/4039268292","cluster_addr":"172.21.15.192:6809/4039268292","heartbeat_back_addr":"172.21.15.192:6811/4039268292","heartbeat_front_addr":"172.21.15.192:6810/4039268292","state":["exists","up"]},{"osd":7,"uuid":"e2f4cedd-43af-4261-9f86-c4b72c97bd1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":3100313121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":3100313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":3100313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":3100313121}]},"public_addr":"172.21.15.192:6812/3100313121","cluster_addr":"172.21.15.192:6813/3100313121","heartbeat_back_addr":"172.21.15.192:6815/3100313121","heartbeat_front_addr":"172.21.15.192:6814/3100313121","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:59.965945+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:08.415040+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:07.967527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:51.297267+0000","dead_epoch":197},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:22:07.868378+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:13.177882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:52.919066+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:49.633874+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.f","osds":[4,1,7]},{"pgid":"6.14","osds":[7,3,6]}],"pg_upmap_items":[{"pgid":"1.12","mappings":[{"from":5,"to":4}]},{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"5.5","mappings":[{"from":0,"to":3}]},{"pgid":"5.15","mappings":[{"from":0,"to":6}]},{"pgid":"5.1d","mappings":[{"from":1,"to":6}]},{"pgid":"6.19","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:6800/2473255846":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/3359773304":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/3105618869":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/1321389422":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/3068874001":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/2719981294":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/858010278":"2024-03-27T02:54:29.078830+0000","172.21.15.66:6800/2558488368":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/2101239378":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/2641107787":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/5961410":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/3720619020":"2024-03-27T02:54:01.878928+0000","172.21.15.66:6800/2059693071":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/952321413":"2024-03-27T02:54:29.078830+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T03:22:10.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:10 smithi066 ceph-mon[37920]: 4.5 scrub starts 2024-03-26T03:22:10.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:10 smithi066 ceph-mon[37920]: 4.5 scrub ok 2024-03-26T03:22:10.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:10 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:10 smithi066 ceph-mon[29666]: 4.5 scrub starts 2024-03-26T03:22:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:10 smithi066 ceph-mon[29666]: 4.5 scrub ok 2024-03-26T03:22:10.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:10 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:10.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:22:10 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:22:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:10 smithi192 ceph-mon[33110]: 4.5 scrub starts 2024-03-26T03:22:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:10 smithi192 ceph-mon[33110]: 4.5 scrub ok 2024-03-26T03:22:10.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:10 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:11.212 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T03:22:11.212 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T03:22:11.473 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: 4.f scrub starts 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: 4.f scrub ok 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: 6.0 scrub starts 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: 6.0 scrub ok 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1092844955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: pgmap v1106: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: 4.17 scrub starts 2024-03-26T03:22:11.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[37920]: 4.17 scrub ok 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: 4.f scrub starts 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: 4.f scrub ok 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: 6.0 scrub starts 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: 6.0 scrub ok 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1092844955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: pgmap v1106: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: 4.17 scrub starts 2024-03-26T03:22:11.596 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:11 smithi066 ceph-mon[29666]: 4.17 scrub ok 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: 4.f scrub starts 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: 4.f scrub ok 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: 6.0 scrub starts 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: 6.0 scrub ok 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1092844955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: pgmap v1106: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: 4.17 scrub starts 2024-03-26T03:22:11.639 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:11 smithi192 ceph-mon[33110]: 4.17 scrub ok 2024-03-26T03:22:12.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:12 smithi066 ceph-mon[37920]: 4.9 scrub starts 2024-03-26T03:22:12.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:12 smithi066 ceph-mon[37920]: 4.9 scrub ok 2024-03-26T03:22:12.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:12 smithi066 ceph-mon[29666]: 4.9 scrub starts 2024-03-26T03:22:12.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:12 smithi066 ceph-mon[29666]: 4.9 scrub ok 2024-03-26T03:22:12.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:12 smithi192 ceph-mon[33110]: 4.9 scrub starts 2024-03-26T03:22:12.654 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:12 smithi192 ceph-mon[33110]: 4.9 scrub ok 2024-03-26T03:22:13.455 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:22:13.455 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":507,"fsid":"eba7a548-eb1b-11ee-95d0-87774f69a715","created":"2024-03-26T02:53:28.887014+0000","modified":"2024-03-26T03:22:08.309697+0000","last_up_change":"2024-03-26T03:14:20.518826+0000","last_in_change":"2024-03-26T03:20:56.311329+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:56:43.176262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"481","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"303","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.809999942779541,"score_stable":1.809999942779541,"optimal_score":1,"raw_score_acting":1.809999942779541,"raw_score_stable":1.809999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-26T02:58:55.077781+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-26T02:58:55.118069+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:58:56.987641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:58:58.940665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:59:01.006117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"90c9bf50-0b39-4fd1-9f00-b02800f07bf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":478,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6801","nonce":1028658447}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6802","nonce":1028658447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6804","nonce":1028658447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6803","nonce":1028658447}]},"public_addr":"172.21.15.66:6801/1028658447","cluster_addr":"172.21.15.66:6802/1028658447","heartbeat_back_addr":"172.21.15.66:6804/1028658447","heartbeat_front_addr":"172.21.15.66:6803/1028658447","state":["exists","up"]},{"osd":1,"uuid":"b49b4dee-f7bc-4873-b8ec-f6c55b3d5fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":478,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6805","nonce":3170415936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6806","nonce":3170415936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6808","nonce":3170415936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6807","nonce":3170415936}]},"public_addr":"172.21.15.66:6805/3170415936","cluster_addr":"172.21.15.66:6806/3170415936","heartbeat_back_addr":"172.21.15.66:6808/3170415936","heartbeat_front_addr":"172.21.15.66:6807/3170415936","state":["exists","up"]},{"osd":2,"uuid":"76639ea3-00c7-4693-9eb2-3421f234283b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":494,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6809","nonce":2520642381}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6810","nonce":2520642381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6812","nonce":2520642381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6811","nonce":2520642381}]},"public_addr":"172.21.15.66:6809/2520642381","cluster_addr":"172.21.15.66:6810/2520642381","heartbeat_back_addr":"172.21.15.66:6812/2520642381","heartbeat_front_addr":"172.21.15.66:6811/2520642381","state":["exists","up"]},{"osd":3,"uuid":"2b99b4bf-ce0e-4e1d-9932-ddf00352b1f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":197,"up_from":331,"up_thru":479,"down_at":198,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6813","nonce":52636497}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6814","nonce":52636497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6816","nonce":52636497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.66:6815","nonce":52636497}]},"public_addr":"172.21.15.66:6813/52636497","cluster_addr":"172.21.15.66:6814/52636497","heartbeat_back_addr":"172.21.15.66:6816/52636497","heartbeat_front_addr":"172.21.15.66:6815/52636497","state":["exists","up"]},{"osd":4,"uuid":"4fae794b-e89c-4931-9ed3-0cc32b445883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":1766416869}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":1766416869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":1766416869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":1766416869}]},"public_addr":"172.21.15.192:6800/1766416869","cluster_addr":"172.21.15.192:6801/1766416869","heartbeat_back_addr":"172.21.15.192:6803/1766416869","heartbeat_front_addr":"172.21.15.192:6802/1766416869","state":["exists","up"]},{"osd":5,"uuid":"04b178c3-9be8-466c-822c-0acfca8a92b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":495,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":2546913262}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":2546913262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":2546913262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":2546913262}]},"public_addr":"172.21.15.192:6804/2546913262","cluster_addr":"172.21.15.192:6805/2546913262","heartbeat_back_addr":"172.21.15.192:6807/2546913262","heartbeat_front_addr":"172.21.15.192:6806/2546913262","state":["exists","up"]},{"osd":6,"uuid":"fc63b52a-e4fe-4396-bf54-2e38696aedcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":4039268292}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":4039268292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":4039268292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":4039268292}]},"public_addr":"172.21.15.192:6808/4039268292","cluster_addr":"172.21.15.192:6809/4039268292","heartbeat_back_addr":"172.21.15.192:6811/4039268292","heartbeat_front_addr":"172.21.15.192:6810/4039268292","state":["exists","up"]},{"osd":7,"uuid":"e2f4cedd-43af-4261-9f86-c4b72c97bd1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":3100313121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":3100313121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":3100313121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":3100313121}]},"public_addr":"172.21.15.192:6812/3100313121","cluster_addr":"172.21.15.192:6813/3100313121","heartbeat_back_addr":"172.21.15.192:6815/3100313121","heartbeat_front_addr":"172.21.15.192:6814/3100313121","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:59.965945+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:08.415040+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:07.967527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:51.297267+0000","dead_epoch":197},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:22:07.868378+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:13.177882+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:52.919066+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:21:49.633874+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.f","osds":[4,1,7]},{"pgid":"6.14","osds":[7,3,6]}],"pg_upmap_items":[{"pgid":"1.12","mappings":[{"from":5,"to":4}]},{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"5.5","mappings":[{"from":0,"to":3}]},{"pgid":"5.15","mappings":[{"from":0,"to":6}]},{"pgid":"5.1d","mappings":[{"from":1,"to":6}]},{"pgid":"6.19","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.66:6800/2473255846":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/3359773304":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/3105618869":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/1321389422":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/3068874001":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/2719981294":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/858010278":"2024-03-27T02:54:29.078830+0000","172.21.15.66:6800/2558488368":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/2101239378":"2024-03-27T02:59:37.893358+0000","172.21.15.66:0/2641107787":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/5961410":"2024-03-27T02:54:01.878928+0000","172.21.15.66:0/3720619020":"2024-03-27T02:54:01.878928+0000","172.21.15.66:6800/2059693071":"2024-03-27T02:54:29.078830+0000","172.21.15.66:0/952321413":"2024-03-27T02:54:29.078830+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T03:22:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[29666]: 2.b scrub starts 2024-03-26T03:22:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[29666]: 2.b scrub ok 2024-03-26T03:22:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[29666]: 1.6 scrub starts 2024-03-26T03:22:13.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[29666]: 1.6 scrub ok 2024-03-26T03:22:13.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[29666]: pgmap v1107: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[37920]: 2.b scrub starts 2024-03-26T03:22:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[37920]: 2.b scrub ok 2024-03-26T03:22:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[37920]: 1.6 scrub starts 2024-03-26T03:22:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[37920]: 1.6 scrub ok 2024-03-26T03:22:13.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-mon[37920]: pgmap v1107: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:13.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:13 smithi192 ceph-mon[33110]: 2.b scrub starts 2024-03-26T03:22:13.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:13 smithi192 ceph-mon[33110]: 2.b scrub ok 2024-03-26T03:22:13.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:13 smithi192 ceph-mon[33110]: 1.6 scrub starts 2024-03-26T03:22:13.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:13 smithi192 ceph-mon[33110]: 1.6 scrub ok 2024-03-26T03:22:13.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:13 smithi192 ceph-mon[33110]: pgmap v1107: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:14.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:22:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:22:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:22:14.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T03:22:14.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T03:22:14.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T03:22:14.257 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T03:22:14.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T03:22:14.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T03:22:14.258 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T03:22:14.259 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: 5.1 scrub starts 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: 5.1 scrub ok 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: 4.3 scrub starts 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: 4.3 scrub ok 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: 4.1f deep-scrub starts 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: 4.1f deep-scrub ok 2024-03-26T03:22:14.611 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2816399885' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: 5.1 scrub starts 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: 5.1 scrub ok 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: 4.3 scrub starts 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: 4.3 scrub ok 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: 4.1f deep-scrub starts 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: 4.1f deep-scrub ok 2024-03-26T03:22:14.612 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:14 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2816399885' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:22:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: 5.1 scrub starts 2024-03-26T03:22:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: 5.1 scrub ok 2024-03-26T03:22:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: 4.3 scrub starts 2024-03-26T03:22:14.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: 4.3 scrub ok 2024-03-26T03:22:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: 4.1f deep-scrub starts 2024-03-26T03:22:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: 4.1f deep-scrub ok 2024-03-26T03:22:14.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:14 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2816399885' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:22:14.828 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.828 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.829 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.830 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.830 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.830 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.859 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:14.899 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:15.594 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: 2.16 scrub starts 2024-03-26T03:22:15.594 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: 2.16 scrub ok 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: 1.7 scrub starts 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: 1.7 scrub ok 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: pgmap v1108: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: 2.16 scrub starts 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: 2.16 scrub ok 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: 1.7 scrub starts 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: 1.7 scrub ok 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: pgmap v1108: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:22:15.595 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:15 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:22:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: 2.16 scrub starts 2024-03-26T03:22:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: 2.16 scrub ok 2024-03-26T03:22:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: 1.7 scrub starts 2024-03-26T03:22:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: 1.7 scrub ok 2024-03-26T03:22:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: pgmap v1108: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:22:15.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:22:15.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:15 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:22:16.416 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: 5.19 scrub starts 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: 5.19 scrub ok 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: 1.9 scrub starts 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: 1.9 scrub ok 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: 5.1c scrub starts 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: 5.1c scrub ok 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[37920]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T03:22:16.417 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: 5.19 scrub starts 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: 5.19 scrub ok 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: 1.9 scrub starts 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: 1.9 scrub ok 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: 5.1c scrub starts 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: 5.1c scrub ok 2024-03-26T03:22:16.418 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:16 smithi066 ceph-mon[29666]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T03:22:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:22:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:22:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: 5.19 scrub starts 2024-03-26T03:22:16.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: 5.19 scrub ok 2024-03-26T03:22:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: 1.9 scrub starts 2024-03-26T03:22:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: 1.9 scrub ok 2024-03-26T03:22:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: 5.1c scrub starts 2024-03-26T03:22:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: 5.1c scrub ok 2024-03-26T03:22:16.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:16 smithi192 ceph-mon[33110]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T03:22:17.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:17 smithi066 ceph-mon[37920]: pgmap v1110: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:17.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:17 smithi066 ceph-mon[37920]: 4.1d scrub starts 2024-03-26T03:22:17.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:17 smithi066 ceph-mon[37920]: 4.1d scrub ok 2024-03-26T03:22:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:17 smithi066 ceph-mon[29666]: pgmap v1110: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:17 smithi066 ceph-mon[29666]: 4.1d scrub starts 2024-03-26T03:22:17.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:17 smithi066 ceph-mon[29666]: 4.1d scrub ok 2024-03-26T03:22:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:17 smithi192 ceph-mon[33110]: pgmap v1110: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:17.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:17 smithi192 ceph-mon[33110]: 4.1d scrub starts 2024-03-26T03:22:17.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:17 smithi192 ceph-mon[33110]: 4.1d scrub ok 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[29666]: 2.3 scrub starts 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[29666]: 2.3 scrub ok 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[29666]: 4.1 scrub starts 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[29666]: 4.1 scrub ok 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[37920]: 2.3 scrub starts 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[37920]: 2.3 scrub ok 2024-03-26T03:22:18.539 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[37920]: 4.1 scrub starts 2024-03-26T03:22:18.540 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:18 smithi066 ceph-mon[37920]: 4.1 scrub ok 2024-03-26T03:22:18.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:18 smithi192 ceph-mon[33110]: 2.3 scrub starts 2024-03-26T03:22:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:18 smithi192 ceph-mon[33110]: 2.3 scrub ok 2024-03-26T03:22:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:18 smithi192 ceph-mon[33110]: 4.1 scrub starts 2024-03-26T03:22:18.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:18 smithi192 ceph-mon[33110]: 4.1 scrub ok 2024-03-26T03:22:19.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:19 smithi066 ceph-mon[37920]: 4.c scrub starts 2024-03-26T03:22:19.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:19 smithi066 ceph-mon[37920]: 4.c scrub ok 2024-03-26T03:22:19.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:19 smithi066 ceph-mon[37920]: pgmap v1111: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:19 smithi066 ceph-mon[29666]: 4.c scrub starts 2024-03-26T03:22:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:19 smithi066 ceph-mon[29666]: 4.c scrub ok 2024-03-26T03:22:19.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:19 smithi066 ceph-mon[29666]: pgmap v1111: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:19 smithi192 ceph-mon[33110]: 4.c scrub starts 2024-03-26T03:22:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:19 smithi192 ceph-mon[33110]: 4.c scrub ok 2024-03-26T03:22:19.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:19 smithi192 ceph-mon[33110]: pgmap v1111: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:20.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:22:20 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:22:20.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:20 smithi192 ceph-mon[33110]: 5.c scrub starts 2024-03-26T03:22:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:20 smithi192 ceph-mon[33110]: 5.c scrub ok 2024-03-26T03:22:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:20 smithi192 ceph-mon[33110]: 1.1a deep-scrub starts 2024-03-26T03:22:20.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:20 smithi192 ceph-mon[33110]: 1.1a deep-scrub ok 2024-03-26T03:22:20.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[29666]: 5.c scrub starts 2024-03-26T03:22:20.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[29666]: 5.c scrub ok 2024-03-26T03:22:20.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[29666]: 1.1a deep-scrub starts 2024-03-26T03:22:20.785 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[29666]: 1.1a deep-scrub ok 2024-03-26T03:22:20.785 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[37920]: 5.c scrub starts 2024-03-26T03:22:20.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[37920]: 5.c scrub ok 2024-03-26T03:22:20.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[37920]: 1.1a deep-scrub starts 2024-03-26T03:22:20.786 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:20 smithi066 ceph-mon[37920]: 1.1a deep-scrub ok 2024-03-26T03:22:21.674 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[37920]: 4.15 scrub starts 2024-03-26T03:22:21.674 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[37920]: 4.15 scrub ok 2024-03-26T03:22:21.674 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:21.674 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[37920]: 4.1e scrub starts 2024-03-26T03:22:21.674 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[37920]: 4.1e scrub ok 2024-03-26T03:22:21.674 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[37920]: pgmap v1112: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:21.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[29666]: 4.15 scrub starts 2024-03-26T03:22:21.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[29666]: 4.15 scrub ok 2024-03-26T03:22:21.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:21.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[29666]: 4.1e scrub starts 2024-03-26T03:22:21.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[29666]: 4.1e scrub ok 2024-03-26T03:22:21.675 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:21 smithi066 ceph-mon[29666]: pgmap v1112: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:21.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:21 smithi192 ceph-mon[33110]: 4.15 scrub starts 2024-03-26T03:22:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:21 smithi192 ceph-mon[33110]: 4.15 scrub ok 2024-03-26T03:22:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:21 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:21 smithi192 ceph-mon[33110]: 4.1e scrub starts 2024-03-26T03:22:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:21 smithi192 ceph-mon[33110]: 4.1e scrub ok 2024-03-26T03:22:21.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:21 smithi192 ceph-mon[33110]: pgmap v1112: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:22.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:22 smithi066 ceph-mon[29666]: 5.f scrub starts 2024-03-26T03:22:22.635 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:22 smithi066 ceph-mon[29666]: 5.f scrub ok 2024-03-26T03:22:22.635 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:22 smithi066 ceph-mon[37920]: 5.f scrub starts 2024-03-26T03:22:22.635 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:22 smithi066 ceph-mon[37920]: 5.f scrub ok 2024-03-26T03:22:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:22 smithi192 ceph-mon[33110]: 5.f scrub starts 2024-03-26T03:22:22.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:22 smithi192 ceph-mon[33110]: 5.f scrub ok 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[29666]: 3.1 scrub starts 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[29666]: 3.1 scrub ok 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[29666]: pgmap v1113: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[29666]: 4.0 scrub starts 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[29666]: 4.0 scrub ok 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[37920]: 3.1 scrub starts 2024-03-26T03:22:23.584 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[37920]: 3.1 scrub ok 2024-03-26T03:22:23.585 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[37920]: pgmap v1113: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:23.585 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[37920]: 4.0 scrub starts 2024-03-26T03:22:23.585 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-mon[37920]: 4.0 scrub ok 2024-03-26T03:22:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:23 smithi192 ceph-mon[33110]: 3.1 scrub starts 2024-03-26T03:22:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:23 smithi192 ceph-mon[33110]: 3.1 scrub ok 2024-03-26T03:22:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:23 smithi192 ceph-mon[33110]: pgmap v1113: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:23 smithi192 ceph-mon[33110]: 4.0 scrub starts 2024-03-26T03:22:23.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:23 smithi192 ceph-mon[33110]: 4.0 scrub ok 2024-03-26T03:22:23.858 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:22:23 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:22:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:22:24.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:24 smithi192 ceph-mon[33110]: 1.13 scrub starts 2024-03-26T03:22:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:24 smithi192 ceph-mon[33110]: 1.13 scrub ok 2024-03-26T03:22:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:24 smithi192 ceph-mon[33110]: 6.e scrub starts 2024-03-26T03:22:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:24 smithi192 ceph-mon[33110]: 6.e scrub ok 2024-03-26T03:22:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:24 smithi192 ceph-mon[33110]: 4.14 scrub starts 2024-03-26T03:22:24.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:24 smithi192 ceph-mon[33110]: 4.14 scrub ok 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[37920]: 1.13 scrub starts 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[37920]: 1.13 scrub ok 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[37920]: 6.e scrub starts 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[37920]: 6.e scrub ok 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[37920]: 4.14 scrub starts 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[37920]: 4.14 scrub ok 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[29666]: 1.13 scrub starts 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[29666]: 1.13 scrub ok 2024-03-26T03:22:24.808 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[29666]: 6.e scrub starts 2024-03-26T03:22:24.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[29666]: 6.e scrub ok 2024-03-26T03:22:24.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[29666]: 4.14 scrub starts 2024-03-26T03:22:24.809 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:24 smithi066 ceph-mon[29666]: 4.14 scrub ok 2024-03-26T03:22:25.385 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[29666]: pgmap v1114: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:25.385 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[29666]: 2.e scrub starts 2024-03-26T03:22:25.385 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[29666]: 2.e scrub ok 2024-03-26T03:22:25.385 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:25.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[37920]: pgmap v1114: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:25.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[37920]: 2.e scrub starts 2024-03-26T03:22:25.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[37920]: 2.e scrub ok 2024-03-26T03:22:25.386 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:25 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:25.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:25 smithi192 ceph-mon[33110]: pgmap v1114: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:25 smithi192 ceph-mon[33110]: 2.e scrub starts 2024-03-26T03:22:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:25 smithi192 ceph-mon[33110]: 2.e scrub ok 2024-03-26T03:22:25.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:25 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:26.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 2.2 scrub starts 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 2.2 scrub ok 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 4.16 scrub starts 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 4.16 scrub ok 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 4.1c scrub starts 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 4.1c scrub ok 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 1.4 scrub starts 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[37920]: 1.4 scrub ok 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 2.2 scrub starts 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 2.2 scrub ok 2024-03-26T03:22:26.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 4.16 scrub starts 2024-03-26T03:22:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 4.16 scrub ok 2024-03-26T03:22:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 4.1c scrub starts 2024-03-26T03:22:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 4.1c scrub ok 2024-03-26T03:22:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 1.4 scrub starts 2024-03-26T03:22:26.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:26 smithi066 ceph-mon[29666]: 1.4 scrub ok 2024-03-26T03:22:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 2.2 scrub starts 2024-03-26T03:22:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 2.2 scrub ok 2024-03-26T03:22:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 4.16 scrub starts 2024-03-26T03:22:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 4.16 scrub ok 2024-03-26T03:22:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 4.1c scrub starts 2024-03-26T03:22:26.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 4.1c scrub ok 2024-03-26T03:22:26.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 1.4 scrub starts 2024-03-26T03:22:26.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:26 smithi192 ceph-mon[33110]: 1.4 scrub ok 2024-03-26T03:22:27.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: 1.17 scrub starts 2024-03-26T03:22:27.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: 1.17 scrub ok 2024-03-26T03:22:27.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: 4.18 deep-scrub starts 2024-03-26T03:22:27.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: 4.18 deep-scrub ok 2024-03-26T03:22:27.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: 4.6 scrub starts 2024-03-26T03:22:27.561 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: 4.6 scrub ok 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[37920]: pgmap v1115: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: 1.17 scrub starts 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: 1.17 scrub ok 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: 4.18 deep-scrub starts 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: 4.18 deep-scrub ok 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: 4.6 scrub starts 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: 4.6 scrub ok 2024-03-26T03:22:27.562 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:27 smithi066 ceph-mon[29666]: pgmap v1115: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:22:27.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: 1.17 scrub starts 2024-03-26T03:22:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: 1.17 scrub ok 2024-03-26T03:22:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: 4.18 deep-scrub starts 2024-03-26T03:22:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: 4.18 deep-scrub ok 2024-03-26T03:22:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: 4.6 scrub starts 2024-03-26T03:22:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: 4.6 scrub ok 2024-03-26T03:22:27.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:27 smithi192 ceph-mon[33110]: pgmap v1115: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:22:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 6.19 scrub starts 2024-03-26T03:22:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 6.19 scrub ok 2024-03-26T03:22:28.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 4.13 scrub starts 2024-03-26T03:22:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 4.13 scrub ok 2024-03-26T03:22:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 1.1c scrub starts 2024-03-26T03:22:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 1.1c scrub ok 2024-03-26T03:22:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 6.16 scrub starts 2024-03-26T03:22:28.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:28 smithi192 ceph-mon[33110]: 6.16 scrub ok 2024-03-26T03:22:28.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 6.19 scrub starts 2024-03-26T03:22:28.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 6.19 scrub ok 2024-03-26T03:22:28.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 4.13 scrub starts 2024-03-26T03:22:28.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 4.13 scrub ok 2024-03-26T03:22:28.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 1.1c scrub starts 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 1.1c scrub ok 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 6.16 scrub starts 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[29666]: 6.16 scrub ok 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 6.19 scrub starts 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 6.19 scrub ok 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 4.13 scrub starts 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 4.13 scrub ok 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 1.1c scrub starts 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 1.1c scrub ok 2024-03-26T03:22:28.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 6.16 scrub starts 2024-03-26T03:22:28.763 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:28 smithi066 ceph-mon[37920]: 6.16 scrub ok 2024-03-26T03:22:28.795 INFO:teuthology.orchestra.run.smithi066.stdout:1421634175074 2024-03-26T03:22:28.795 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T03:22:29.645 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:29 smithi066 ceph-mon[29666]: pgmap v1116: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:29.646 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:29 smithi066 ceph-mon[37920]: pgmap v1116: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:29.675 INFO:teuthology.orchestra.run.smithi066.stdout:133143986477 2024-03-26T03:22:29.676 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T03:22:29.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:29 smithi192 ceph-mon[33110]: pgmap v1116: 162 pgs: 162 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:30.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:30 smithi066 ceph-mon[29666]: 6.8 scrub starts 2024-03-26T03:22:30.725 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:30 smithi066 ceph-mon[29666]: 6.8 scrub ok 2024-03-26T03:22:30.725 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:30 smithi066 ceph-mon[37920]: 6.8 scrub starts 2024-03-26T03:22:30.725 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:30 smithi066 ceph-mon[37920]: 6.8 scrub ok 2024-03-26T03:22:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:30 smithi192 ceph-mon[33110]: 6.8 scrub starts 2024-03-26T03:22:30.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:30 smithi192 ceph-mon[33110]: 6.8 scrub ok 2024-03-26T03:22:30.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:22:30 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:22:30.753 INFO:teuthology.orchestra.run.smithi066.stdout:55834575164 2024-03-26T03:22:30.753 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T03:22:31.222 INFO:teuthology.orchestra.run.smithi066.stdout:171798692137 2024-03-26T03:22:31.222 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T03:22:31.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:31.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[37920]: pgmap v1117: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:22:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[37920]: 1.11 scrub starts 2024-03-26T03:22:31.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[37920]: 1.11 scrub ok 2024-03-26T03:22:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[29666]: pgmap v1117: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:22:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[29666]: 1.11 scrub starts 2024-03-26T03:22:31.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:31 smithi066 ceph-mon[29666]: 1.11 scrub ok 2024-03-26T03:22:31.718 INFO:teuthology.orchestra.run.smithi066.stdout:77309411638 2024-03-26T03:22:31.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T03:22:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:31 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:31 smithi192 ceph-mon[33110]: pgmap v1117: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:22:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:31 smithi192 ceph-mon[33110]: 1.11 scrub starts 2024-03-26T03:22:31.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:31 smithi192 ceph-mon[33110]: 1.11 scrub ok 2024-03-26T03:22:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:32 smithi192 ceph-mon[33110]: 6.1d scrub starts 2024-03-26T03:22:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:32 smithi192 ceph-mon[33110]: 6.1d scrub ok 2024-03-26T03:22:32.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:32 smithi192 ceph-mon[33110]: 2.0 deep-scrub starts 2024-03-26T03:22:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:32 smithi192 ceph-mon[33110]: 2.0 deep-scrub ok 2024-03-26T03:22:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:32 smithi192 ceph-mon[33110]: 2.10 scrub starts 2024-03-26T03:22:32.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:32 smithi192 ceph-mon[33110]: 2.10 scrub ok 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[29666]: 6.1d scrub starts 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[29666]: 6.1d scrub ok 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[29666]: 2.0 deep-scrub starts 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[29666]: 2.0 deep-scrub ok 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[29666]: 2.10 scrub starts 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[29666]: 2.10 scrub ok 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[37920]: 6.1d scrub starts 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[37920]: 6.1d scrub ok 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[37920]: 2.0 deep-scrub starts 2024-03-26T03:22:32.739 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[37920]: 2.0 deep-scrub ok 2024-03-26T03:22:32.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[37920]: 2.10 scrub starts 2024-03-26T03:22:32.740 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:32 smithi066 ceph-mon[37920]: 2.10 scrub ok 2024-03-26T03:22:32.778 INFO:teuthology.orchestra.run.smithi066.stdout:249108103455 2024-03-26T03:22:32.779 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T03:22:33.177 INFO:teuthology.orchestra.run.smithi066.stdout:34359738688 2024-03-26T03:22:33.178 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T03:22:33.301 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:33.343 INFO:teuthology.orchestra.run.smithi066.stdout:206158430500 2024-03-26T03:22:33.343 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T03:22:33.377 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:33.382 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:33.382 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:33.408 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-mon[29666]: 4.2 deep-scrub starts 2024-03-26T03:22:33.409 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-mon[29666]: 4.2 deep-scrub ok 2024-03-26T03:22:33.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-mon[29666]: pgmap v1118: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:33.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-mon[37920]: 4.2 deep-scrub starts 2024-03-26T03:22:33.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-mon[37920]: 4.2 deep-scrub ok 2024-03-26T03:22:33.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-mon[37920]: pgmap v1118: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:33 smithi192 ceph-mon[33110]: 4.2 deep-scrub starts 2024-03-26T03:22:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:33 smithi192 ceph-mon[33110]: 4.2 deep-scrub ok 2024-03-26T03:22:33.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:33 smithi192 ceph-mon[33110]: pgmap v1118: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:34.027 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:34.059 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:34.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:22:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:22:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:22:34.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[29666]: purged_snaps scrub starts 2024-03-26T03:22:34.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[29666]: purged_snaps scrub ok 2024-03-26T03:22:34.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[29666]: 6.12 scrub starts 2024-03-26T03:22:34.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[29666]: 6.12 scrub ok 2024-03-26T03:22:34.704 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[29666]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T03:22:34.704 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[37920]: purged_snaps scrub starts 2024-03-26T03:22:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[37920]: purged_snaps scrub ok 2024-03-26T03:22:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[37920]: 6.12 scrub starts 2024-03-26T03:22:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[37920]: 6.12 scrub ok 2024-03-26T03:22:34.705 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:34 smithi066 ceph-mon[37920]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T03:22:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:34 smithi192 ceph-mon[33110]: purged_snaps scrub starts 2024-03-26T03:22:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:34 smithi192 ceph-mon[33110]: purged_snaps scrub ok 2024-03-26T03:22:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:34 smithi192 ceph-mon[33110]: 6.12 scrub starts 2024-03-26T03:22:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:34 smithi192 ceph-mon[33110]: 6.12 scrub ok 2024-03-26T03:22:34.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:34 smithi192 ceph-mon[33110]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T03:22:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[29666]: pgmap v1120: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[29666]: 6.f deep-scrub starts 2024-03-26T03:22:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[29666]: 5.13 scrub starts 2024-03-26T03:22:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[29666]: 5.13 scrub ok 2024-03-26T03:22:35.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[29666]: 6.f deep-scrub ok 2024-03-26T03:22:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[37920]: pgmap v1120: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[37920]: 6.f deep-scrub starts 2024-03-26T03:22:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[37920]: 5.13 scrub starts 2024-03-26T03:22:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[37920]: 5.13 scrub ok 2024-03-26T03:22:35.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:35 smithi066 ceph-mon[37920]: 6.f deep-scrub ok 2024-03-26T03:22:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:35 smithi192 ceph-mon[33110]: pgmap v1120: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:35.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:35 smithi192 ceph-mon[33110]: 6.f deep-scrub starts 2024-03-26T03:22:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:35 smithi192 ceph-mon[33110]: 5.13 scrub starts 2024-03-26T03:22:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:35 smithi192 ceph-mon[33110]: 5.13 scrub ok 2024-03-26T03:22:35.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:35 smithi192 ceph-mon[33110]: 6.f deep-scrub ok 2024-03-26T03:22:36.667 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[29666]: 1.d scrub starts 2024-03-26T03:22:36.667 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[29666]: 1.d scrub ok 2024-03-26T03:22:36.667 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[29666]: 4.12 scrub starts 2024-03-26T03:22:36.667 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[29666]: 4.12 scrub ok 2024-03-26T03:22:36.668 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[37920]: 1.d scrub starts 2024-03-26T03:22:36.668 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[37920]: 1.d scrub ok 2024-03-26T03:22:36.668 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[37920]: 4.12 scrub starts 2024-03-26T03:22:36.668 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:36 smithi066 ceph-mon[37920]: 4.12 scrub ok 2024-03-26T03:22:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:36 smithi192 ceph-mon[33110]: 1.d scrub starts 2024-03-26T03:22:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:36 smithi192 ceph-mon[33110]: 1.d scrub ok 2024-03-26T03:22:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:36 smithi192 ceph-mon[33110]: 4.12 scrub starts 2024-03-26T03:22:36.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:36 smithi192 ceph-mon[33110]: 4.12 scrub ok 2024-03-26T03:22:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:37 smithi066 ceph-mon[29666]: pgmap v1121: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:37.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:37 smithi066 ceph-mon[29666]: 5.1f scrub starts 2024-03-26T03:22:37.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:37 smithi066 ceph-mon[29666]: 5.1f scrub ok 2024-03-26T03:22:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:37 smithi066 ceph-mon[37920]: pgmap v1121: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:37 smithi066 ceph-mon[37920]: 5.1f scrub starts 2024-03-26T03:22:37.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:37 smithi066 ceph-mon[37920]: 5.1f scrub ok 2024-03-26T03:22:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:37 smithi192 ceph-mon[33110]: pgmap v1121: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:37.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:37 smithi192 ceph-mon[33110]: 5.1f scrub starts 2024-03-26T03:22:37.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:37 smithi192 ceph-mon[33110]: 5.1f scrub ok 2024-03-26T03:22:38.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 5.7 scrub starts 2024-03-26T03:22:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 5.7 scrub ok 2024-03-26T03:22:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 5.18 scrub starts 2024-03-26T03:22:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 5.18 scrub ok 2024-03-26T03:22:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 6.15 scrub starts 2024-03-26T03:22:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 5.1e scrub starts 2024-03-26T03:22:38.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:38 smithi192 ceph-mon[33110]: 5.1e scrub ok 2024-03-26T03:22:38.840 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 5.7 scrub starts 2024-03-26T03:22:38.840 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 5.7 scrub ok 2024-03-26T03:22:38.840 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 5.18 scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 5.18 scrub ok 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 6.15 scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 5.1e scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[29666]: 5.1e scrub ok 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 5.7 scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 5.7 scrub ok 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 5.18 scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 5.18 scrub ok 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 6.15 scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 5.1e scrub starts 2024-03-26T03:22:38.841 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:38 smithi066 ceph-mon[37920]: 5.1e scrub ok 2024-03-26T03:22:39.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 6.15 scrub ok 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 1.16 scrub starts 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 1.16 scrub ok 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 5.15 scrub starts 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 5.15 scrub ok 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: pgmap v1122: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 1.f scrub starts 2024-03-26T03:22:39.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:39 smithi192 ceph-mon[33110]: 1.f scrub ok 2024-03-26T03:22:39.858 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 6.15 scrub ok 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 1.16 scrub starts 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 1.16 scrub ok 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 5.15 scrub starts 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 5.15 scrub ok 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: pgmap v1122: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 1.f scrub starts 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[37920]: 1.f scrub ok 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 6.15 scrub ok 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 1.16 scrub starts 2024-03-26T03:22:39.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 1.16 scrub ok 2024-03-26T03:22:39.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 5.15 scrub starts 2024-03-26T03:22:39.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 5.15 scrub ok 2024-03-26T03:22:39.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: pgmap v1122: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:39.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 1.f scrub starts 2024-03-26T03:22:39.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:39 smithi066 ceph-mon[29666]: 1.f scrub ok 2024-03-26T03:22:40.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[29666]: 2.4 scrub starts 2024-03-26T03:22:40.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[29666]: 2.4 scrub ok 2024-03-26T03:22:40.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[29666]: 2.11 scrub starts 2024-03-26T03:22:40.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[29666]: 2.11 scrub ok 2024-03-26T03:22:40.537 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:40.537 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[37920]: 2.4 scrub starts 2024-03-26T03:22:40.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[37920]: 2.4 scrub ok 2024-03-26T03:22:40.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[37920]: 2.11 scrub starts 2024-03-26T03:22:40.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[37920]: 2.11 scrub ok 2024-03-26T03:22:40.538 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:40 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:40.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:22:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:22:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:40 smithi192 ceph-mon[33110]: 2.4 scrub starts 2024-03-26T03:22:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:40 smithi192 ceph-mon[33110]: 2.4 scrub ok 2024-03-26T03:22:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:40 smithi192 ceph-mon[33110]: 2.11 scrub starts 2024-03-26T03:22:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:40 smithi192 ceph-mon[33110]: 2.11 scrub ok 2024-03-26T03:22:40.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:40 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:40.781 INFO:teuthology.orchestra.run.smithi066.stdout:77309411640 2024-03-26T03:22:41.396 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411638 got 77309411640 for osd.2 2024-03-26T03:22:41.397 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:41.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:41 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:41.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:41 smithi066 ceph-mon[29666]: pgmap v1123: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-26T03:22:41.469 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:41 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/842584755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:22:41.469 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:41 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:41.470 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:41 smithi066 ceph-mon[37920]: pgmap v1123: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-26T03:22:41.470 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:41 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/842584755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:22:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:41 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:41.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:41 smithi192 ceph-mon[33110]: pgmap v1123: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-26T03:22:41.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:41 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/842584755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:22:42.007 INFO:teuthology.orchestra.run.smithi066.stdout:133143986480 2024-03-26T03:22:42.222 INFO:teuthology.orchestra.run.smithi066.stdout:171798692140 2024-03-26T03:22:42.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[29666]: 6.7 scrub starts 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[29666]: 6.7 scrub ok 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[29666]: 5.a scrub starts 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[29666]: 5.a scrub ok 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1120199715' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2362869325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[37920]: 6.7 scrub starts 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[37920]: 6.7 scrub ok 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[37920]: 5.a scrub starts 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[37920]: 5.a scrub ok 2024-03-26T03:22:42.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1120199715' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:22:42.730 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:42 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2362869325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:22:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:42 smithi192 ceph-mon[33110]: 6.7 scrub starts 2024-03-26T03:22:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:42 smithi192 ceph-mon[33110]: 6.7 scrub ok 2024-03-26T03:22:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:42 smithi192 ceph-mon[33110]: 5.a scrub starts 2024-03-26T03:22:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:42 smithi192 ceph-mon[33110]: 5.a scrub ok 2024-03-26T03:22:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:42 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1120199715' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:22:42.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:42 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2362869325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:22:43.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692137 got 171798692140 for osd.5 2024-03-26T03:22:43.196 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:43.588 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986477 got 133143986480 for osd.4 2024-03-26T03:22:43.589 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:43.727 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 2.6 deep-scrub starts 2024-03-26T03:22:43.727 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 2.6 deep-scrub ok 2024-03-26T03:22:43.727 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 4.d scrub starts 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 4.d scrub ok 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 1.14 deep-scrub starts 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 1.14 deep-scrub ok 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: pgmap v1124: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 5.2 scrub starts 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 5.2 scrub ok 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 6.c scrub starts 2024-03-26T03:22:43.728 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[29666]: 6.c scrub ok 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 2.6 deep-scrub starts 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 2.6 deep-scrub ok 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 4.d scrub starts 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 4.d scrub ok 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 1.14 deep-scrub starts 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 1.14 deep-scrub ok 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: pgmap v1124: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 5.2 scrub starts 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 5.2 scrub ok 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 6.c scrub starts 2024-03-26T03:22:43.729 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-mon[37920]: 6.c scrub ok 2024-03-26T03:22:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 2.6 deep-scrub starts 2024-03-26T03:22:43.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 2.6 deep-scrub ok 2024-03-26T03:22:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 4.d scrub starts 2024-03-26T03:22:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 4.d scrub ok 2024-03-26T03:22:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 1.14 deep-scrub starts 2024-03-26T03:22:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 1.14 deep-scrub ok 2024-03-26T03:22:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: pgmap v1124: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:22:43.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 5.2 scrub starts 2024-03-26T03:22:43.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 5.2 scrub ok 2024-03-26T03:22:43.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 6.c scrub starts 2024-03-26T03:22:43.739 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:43 smithi192 ceph-mon[33110]: 6.c scrub ok 2024-03-26T03:22:44.029 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:22:43 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:22:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:22:44.036 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:44.037 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:44.275 INFO:teuthology.orchestra.run.smithi066.stdout:55834575167 2024-03-26T03:22:44.482 INFO:teuthology.orchestra.run.smithi066.stdout:1421634175078 2024-03-26T03:22:44.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: 5.1a scrub starts 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: 5.1a scrub ok 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: 6.9 scrub starts 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: 4.11 scrub starts 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: 4.11 scrub ok 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: 6.9 scrub ok 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2224435530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: 5.1a scrub starts 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: 5.1a scrub ok 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: 6.9 scrub starts 2024-03-26T03:22:44.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: 4.11 scrub starts 2024-03-26T03:22:44.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: 4.11 scrub ok 2024-03-26T03:22:44.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: 6.9 scrub ok 2024-03-26T03:22:44.610 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:44 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2224435530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:22:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: 5.1a scrub starts 2024-03-26T03:22:44.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: 5.1a scrub ok 2024-03-26T03:22:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: 6.9 scrub starts 2024-03-26T03:22:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: 4.11 scrub starts 2024-03-26T03:22:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: 4.11 scrub ok 2024-03-26T03:22:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: 6.9 scrub ok 2024-03-26T03:22:44.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:44 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2224435530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:22:45.200 INFO:teuthology.orchestra.run.smithi066.stdout:249108103458 2024-03-26T03:22:45.465 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3002710311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:22:45.465 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:45 smithi066 ceph-mon[29666]: pgmap v1125: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-26T03:22:45.465 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:45 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/2161700858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:22:45.465 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3002710311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:22:45.465 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:45 smithi066 ceph-mon[37920]: pgmap v1125: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-26T03:22:45.465 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:45 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/2161700858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:22:45.491 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1421634175074 got 1421634175078 for osd.3 2024-03-26T03:22:45.492 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:45.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3002710311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:22:45.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:45 smithi192 ceph-mon[33110]: pgmap v1125: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-26T03:22:45.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:45 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/2161700858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:22:46.454 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575164 got 55834575167 for osd.1 2024-03-26T03:22:46.455 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:46.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:46 smithi066 ceph-mon[29666]: 6.5 scrub starts 2024-03-26T03:22:46.547 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:46 smithi066 ceph-mon[29666]: 6.5 scrub ok 2024-03-26T03:22:46.547 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:46 smithi066 ceph-mon[37920]: 6.5 scrub starts 2024-03-26T03:22:46.548 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:46 smithi066 ceph-mon[37920]: 6.5 scrub ok 2024-03-26T03:22:46.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:46 smithi192 ceph-mon[33110]: 6.5 scrub starts 2024-03-26T03:22:46.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:46 smithi192 ceph-mon[33110]: 6.5 scrub ok 2024-03-26T03:22:47.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:47 smithi066 ceph-mon[37920]: 1.1d scrub starts 2024-03-26T03:22:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:47 smithi066 ceph-mon[37920]: 1.1d scrub ok 2024-03-26T03:22:47.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:47 smithi066 ceph-mon[37920]: pgmap v1126: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:47 smithi066 ceph-mon[29666]: 1.1d scrub starts 2024-03-26T03:22:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:47 smithi066 ceph-mon[29666]: 1.1d scrub ok 2024-03-26T03:22:47.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:47 smithi066 ceph-mon[29666]: pgmap v1126: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:47.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:47 smithi192 ceph-mon[33110]: 1.1d scrub starts 2024-03-26T03:22:47.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:47 smithi192 ceph-mon[33110]: 1.1d scrub ok 2024-03-26T03:22:47.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:47 smithi192 ceph-mon[33110]: pgmap v1126: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:48.761 INFO:teuthology.orchestra.run.smithi066.stdout:34359738693 2024-03-26T03:22:48.761 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[29666]: 2.1a scrub starts 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[29666]: 2.1a scrub ok 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[29666]: 6.b scrub starts 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[29666]: 6.b scrub ok 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[37920]: 2.1a scrub starts 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[37920]: 2.1a scrub ok 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[37920]: 6.b scrub starts 2024-03-26T03:22:48.762 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:48 smithi066 ceph-mon[37920]: 6.b scrub ok 2024-03-26T03:22:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:48 smithi192 ceph-mon[33110]: 2.1a scrub starts 2024-03-26T03:22:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:48 smithi192 ceph-mon[33110]: 2.1a scrub ok 2024-03-26T03:22:48.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:48 smithi192 ceph-mon[33110]: 6.b scrub starts 2024-03-26T03:22:48.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:48 smithi192 ceph-mon[33110]: 6.b scrub ok 2024-03-26T03:22:49.602 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[29666]: 2.15 deep-scrub starts 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[29666]: 2.15 deep-scrub ok 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[29666]: 1.12 scrub starts 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[29666]: 1.12 scrub ok 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[29666]: pgmap v1127: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/1462136981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[37920]: 2.15 deep-scrub starts 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[37920]: 2.15 deep-scrub ok 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[37920]: 1.12 scrub starts 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[37920]: 1.12 scrub ok 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[37920]: pgmap v1127: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:49.603 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:49 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/1462136981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:22:49.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738688 got 34359738693 for osd.0 2024-03-26T03:22:49.633 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:49 smithi192 ceph-mon[33110]: 2.15 deep-scrub starts 2024-03-26T03:22:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:49 smithi192 ceph-mon[33110]: 2.15 deep-scrub ok 2024-03-26T03:22:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:49 smithi192 ceph-mon[33110]: 1.12 scrub starts 2024-03-26T03:22:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:49 smithi192 ceph-mon[33110]: 1.12 scrub ok 2024-03-26T03:22:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:49 smithi192 ceph-mon[33110]: pgmap v1127: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:49.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:49 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/1462136981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:22:50.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103455 got 249108103458 for osd.7 2024-03-26T03:22:50.332 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:50.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[29666]: 5.b scrub starts 2024-03-26T03:22:50.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[29666]: 5.b scrub ok 2024-03-26T03:22:50.576 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[29666]: 2.1c scrub starts 2024-03-26T03:22:50.577 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[29666]: 2.1c scrub ok 2024-03-26T03:22:50.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[37920]: 5.b scrub starts 2024-03-26T03:22:50.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[37920]: 5.b scrub ok 2024-03-26T03:22:50.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[37920]: 2.1c scrub starts 2024-03-26T03:22:50.577 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:50 smithi066 ceph-mon[37920]: 2.1c scrub ok 2024-03-26T03:22:50.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:50 smithi192 ceph-mon[33110]: 5.b scrub starts 2024-03-26T03:22:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:50 smithi192 ceph-mon[33110]: 5.b scrub ok 2024-03-26T03:22:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:50 smithi192 ceph-mon[33110]: 2.1c scrub starts 2024-03-26T03:22:50.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:50 smithi192 ceph-mon[33110]: 2.1c scrub ok 2024-03-26T03:22:50.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:22:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:22:51.045 INFO:teuthology.orchestra.run.smithi066.stdout:206158430504 2024-03-26T03:22:51.608 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: pgmap v1128: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: 2.13 scrub starts 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: 2.13 scrub ok 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: 2.19 deep-scrub starts 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: 2.19 deep-scrub ok 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3745085332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: pgmap v1128: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: 2.13 scrub starts 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: 2.13 scrub ok 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: 2.19 deep-scrub starts 2024-03-26T03:22:51.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: 2.19 deep-scrub ok 2024-03-26T03:22:51.610 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:51 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3745085332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T03:22:51.662 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430500 got 206158430504 for osd.6 2024-03-26T03:22:51.662 DEBUG:teuthology.parallel:result is None 2024-03-26T03:22:51.662 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T03:22:51.662 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:22:51.920 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:51.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:22:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: pgmap v1128: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: 2.13 scrub starts 2024-03-26T03:22:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: 2.13 scrub ok 2024-03-26T03:22:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: 2.19 deep-scrub starts 2024-03-26T03:22:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: 2.19 deep-scrub ok 2024-03-26T03:22:51.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:51 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3745085332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T03:22:52.736 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:52 smithi066 ceph-mon[29666]: 5.1b scrub starts 2024-03-26T03:22:52.737 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:52 smithi066 ceph-mon[29666]: 5.1b scrub ok 2024-03-26T03:22:52.737 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:52 smithi066 ceph-mon[37920]: 5.1b scrub starts 2024-03-26T03:22:52.737 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:52 smithi066 ceph-mon[37920]: 5.1b scrub ok 2024-03-26T03:22:52.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:52 smithi192 ceph-mon[33110]: 5.1b scrub starts 2024-03-26T03:22:52.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:52 smithi192 ceph-mon[33110]: 5.1b scrub ok 2024-03-26T03:22:53.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[29666]: 1.3 scrub starts 2024-03-26T03:22:53.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[29666]: 1.3 scrub ok 2024-03-26T03:22:53.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[29666]: 5.1d scrub starts 2024-03-26T03:22:53.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[29666]: 5.1d scrub ok 2024-03-26T03:22:53.831 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[29666]: pgmap v1129: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[37920]: 1.3 scrub starts 2024-03-26T03:22:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[37920]: 1.3 scrub ok 2024-03-26T03:22:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[37920]: 5.1d scrub starts 2024-03-26T03:22:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[37920]: 5.1d scrub ok 2024-03-26T03:22:53.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-mon[37920]: pgmap v1129: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:53.883 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:22:53.886 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:22:53.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:53 smithi192 ceph-mon[33110]: 1.3 scrub starts 2024-03-26T03:22:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:53 smithi192 ceph-mon[33110]: 1.3 scrub ok 2024-03-26T03:22:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:53 smithi192 ceph-mon[33110]: 5.1d scrub starts 2024-03-26T03:22:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:53 smithi192 ceph-mon[33110]: 5.1d scrub ok 2024-03-26T03:22:53.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:53 smithi192 ceph-mon[33110]: pgmap v1129: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:54.108 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:22:53 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:22:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:22:54.497 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1129,"stamp":"2024-03-26T03:22:52.516212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3270,"num_read_kb":2990,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":1519626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5358,"ondisk_log_size":5358,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":487,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6926580,"kb_used_data":8564,"kb_used_omap":15,"kb_used_meta":6917936,"kb_avail":743034636,"statfs":{"total":767960285184,"available":760867467264,"internally_reserved":0,"allocated":8769536,"data_stored":5332468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15604,"internal_metadata":7083967244},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003336"},"pg_stats":[{"pgid":"1.d","version":"82'128","reported_seq":385,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:35.381605+0000","last_change":"2024-03-26T03:22:35.381632+0000","last_active":"2024-03-26T03:22:35.381605+0000","last_peered":"2024-03-26T03:22:35.381605+0000","last_clean":"2024-03-26T03:22:35.381605+0000","last_became_active":"2024-03-26T03:21:00.295100+0000","last_became_peered":"2024-03-26T03:21:00.295100+0000","last_unstale":"2024-03-26T03:22:35.381605+0000","last_undegraded":"2024-03-26T03:22:35.381605+0000","last_fullsized":"2024-03-26T03:22:35.381605+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:35.381529+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:22:35.381529+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.354050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"477'11","reported_seq":109,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297675+0000","last_change":"2024-03-26T03:21:51.123948+0000","last_active":"2024-03-26T03:22:42.297675+0000","last_peered":"2024-03-26T03:22:42.297675+0000","last_clean":"2024-03-26T03:22:42.297675+0000","last_became_active":"2024-03-26T03:20:57.376505+0000","last_became_peered":"2024-03-26T03:20:57.376505+0000","last_unstale":"2024-03-26T03:22:42.297675+0000","last_undegraded":"2024-03-26T03:22:42.297675+0000","last_fullsized":"2024-03-26T03:22:42.297675+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'11","last_scrub_stamp":"2024-03-26T03:21:51.123834+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:21:51.123834+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:05.400320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"446'51","reported_seq":1148,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553058+0000","last_change":"2024-03-26T03:21:20.956466+0000","last_active":"2024-03-26T03:22:33.553058+0000","last_peered":"2024-03-26T03:22:33.553058+0000","last_clean":"2024-03-26T03:22:33.553058+0000","last_became_active":"2024-03-26T03:20:57.364284+0000","last_became_peered":"2024-03-26T03:20:57.364284+0000","last_unstale":"2024-03-26T03:22:33.553058+0000","last_undegraded":"2024-03-26T03:22:33.553058+0000","last_fullsized":"2024-03-26T03:22:33.553058+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'51","last_scrub_stamp":"2024-03-26T03:21:20.956352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:20.956352+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":617,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.411010+0000","last_change":"2024-03-26T03:22:24.597541+0000","last_active":"2024-03-26T03:22:33.411010+0000","last_peered":"2024-03-26T03:22:33.411010+0000","last_clean":"2024-03-26T03:22:33.411010+0000","last_became_active":"2024-03-26T03:21:26.466719+0000","last_became_peered":"2024-03-26T03:21:26.466719+0000","last_unstale":"2024-03-26T03:22:33.411010+0000","last_undegraded":"2024-03-26T03:22:33.411010+0000","last_fullsized":"2024-03-26T03:22:33.411010+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:24.597458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:24.597458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.424636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":734,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553304+0000","last_change":"2024-03-26T03:21:53.893295+0000","last_active":"2024-03-26T03:22:33.553304+0000","last_peered":"2024-03-26T03:22:33.553304+0000","last_clean":"2024-03-26T03:22:33.553304+0000","last_became_active":"2024-03-26T03:21:43.568404+0000","last_became_peered":"2024-03-26T03:21:43.568404+0000","last_unstale":"2024-03-26T03:22:33.553304+0000","last_undegraded":"2024-03-26T03:22:33.553304+0000","last_fullsized":"2024-03-26T03:22:33.553304+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:53.893197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:47.274511+0000","last_clean_scrub_stamp":"2024-03-26T03:21:53.893197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:08.415875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":498,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410468+0000","last_change":"2024-03-26T03:21:43.446862+0000","last_active":"2024-03-26T03:22:33.410468+0000","last_peered":"2024-03-26T03:22:33.410468+0000","last_clean":"2024-03-26T03:22:33.410468+0000","last_became_active":"2024-03-26T03:21:00.290504+0000","last_became_peered":"2024-03-26T03:21:00.290504+0000","last_unstale":"2024-03-26T03:22:33.410468+0000","last_undegraded":"2024-03-26T03:22:33.410468+0000","last_fullsized":"2024-03-26T03:22:33.410468+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:43.446783+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.446783+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:45.619393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"446'32","reported_seq":138,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416904+0000","last_change":"2024-03-26T03:21:59.340085+0000","last_active":"2024-03-26T03:22:33.416904+0000","last_peered":"2024-03-26T03:22:33.416904+0000","last_clean":"2024-03-26T03:22:33.416904+0000","last_became_active":"2024-03-26T03:20:12.725584+0000","last_became_peered":"2024-03-26T03:20:12.725584+0000","last_unstale":"2024-03-26T03:22:33.416904+0000","last_undegraded":"2024-03-26T03:22:33.416904+0000","last_fullsized":"2024-03-26T03:22:33.416904+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'32","last_scrub_stamp":"2024-03-26T03:21:59.339988+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:21:59.339988+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:06.819009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":980,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549562+0000","last_change":"2024-03-26T03:21:58.937860+0000","last_active":"2024-03-26T03:22:34.549562+0000","last_peered":"2024-03-26T03:22:34.549562+0000","last_clean":"2024-03-26T03:22:34.549562+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:22:34.549562+0000","last_undegraded":"2024-03-26T03:22:34.549562+0000","last_fullsized":"2024-03-26T03:22:34.549562+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:58.937775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:21:58.937775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.625992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":609,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:41.400718+0000","last_change":"2024-03-26T03:22:41.400759+0000","last_active":"2024-03-26T03:22:41.400718+0000","last_peered":"2024-03-26T03:22:41.400718+0000","last_clean":"2024-03-26T03:22:41.400718+0000","last_became_active":"2024-03-26T03:21:26.466644+0000","last_became_peered":"2024-03-26T03:21:26.466644+0000","last_unstale":"2024-03-26T03:22:41.400718+0000","last_undegraded":"2024-03-26T03:22:41.400718+0000","last_fullsized":"2024-03-26T03:22:41.400718+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:41.400646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:41.400646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:09.036294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:43.612985+0000","last_change":"2024-03-26T03:22:43.613009+0000","last_active":"2024-03-26T03:22:43.612985+0000","last_peered":"2024-03-26T03:22:43.612985+0000","last_clean":"2024-03-26T03:22:43.612985+0000","last_became_active":"2024-03-26T03:20:57.353859+0000","last_became_peered":"2024-03-26T03:20:57.353859+0000","last_unstale":"2024-03-26T03:22:43.612985+0000","last_undegraded":"2024-03-26T03:22:43.612985+0000","last_fullsized":"2024-03-26T03:22:43.612985+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:43.612922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:22:43.612922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:13.063334+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":1076,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:38.564775+0000","last_change":"2024-03-26T03:22:38.564826+0000","last_active":"2024-03-26T03:22:38.564775+0000","last_peered":"2024-03-26T03:22:38.564775+0000","last_clean":"2024-03-26T03:22:38.564775+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:22:38.564775+0000","last_undegraded":"2024-03-26T03:22:38.564775+0000","last_fullsized":"2024-03-26T03:22:38.564775+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:38.564662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:38.564662+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.250010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"446'57","reported_seq":1162,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.460768+0000","last_change":"2024-03-26T03:21:37.186655+0000","last_active":"2024-03-26T03:22:34.460768+0000","last_peered":"2024-03-26T03:22:34.460768+0000","last_clean":"2024-03-26T03:22:34.460768+0000","last_became_active":"2024-03-26T03:20:57.371493+0000","last_became_peered":"2024-03-26T03:20:57.371493+0000","last_unstale":"2024-03-26T03:22:34.460768+0000","last_undegraded":"2024-03-26T03:22:34.460768+0000","last_fullsized":"2024-03-26T03:22:34.460768+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'57","last_scrub_stamp":"2024-03-26T03:21:37.186539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.186539+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:57.985345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":823,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.412457+0000","last_change":"2024-03-26T03:22:07.632621+0000","last_active":"2024-03-26T03:22:33.412457+0000","last_peered":"2024-03-26T03:22:33.412457+0000","last_clean":"2024-03-26T03:22:33.412457+0000","last_became_active":"2024-03-26T03:21:26.463723+0000","last_became_peered":"2024-03-26T03:21:26.463723+0000","last_unstale":"2024-03-26T03:22:33.412457+0000","last_undegraded":"2024-03-26T03:22:33.412457+0000","last_fullsized":"2024-03-26T03:22:33.412457+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:07.632516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:22:07.632516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.434772+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:49.331509+0000","last_change":"2024-03-26T03:22:49.331531+0000","last_active":"2024-03-26T03:22:49.331509+0000","last_peered":"2024-03-26T03:22:49.331509+0000","last_clean":"2024-03-26T03:22:49.331509+0000","last_became_active":"2024-03-26T03:20:12.719811+0000","last_became_peered":"2024-03-26T03:20:12.719811+0000","last_unstale":"2024-03-26T03:22:49.331509+0000","last_undegraded":"2024-03-26T03:22:49.331509+0000","last_fullsized":"2024-03-26T03:22:49.331509+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:49.331438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:16.301182+0000","last_clean_scrub_stamp":"2024-03-26T03:22:49.331438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.235938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549133+0000","last_change":"2024-03-26T03:22:29.333875+0000","last_active":"2024-03-26T03:22:34.549133+0000","last_peered":"2024-03-26T03:22:34.549133+0000","last_clean":"2024-03-26T03:22:34.549133+0000","last_became_active":"2024-03-26T03:20:57.359873+0000","last_became_peered":"2024-03-26T03:20:57.359873+0000","last_unstale":"2024-03-26T03:22:34.549133+0000","last_undegraded":"2024-03-26T03:22:34.549133+0000","last_fullsized":"2024-03-26T03:22:34.549133+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:29.333767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:29.333767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:33.278689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"446'53","reported_seq":230,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.331265+0000","last_change":"2024-03-26T03:22:42.331310+0000","last_active":"2024-03-26T03:22:42.331265+0000","last_peered":"2024-03-26T03:22:42.331265+0000","last_clean":"2024-03-26T03:22:42.331265+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:22:42.331265+0000","last_undegraded":"2024-03-26T03:22:42.331265+0000","last_fullsized":"2024-03-26T03:22:42.331265+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'53","last_scrub_stamp":"2024-03-26T03:22:42.331184+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.331184+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:44.444720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549516+0000","last_change":"2024-03-26T03:22:12.147474+0000","last_active":"2024-03-26T03:22:34.549516+0000","last_peered":"2024-03-26T03:22:34.549516+0000","last_clean":"2024-03-26T03:22:34.549516+0000","last_became_active":"2024-03-26T03:20:57.375755+0000","last_became_peered":"2024-03-26T03:20:57.375755+0000","last_unstale":"2024-03-26T03:22:34.549516+0000","last_undegraded":"2024-03-26T03:22:34.549516+0000","last_fullsized":"2024-03-26T03:22:34.549516+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:12.147358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:12.147358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.758913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":987,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549200+0000","last_change":"2024-03-26T03:22:19.054387+0000","last_active":"2024-03-26T03:22:34.549200+0000","last_peered":"2024-03-26T03:22:34.549200+0000","last_clean":"2024-03-26T03:22:34.549200+0000","last_became_active":"2024-03-26T03:20:12.720725+0000","last_became_peered":"2024-03-26T03:20:12.720725+0000","last_unstale":"2024-03-26T03:22:34.549200+0000","last_undegraded":"2024-03-26T03:22:34.549200+0000","last_fullsized":"2024-03-26T03:22:34.549200+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:19.054291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:22:19.054291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:46.668028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":605,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.690729+0000","last_change":"2024-03-26T03:22:34.690752+0000","last_active":"2024-03-26T03:22:34.690729+0000","last_peered":"2024-03-26T03:22:34.690729+0000","last_clean":"2024-03-26T03:22:34.690729+0000","last_became_active":"2024-03-26T03:21:26.467423+0000","last_became_peered":"2024-03-26T03:21:26.467423+0000","last_unstale":"2024-03-26T03:22:34.690729+0000","last_undegraded":"2024-03-26T03:22:34.690729+0000","last_fullsized":"2024-03-26T03:22:34.690729+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:34.690650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:34.690650+0000","last_clean_scrub_stamp":"2024-03-26T03:22:34.690650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:48.469907+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":360,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553158+0000","last_change":"2024-03-26T03:21:43.919009+0000","last_active":"2024-03-26T03:22:33.553158+0000","last_peered":"2024-03-26T03:22:33.553158+0000","last_clean":"2024-03-26T03:22:33.553158+0000","last_became_active":"2024-03-26T03:21:43.565798+0000","last_became_peered":"2024-03-26T03:21:43.565798+0000","last_unstale":"2024-03-26T03:22:33.553158+0000","last_undegraded":"2024-03-26T03:22:33.553158+0000","last_fullsized":"2024-03-26T03:22:33.553158+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:43.918932+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:21:27.137087+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.918932+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:02.901198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":569,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549135+0000","last_change":"2024-03-26T03:22:15.192691+0000","last_active":"2024-03-26T03:22:34.549135+0000","last_peered":"2024-03-26T03:22:34.549135+0000","last_clean":"2024-03-26T03:22:34.549135+0000","last_became_active":"2024-03-26T03:21:00.293608+0000","last_became_peered":"2024-03-26T03:21:00.293608+0000","last_unstale":"2024-03-26T03:22:34.549135+0000","last_undegraded":"2024-03-26T03:22:34.549135+0000","last_fullsized":"2024-03-26T03:22:34.549135+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:15.192599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:15.192599+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:18.491992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"446'30","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410218+0000","last_change":"2024-03-26T03:22:18.293766+0000","last_active":"2024-03-26T03:22:33.410218+0000","last_peered":"2024-03-26T03:22:33.410218+0000","last_clean":"2024-03-26T03:22:33.410218+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:22:33.410218+0000","last_undegraded":"2024-03-26T03:22:33.410218+0000","last_fullsized":"2024-03-26T03:22:33.410218+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'30","last_scrub_stamp":"2024-03-26T03:22:18.293639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:18.293639+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:27.339943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":973,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548762+0000","last_change":"2024-03-26T03:21:42.933870+0000","last_active":"2024-03-26T03:22:34.548762+0000","last_peered":"2024-03-26T03:22:34.548762+0000","last_clean":"2024-03-26T03:22:34.548762+0000","last_became_active":"2024-03-26T03:20:57.364110+0000","last_became_peered":"2024-03-26T03:20:57.364110+0000","last_unstale":"2024-03-26T03:22:34.548762+0000","last_undegraded":"2024-03-26T03:22:34.548762+0000","last_fullsized":"2024-03-26T03:22:34.548762+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:42.933794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:42.933794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.532956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"490'14","reported_seq":1052,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297590+0000","last_change":"2024-03-26T03:21:37.303210+0000","last_active":"2024-03-26T03:22:42.297590+0000","last_peered":"2024-03-26T03:22:42.297590+0000","last_clean":"2024-03-26T03:22:42.297590+0000","last_became_active":"2024-03-26T03:21:26.466292+0000","last_became_peered":"2024-03-26T03:21:26.466292+0000","last_unstale":"2024-03-26T03:22:42.297590+0000","last_undegraded":"2024-03-26T03:22:42.297590+0000","last_fullsized":"2024-03-26T03:22:42.297590+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'14","last_scrub_stamp":"2024-03-26T03:21:37.303105+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.303105+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:05.462102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":191,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410502+0000","last_change":"2024-03-26T03:22:23.251448+0000","last_active":"2024-03-26T03:22:33.410502+0000","last_peered":"2024-03-26T03:22:33.410502+0000","last_clean":"2024-03-26T03:22:33.410502+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:22:33.410502+0000","last_undegraded":"2024-03-26T03:22:33.410502+0000","last_fullsized":"2024-03-26T03:22:33.410502+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:23.251349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:23.251349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.640251+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":340,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553135+0000","last_change":"2024-03-26T03:21:44.890578+0000","last_active":"2024-03-26T03:22:33.553135+0000","last_peered":"2024-03-26T03:22:33.553135+0000","last_clean":"2024-03-26T03:22:33.553135+0000","last_became_active":"2024-03-26T03:21:43.566139+0000","last_became_peered":"2024-03-26T03:21:43.566139+0000","last_unstale":"2024-03-26T03:22:33.553135+0000","last_undegraded":"2024-03-26T03:22:33.553135+0000","last_fullsized":"2024-03-26T03:22:33.553135+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:44.890494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:44.890494+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.294185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"446'39","reported_seq":1048,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549244+0000","last_change":"2024-03-26T03:22:10.066546+0000","last_active":"2024-03-26T03:22:34.549244+0000","last_peered":"2024-03-26T03:22:34.549244+0000","last_clean":"2024-03-26T03:22:34.549244+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:22:34.549244+0000","last_undegraded":"2024-03-26T03:22:34.549244+0000","last_fullsized":"2024-03-26T03:22:34.549244+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:22:10.066431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:10.066431+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:34.409681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":973,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548524+0000","last_change":"2024-03-26T03:21:52.021391+0000","last_active":"2024-03-26T03:22:34.548524+0000","last_peered":"2024-03-26T03:22:34.548524+0000","last_clean":"2024-03-26T03:22:34.548524+0000","last_became_active":"2024-03-26T03:20:57.364582+0000","last_became_peered":"2024-03-26T03:20:57.364582+0000","last_unstale":"2024-03-26T03:22:34.548524+0000","last_undegraded":"2024-03-26T03:22:34.548524+0000","last_fullsized":"2024-03-26T03:22:34.548524+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:52.021309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:52.021309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.642241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"460'13","reported_seq":266,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297667+0000","last_change":"2024-03-26T03:21:46.484839+0000","last_active":"2024-03-26T03:22:42.297667+0000","last_peered":"2024-03-26T03:22:42.297667+0000","last_clean":"2024-03-26T03:22:42.297667+0000","last_became_active":"2024-03-26T03:20:12.733250+0000","last_became_peered":"2024-03-26T03:20:12.733250+0000","last_unstale":"2024-03-26T03:22:42.297667+0000","last_undegraded":"2024-03-26T03:22:42.297667+0000","last_fullsized":"2024-03-26T03:22:42.297667+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'13","last_scrub_stamp":"2024-03-26T03:21:46.484733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.484733+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:08.987183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":710,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.554022+0000","last_change":"2024-03-26T03:21:43.857733+0000","last_active":"2024-03-26T03:22:33.554022+0000","last_peered":"2024-03-26T03:22:33.554022+0000","last_clean":"2024-03-26T03:22:33.554022+0000","last_became_active":"2024-03-26T03:21:43.857406+0000","last_became_peered":"2024-03-26T03:21:43.857406+0000","last_unstale":"2024-03-26T03:22:33.554022+0000","last_undegraded":"2024-03-26T03:22:33.554022+0000","last_fullsized":"2024-03-26T03:22:33.554022+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:26.956797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:40.962737+0000","last_clean_scrub_stamp":"2024-03-26T03:21:26.956797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.799452+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":1109,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548841+0000","last_change":"2024-03-26T03:21:20.908040+0000","last_active":"2024-03-26T03:22:34.548841+0000","last_peered":"2024-03-26T03:22:34.548841+0000","last_clean":"2024-03-26T03:22:34.548841+0000","last_became_active":"2024-03-26T03:20:15.312092+0000","last_became_peered":"2024-03-26T03:20:15.312092+0000","last_unstale":"2024-03-26T03:22:34.548841+0000","last_undegraded":"2024-03-26T03:22:34.548841+0000","last_fullsized":"2024-03-26T03:22:34.548841+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:20.907837+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:21:20.907837+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:47.661090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"446'39","reported_seq":1038,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410673+0000","last_change":"2024-03-26T03:21:16.400009+0000","last_active":"2024-03-26T03:22:33.410673+0000","last_peered":"2024-03-26T03:22:33.410673+0000","last_clean":"2024-03-26T03:22:33.410673+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:22:33.410673+0000","last_undegraded":"2024-03-26T03:22:33.410673+0000","last_fullsized":"2024-03-26T03:22:33.410673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:21:16.399895+0000","last_deep_scrub":"446'39","last_deep_scrub_stamp":"2024-03-26T03:21:16.399895+0000","last_clean_scrub_stamp":"2024-03-26T03:21:16.399895+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:41.176504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549379+0000","last_change":"2024-03-26T03:21:46.062451+0000","last_active":"2024-03-26T03:22:34.549379+0000","last_peered":"2024-03-26T03:22:34.549379+0000","last_clean":"2024-03-26T03:22:34.549379+0000","last_became_active":"2024-03-26T03:20:57.369214+0000","last_became_peered":"2024-03-26T03:20:57.369214+0000","last_unstale":"2024-03-26T03:22:34.549379+0000","last_undegraded":"2024-03-26T03:22:34.549379+0000","last_fullsized":"2024-03-26T03:22:34.549379+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:46.062330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.062330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:00.923161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":793,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410824+0000","last_change":"2024-03-26T03:22:21.270329+0000","last_active":"2024-03-26T03:22:33.410824+0000","last_peered":"2024-03-26T03:22:33.410824+0000","last_clean":"2024-03-26T03:22:33.410824+0000","last_became_active":"2024-03-26T03:20:57.356625+0000","last_became_peered":"2024-03-26T03:20:57.356625+0000","last_unstale":"2024-03-26T03:22:33.410824+0000","last_undegraded":"2024-03-26T03:22:33.410824+0000","last_fullsized":"2024-03-26T03:22:33.410824+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:21.270217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:21.270217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.790519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":608,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.560797+0000","last_change":"2024-03-26T03:22:42.560832+0000","last_active":"2024-03-26T03:22:42.560797+0000","last_peered":"2024-03-26T03:22:42.560797+0000","last_clean":"2024-03-26T03:22:42.560797+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:22:42.560797+0000","last_undegraded":"2024-03-26T03:22:42.560797+0000","last_fullsized":"2024-03-26T03:22:42.560797+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:42.560729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.560729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:12.481917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":829,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297662+0000","last_change":"2024-03-26T03:21:01.701749+0000","last_active":"2024-03-26T03:22:42.297662+0000","last_peered":"2024-03-26T03:22:42.297662+0000","last_clean":"2024-03-26T03:22:42.297662+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:22:42.297662+0000","last_undegraded":"2024-03-26T03:22:42.297662+0000","last_fullsized":"2024-03-26T03:22:42.297662+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:21:01.701636+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:21:01.701636+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"446'48","reported_seq":1061,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409845+0000","last_change":"2024-03-26T03:22:17.335701+0000","last_active":"2024-03-26T03:22:33.409845+0000","last_peered":"2024-03-26T03:22:33.409845+0000","last_clean":"2024-03-26T03:22:33.409845+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:22:33.409845+0000","last_undegraded":"2024-03-26T03:22:33.409845+0000","last_fullsized":"2024-03-26T03:22:33.409845+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'48","last_scrub_stamp":"2024-03-26T03:22:17.335576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:17.335576+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.479184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1004,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461363+0000","last_change":"2024-03-26T03:21:46.255611+0000","last_active":"2024-03-26T03:22:34.461363+0000","last_peered":"2024-03-26T03:22:34.461363+0000","last_clean":"2024-03-26T03:22:34.461363+0000","last_became_active":"2024-03-26T03:20:57.357348+0000","last_became_peered":"2024-03-26T03:20:57.357348+0000","last_unstale":"2024-03-26T03:22:34.461363+0000","last_undegraded":"2024-03-26T03:22:34.461363+0000","last_fullsized":"2024-03-26T03:22:34.461363+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:46.255506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:19.268683+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.255506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:54.166393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549535+0000","last_change":"2024-03-26T03:21:43.163634+0000","last_active":"2024-03-26T03:22:34.549535+0000","last_peered":"2024-03-26T03:22:34.549535+0000","last_clean":"2024-03-26T03:22:34.549535+0000","last_became_active":"2024-03-26T03:20:57.362070+0000","last_became_peered":"2024-03-26T03:20:57.362070+0000","last_unstale":"2024-03-26T03:22:34.549535+0000","last_undegraded":"2024-03-26T03:22:34.549535+0000","last_fullsized":"2024-03-26T03:22:34.549535+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:43.163526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:43.163526+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.163526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:00.782062+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":789,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469315+0000","last_change":"2024-03-26T03:22:25.641863+0000","last_active":"2024-03-26T03:22:34.469315+0000","last_peered":"2024-03-26T03:22:34.469315+0000","last_clean":"2024-03-26T03:22:34.469315+0000","last_became_active":"2024-03-26T03:21:00.294201+0000","last_became_peered":"2024-03-26T03:21:00.294201+0000","last_unstale":"2024-03-26T03:22:34.469315+0000","last_undegraded":"2024-03-26T03:22:34.469315+0000","last_fullsized":"2024-03-26T03:22:34.469315+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:25.641776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.641776+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.062856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":969,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410003+0000","last_change":"2024-03-26T03:22:13.330407+0000","last_active":"2024-03-26T03:22:33.410003+0000","last_peered":"2024-03-26T03:22:33.410003+0000","last_clean":"2024-03-26T03:22:33.410003+0000","last_became_active":"2024-03-26T03:20:57.355658+0000","last_became_peered":"2024-03-26T03:20:57.355658+0000","last_unstale":"2024-03-26T03:22:33.410003+0000","last_undegraded":"2024-03-26T03:22:33.410003+0000","last_fullsized":"2024-03-26T03:22:33.410003+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:13.330308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.330308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:16.417609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"446'66","reported_seq":875,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468456+0000","last_change":"2024-03-26T03:22:22.616080+0000","last_active":"2024-03-26T03:22:34.468456+0000","last_peered":"2024-03-26T03:22:34.468456+0000","last_clean":"2024-03-26T03:22:34.468456+0000","last_became_active":"2024-03-26T03:20:57.364272+0000","last_became_peered":"2024-03-26T03:20:57.364272+0000","last_unstale":"2024-03-26T03:22:34.468456+0000","last_undegraded":"2024-03-26T03:22:34.468456+0000","last_fullsized":"2024-03-26T03:22:34.468456+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'66","last_scrub_stamp":"2024-03-26T03:22:22.615941+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:22:22.615941+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.127529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":994,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:41.845584+0000","last_change":"2024-03-26T03:22:41.845584+0000","last_active":"2024-03-26T03:22:41.845584+0000","last_peered":"2024-03-26T03:22:41.845584+0000","last_clean":"2024-03-26T03:22:41.845584+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:22:41.845584+0000","last_undegraded":"2024-03-26T03:22:41.845584+0000","last_fullsized":"2024-03-26T03:22:41.845584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:22:41.845497+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:22:41.845497+0000","last_clean_scrub_stamp":"2024-03-26T03:22:41.845497+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:43.894102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":183,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409809+0000","last_change":"2024-03-26T03:21:39.470235+0000","last_active":"2024-03-26T03:22:33.409809+0000","last_peered":"2024-03-26T03:22:33.409809+0000","last_clean":"2024-03-26T03:22:33.409809+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:22:33.409809+0000","last_undegraded":"2024-03-26T03:22:33.409809+0000","last_fullsized":"2024-03-26T03:22:33.409809+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:39.470125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:39.470125+0000","last_clean_scrub_stamp":"2024-03-26T03:21:39.470125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:43.620499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":1083,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410714+0000","last_change":"2024-03-26T03:21:37.466092+0000","last_active":"2024-03-26T03:22:33.410714+0000","last_peered":"2024-03-26T03:22:33.410714+0000","last_clean":"2024-03-26T03:22:33.410714+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:22:33.410714+0000","last_undegraded":"2024-03-26T03:22:33.410714+0000","last_fullsized":"2024-03-26T03:22:33.410714+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:37.466001+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:21:37.466001+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.466001+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:49.706819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1004,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.523429+0000","last_change":"2024-03-26T03:22:42.523457+0000","last_active":"2024-03-26T03:22:42.523429+0000","last_peered":"2024-03-26T03:22:42.523429+0000","last_clean":"2024-03-26T03:22:42.523429+0000","last_became_active":"2024-03-26T03:20:12.719873+0000","last_became_peered":"2024-03-26T03:20:12.719873+0000","last_unstale":"2024-03-26T03:22:42.523429+0000","last_undegraded":"2024-03-26T03:22:42.523429+0000","last_fullsized":"2024-03-26T03:22:42.523429+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:42.523355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.523355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:54.434095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"446'45","reported_seq":172,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416774+0000","last_change":"2024-03-26T03:22:13.361000+0000","last_active":"2024-03-26T03:22:33.416774+0000","last_peered":"2024-03-26T03:22:33.416774+0000","last_clean":"2024-03-26T03:22:33.416774+0000","last_became_active":"2024-03-26T03:20:57.356509+0000","last_became_peered":"2024-03-26T03:20:57.356509+0000","last_unstale":"2024-03-26T03:22:33.416774+0000","last_undegraded":"2024-03-26T03:22:33.416774+0000","last_fullsized":"2024-03-26T03:22:33.416774+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'45","last_scrub_stamp":"2024-03-26T03:22:13.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.360829+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:14.432214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410623+0000","last_change":"2024-03-26T03:21:43.445246+0000","last_active":"2024-03-26T03:22:33.410623+0000","last_peered":"2024-03-26T03:22:33.410623+0000","last_clean":"2024-03-26T03:22:33.410623+0000","last_became_active":"2024-03-26T03:21:42.741369+0000","last_became_peered":"2024-03-26T03:21:42.741369+0000","last_unstale":"2024-03-26T03:22:33.410623+0000","last_undegraded":"2024-03-26T03:22:33.410623+0000","last_fullsized":"2024-03-26T03:22:33.410623+0000","mapping_epoch":494,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:43.445121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.445121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.474043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":975,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548714+0000","last_change":"2024-03-26T03:21:29.932255+0000","last_active":"2024-03-26T03:22:34.548714+0000","last_peered":"2024-03-26T03:22:34.548714+0000","last_clean":"2024-03-26T03:22:34.548714+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:22:34.548714+0000","last_undegraded":"2024-03-26T03:22:34.548714+0000","last_fullsized":"2024-03-26T03:22:34.548714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:29.932140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:21:29.932140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.824323+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":1078,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461260+0000","last_change":"2024-03-26T03:22:12.414426+0000","last_active":"2024-03-26T03:22:34.461260+0000","last_peered":"2024-03-26T03:22:34.461260+0000","last_clean":"2024-03-26T03:22:34.461260+0000","last_became_active":"2024-03-26T03:20:15.306151+0000","last_became_peered":"2024-03-26T03:20:15.306151+0000","last_unstale":"2024-03-26T03:22:34.461260+0000","last_undegraded":"2024-03-26T03:22:34.461260+0000","last_fullsized":"2024-03-26T03:22:34.461260+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:12.414324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:12.414324+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:36.107527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:36.946650+0000","last_change":"2024-03-26T03:22:36.946676+0000","last_active":"2024-03-26T03:22:36.946650+0000","last_peered":"2024-03-26T03:22:36.946650+0000","last_clean":"2024-03-26T03:22:36.946650+0000","last_became_active":"2024-03-26T03:20:12.732621+0000","last_became_peered":"2024-03-26T03:20:12.732621+0000","last_unstale":"2024-03-26T03:22:36.946650+0000","last_undegraded":"2024-03-26T03:22:36.946650+0000","last_fullsized":"2024-03-26T03:22:36.946650+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:36.946581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:36.946581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.948567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"446'42","reported_seq":140,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416782+0000","last_change":"2024-03-26T03:22:26.443959+0000","last_active":"2024-03-26T03:22:33.416782+0000","last_peered":"2024-03-26T03:22:33.416782+0000","last_clean":"2024-03-26T03:22:33.416782+0000","last_became_active":"2024-03-26T03:20:12.734007+0000","last_became_peered":"2024-03-26T03:20:12.734007+0000","last_unstale":"2024-03-26T03:22:33.416782+0000","last_undegraded":"2024-03-26T03:22:33.416782+0000","last_fullsized":"2024-03-26T03:22:33.416782+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'42","last_scrub_stamp":"2024-03-26T03:22:26.443851+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:22:26.443851+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.007652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548956+0000","last_change":"2024-03-26T03:22:31.323174+0000","last_active":"2024-03-26T03:22:34.548956+0000","last_peered":"2024-03-26T03:22:34.548956+0000","last_clean":"2024-03-26T03:22:34.548956+0000","last_became_active":"2024-03-26T03:20:57.367476+0000","last_became_peered":"2024-03-26T03:20:57.367476+0000","last_unstale":"2024-03-26T03:22:34.548956+0000","last_undegraded":"2024-03-26T03:22:34.548956+0000","last_fullsized":"2024-03-26T03:22:34.548956+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:31.323076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:31.323076+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.323076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:56.126169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":126,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416301+0000","last_change":"2024-03-26T03:22:22.437982+0000","last_active":"2024-03-26T03:22:33.416301+0000","last_peered":"2024-03-26T03:22:33.416301+0000","last_clean":"2024-03-26T03:22:33.416301+0000","last_became_active":"2024-03-26T03:20:12.745764+0000","last_became_peered":"2024-03-26T03:20:12.745764+0000","last_unstale":"2024-03-26T03:22:33.416301+0000","last_undegraded":"2024-03-26T03:22:33.416301+0000","last_fullsized":"2024-03-26T03:22:33.416301+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:22.437874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:22:22.437874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:41.213886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":957,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549214+0000","last_change":"2024-03-26T03:21:48.072831+0000","last_active":"2024-03-26T03:22:34.549214+0000","last_peered":"2024-03-26T03:22:34.549214+0000","last_clean":"2024-03-26T03:22:34.549214+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:22:34.549214+0000","last_undegraded":"2024-03-26T03:22:34.549214+0000","last_fullsized":"2024-03-26T03:22:34.549214+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:48.072751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:48.072751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:58.802616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":1089,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548947+0000","last_change":"2024-03-26T03:21:19.914128+0000","last_active":"2024-03-26T03:22:34.548947+0000","last_peered":"2024-03-26T03:22:34.548947+0000","last_clean":"2024-03-26T03:22:34.548947+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:22:34.548947+0000","last_undegraded":"2024-03-26T03:22:34.548947+0000","last_fullsized":"2024-03-26T03:22:34.548947+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:19.914019+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:21:19.914019+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:44.198335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549771+0000","last_change":"2024-03-26T03:21:49.077589+0000","last_active":"2024-03-26T03:22:34.549771+0000","last_peered":"2024-03-26T03:22:34.549771+0000","last_clean":"2024-03-26T03:22:34.549771+0000","last_became_active":"2024-03-26T03:20:57.370336+0000","last_became_peered":"2024-03-26T03:20:57.370336+0000","last_unstale":"2024-03-26T03:22:34.549771+0000","last_undegraded":"2024-03-26T03:22:34.549771+0000","last_fullsized":"2024-03-26T03:22:34.549771+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:49.077483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:49.077483+0000","last_clean_scrub_stamp":"2024-03-26T03:21:49.077483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:10.283581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"477'43","reported_seq":1414,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297669+0000","last_change":"2024-03-26T03:22:09.369344+0000","last_active":"2024-03-26T03:22:42.297669+0000","last_peered":"2024-03-26T03:22:42.297669+0000","last_clean":"2024-03-26T03:22:42.297669+0000","last_became_active":"2024-03-26T03:20:57.372520+0000","last_became_peered":"2024-03-26T03:20:57.372520+0000","last_unstale":"2024-03-26T03:22:42.297669+0000","last_undegraded":"2024-03-26T03:22:42.297669+0000","last_fullsized":"2024-03-26T03:22:42.297669+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'43","last_scrub_stamp":"2024-03-26T03:22:09.369202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:09.369202+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:35.501851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":952,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548872+0000","last_change":"2024-03-26T03:21:29.084072+0000","last_active":"2024-03-26T03:22:34.548872+0000","last_peered":"2024-03-26T03:22:34.548872+0000","last_clean":"2024-03-26T03:22:34.548872+0000","last_became_active":"2024-03-26T03:21:00.294086+0000","last_became_peered":"2024-03-26T03:21:00.294086+0000","last_unstale":"2024-03-26T03:22:34.548872+0000","last_undegraded":"2024-03-26T03:22:34.548872+0000","last_fullsized":"2024-03-26T03:22:34.548872+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:29.083945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:29.083945+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:52.694410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1041,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553531+0000","last_change":"2024-03-26T03:22:16.900629+0000","last_active":"2024-03-26T03:22:33.553531+0000","last_peered":"2024-03-26T03:22:33.553531+0000","last_clean":"2024-03-26T03:22:33.553531+0000","last_became_active":"2024-03-26T03:20:57.363509+0000","last_became_peered":"2024-03-26T03:20:57.363509+0000","last_unstale":"2024-03-26T03:22:33.553531+0000","last_undegraded":"2024-03-26T03:22:33.553531+0000","last_fullsized":"2024-03-26T03:22:33.553531+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:16.900533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:22:16.900533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:42.851427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":663,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468555+0000","last_change":"2024-03-26T03:21:22.638381+0000","last_active":"2024-03-26T03:22:34.468555+0000","last_peered":"2024-03-26T03:22:34.468555+0000","last_clean":"2024-03-26T03:22:34.468555+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:22:34.468555+0000","last_undegraded":"2024-03-26T03:22:34.468555+0000","last_fullsized":"2024-03-26T03:22:34.468555+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:21:22.638292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:22.638292+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:44.181572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:41.038828+0000","last_change":"2024-03-26T03:22:41.038852+0000","last_active":"2024-03-26T03:22:41.038828+0000","last_peered":"2024-03-26T03:22:41.038828+0000","last_clean":"2024-03-26T03:22:41.038828+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:22:41.038828+0000","last_undegraded":"2024-03-26T03:22:41.038828+0000","last_fullsized":"2024-03-26T03:22:41.038828+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:41.038751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:41.038751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.977874+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":871,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.411202+0000","last_change":"2024-03-26T03:21:38.315997+0000","last_active":"2024-03-26T03:22:33.411202+0000","last_peered":"2024-03-26T03:22:33.411202+0000","last_clean":"2024-03-26T03:22:33.411202+0000","last_became_active":"2024-03-26T03:21:26.458997+0000","last_became_peered":"2024-03-26T03:21:26.458997+0000","last_unstale":"2024-03-26T03:22:33.411202+0000","last_undegraded":"2024-03-26T03:22:33.411202+0000","last_fullsized":"2024-03-26T03:22:33.411202+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:38.315873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.315873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.854318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"446'27","reported_seq":1091,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549444+0000","last_change":"2024-03-26T03:21:34.875552+0000","last_active":"2024-03-26T03:22:34.549444+0000","last_peered":"2024-03-26T03:22:34.549444+0000","last_clean":"2024-03-26T03:22:34.549444+0000","last_became_active":"2024-03-26T03:20:12.726379+0000","last_became_peered":"2024-03-26T03:20:12.726379+0000","last_unstale":"2024-03-26T03:22:34.549444+0000","last_undegraded":"2024-03-26T03:22:34.549444+0000","last_fullsized":"2024-03-26T03:22:34.549444+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'27","last_scrub_stamp":"2024-03-26T03:21:34.875458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:34.875458+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":791,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410255+0000","last_change":"2024-03-26T03:21:48.362289+0000","last_active":"2024-03-26T03:22:33.410255+0000","last_peered":"2024-03-26T03:22:33.410255+0000","last_clean":"2024-03-26T03:22:33.410255+0000","last_became_active":"2024-03-26T03:21:26.466136+0000","last_became_peered":"2024-03-26T03:21:26.466136+0000","last_unstale":"2024-03-26T03:22:33.410255+0000","last_undegraded":"2024-03-26T03:22:33.410255+0000","last_fullsized":"2024-03-26T03:22:33.410255+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:48.362208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:48.362208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:01.692281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2424,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:49.638615+0000","last_change":"2024-03-26T03:21:41.967044+0000","last_active":"2024-03-26T03:22:49.638615+0000","last_peered":"2024-03-26T03:22:49.638615+0000","last_clean":"2024-03-26T03:22:49.638615+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:22:49.638615+0000","last_undegraded":"2024-03-26T03:22:49.638615+0000","last_fullsized":"2024-03-26T03:22:49.638615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:21:41.966885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:41.966885+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.693689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1422,"num_read_kb":1417,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:45.555213+0000","last_change":"2024-03-26T03:22:45.555255+0000","last_active":"2024-03-26T03:22:45.555213+0000","last_peered":"2024-03-26T03:22:45.555213+0000","last_clean":"2024-03-26T03:22:45.555213+0000","last_became_active":"2024-03-26T03:20:57.362304+0000","last_became_peered":"2024-03-26T03:20:57.362304+0000","last_unstale":"2024-03-26T03:22:45.555213+0000","last_undegraded":"2024-03-26T03:22:45.555213+0000","last_fullsized":"2024-03-26T03:22:45.555213+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:45.555093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:45.555093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:46.190863+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":1073,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549404+0000","last_change":"2024-03-26T03:21:54.144030+0000","last_active":"2024-03-26T03:22:34.549404+0000","last_peered":"2024-03-26T03:22:34.549404+0000","last_clean":"2024-03-26T03:22:34.549404+0000","last_became_active":"2024-03-26T03:21:00.290993+0000","last_became_peered":"2024-03-26T03:21:00.290993+0000","last_unstale":"2024-03-26T03:22:34.549404+0000","last_undegraded":"2024-03-26T03:22:34.549404+0000","last_fullsized":"2024-03-26T03:22:34.549404+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:54.143892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:54.143892+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:58.637015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":96,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468213+0000","last_change":"2024-03-26T03:22:05.700859+0000","last_active":"2024-03-26T03:22:34.468213+0000","last_peered":"2024-03-26T03:22:34.468213+0000","last_clean":"2024-03-26T03:22:34.468213+0000","last_became_active":"2024-03-26T03:20:42.255520+0000","last_became_peered":"2024-03-26T03:20:42.255520+0000","last_unstale":"2024-03-26T03:22:34.468213+0000","last_undegraded":"2024-03-26T03:22:34.468213+0000","last_fullsized":"2024-03-26T03:22:34.468213+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:05.700775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:05.700775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:35.202375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"446'71","reported_seq":1375,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297649+0000","last_change":"2024-03-26T03:21:59.973704+0000","last_active":"2024-03-26T03:22:42.297649+0000","last_peered":"2024-03-26T03:22:42.297649+0000","last_clean":"2024-03-26T03:22:42.297649+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:22:42.297649+0000","last_undegraded":"2024-03-26T03:22:42.297649+0000","last_fullsized":"2024-03-26T03:22:42.297649+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'71","last_scrub_stamp":"2024-03-26T03:21:59.973561+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:21:59.973561+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:17.204881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":259,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553451+0000","last_change":"2024-03-26T03:22:25.002868+0000","last_active":"2024-03-26T03:22:33.553451+0000","last_peered":"2024-03-26T03:22:33.553451+0000","last_clean":"2024-03-26T03:22:33.553451+0000","last_became_active":"2024-03-26T03:21:43.857186+0000","last_became_peered":"2024-03-26T03:21:43.857186+0000","last_unstale":"2024-03-26T03:22:33.553451+0000","last_undegraded":"2024-03-26T03:22:33.553451+0000","last_fullsized":"2024-03-26T03:22:33.553451+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:25.002761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.002761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:36.907401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":680,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469404+0000","last_change":"2024-03-26T03:21:46.379636+0000","last_active":"2024-03-26T03:22:34.469404+0000","last_peered":"2024-03-26T03:22:34.469404+0000","last_clean":"2024-03-26T03:22:34.469404+0000","last_became_active":"2024-03-26T03:20:57.353414+0000","last_became_peered":"2024-03-26T03:20:57.353414+0000","last_unstale":"2024-03-26T03:22:34.469404+0000","last_undegraded":"2024-03-26T03:22:34.469404+0000","last_fullsized":"2024-03-26T03:22:34.469404+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:46.379554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.379554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:52.441795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":341,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553133+0000","last_change":"2024-03-26T03:21:45.876199+0000","last_active":"2024-03-26T03:22:33.553133+0000","last_peered":"2024-03-26T03:22:33.553133+0000","last_clean":"2024-03-26T03:22:33.553133+0000","last_became_active":"2024-03-26T03:21:43.566527+0000","last_became_peered":"2024-03-26T03:21:43.566527+0000","last_unstale":"2024-03-26T03:22:33.553133+0000","last_undegraded":"2024-03-26T03:22:33.553133+0000","last_fullsized":"2024-03-26T03:22:33.553133+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:45.876100+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:21:45.876100+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.015082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"460'13","reported_seq":163,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297586+0000","last_change":"2024-03-26T03:22:01.354367+0000","last_active":"2024-03-26T03:22:42.297586+0000","last_peered":"2024-03-26T03:22:42.297586+0000","last_clean":"2024-03-26T03:22:42.297586+0000","last_became_active":"2024-03-26T03:20:12.737843+0000","last_became_peered":"2024-03-26T03:20:12.737843+0000","last_unstale":"2024-03-26T03:22:42.297586+0000","last_undegraded":"2024-03-26T03:22:42.297586+0000","last_fullsized":"2024-03-26T03:22:42.297586+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'13","last_scrub_stamp":"2024-03-26T03:22:01.354278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:01.354278+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:15.852785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"446'42","reported_seq":1086,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548736+0000","last_change":"2024-03-26T03:22:31.944057+0000","last_active":"2024-03-26T03:22:34.548736+0000","last_peered":"2024-03-26T03:22:34.548736+0000","last_clean":"2024-03-26T03:22:34.548736+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:22:34.548736+0000","last_undegraded":"2024-03-26T03:22:34.548736+0000","last_fullsized":"2024-03-26T03:22:34.548736+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'42","last_scrub_stamp":"2024-03-26T03:22:31.943952+0000","last_deep_scrub":"446'42","last_deep_scrub_stamp":"2024-03-26T03:22:31.943952+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.943952+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:50.597560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1016,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:38.785021+0000","last_change":"2024-03-26T03:22:38.785045+0000","last_active":"2024-03-26T03:22:38.785021+0000","last_peered":"2024-03-26T03:22:38.785021+0000","last_clean":"2024-03-26T03:22:38.785021+0000","last_became_active":"2024-03-26T03:20:57.364467+0000","last_became_peered":"2024-03-26T03:20:57.364467+0000","last_unstale":"2024-03-26T03:22:38.785021+0000","last_undegraded":"2024-03-26T03:22:38.785021+0000","last_fullsized":"2024-03-26T03:22:38.785021+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:38.784961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:38.784961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.147376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":126,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.415978+0000","last_change":"2024-03-26T03:22:10.276162+0000","last_active":"2024-03-26T03:22:33.415978+0000","last_peered":"2024-03-26T03:22:33.415978+0000","last_clean":"2024-03-26T03:22:33.415978+0000","last_became_active":"2024-03-26T03:20:12.746518+0000","last_became_peered":"2024-03-26T03:20:12.746518+0000","last_unstale":"2024-03-26T03:22:33.415978+0000","last_undegraded":"2024-03-26T03:22:33.415978+0000","last_fullsized":"2024-03-26T03:22:33.415978+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:10.276075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:22:10.276075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.374414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":1079,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461133+0000","last_change":"2024-03-26T03:22:14.388219+0000","last_active":"2024-03-26T03:22:34.461133+0000","last_peered":"2024-03-26T03:22:34.461133+0000","last_clean":"2024-03-26T03:22:34.461133+0000","last_became_active":"2024-03-26T03:21:00.294477+0000","last_became_peered":"2024-03-26T03:21:00.294477+0000","last_unstale":"2024-03-26T03:22:34.461133+0000","last_undegraded":"2024-03-26T03:22:34.461133+0000","last_fullsized":"2024-03-26T03:22:34.461133+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:14.388136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:14.388136+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:15.088059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":799,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468909+0000","last_change":"2024-03-26T03:21:50.413946+0000","last_active":"2024-03-26T03:22:34.468909+0000","last_peered":"2024-03-26T03:22:34.468909+0000","last_clean":"2024-03-26T03:22:34.468909+0000","last_became_active":"2024-03-26T03:20:15.308673+0000","last_became_peered":"2024-03-26T03:20:15.308673+0000","last_unstale":"2024-03-26T03:22:34.468909+0000","last_undegraded":"2024-03-26T03:22:34.468909+0000","last_fullsized":"2024-03-26T03:22:34.468909+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:50.413841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:50.413841+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:10.104423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":816,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409814+0000","last_change":"2024-03-26T03:22:05.568746+0000","last_active":"2024-03-26T03:22:33.409814+0000","last_peered":"2024-03-26T03:22:33.409814+0000","last_clean":"2024-03-26T03:22:33.409814+0000","last_became_active":"2024-03-26T03:21:26.464824+0000","last_became_peered":"2024-03-26T03:21:26.464824+0000","last_unstale":"2024-03-26T03:22:33.409814+0000","last_undegraded":"2024-03-26T03:22:33.409814+0000","last_fullsized":"2024-03-26T03:22:33.409814+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:05.568661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:05.568661+0000","last_clean_scrub_stamp":"2024-03-26T03:22:05.568661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:29.644568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"474'43","reported_seq":1057,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409889+0000","last_change":"2024-03-26T03:22:11.351736+0000","last_active":"2024-03-26T03:22:33.409889+0000","last_peered":"2024-03-26T03:22:33.409889+0000","last_clean":"2024-03-26T03:22:33.409889+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:22:33.409889+0000","last_undegraded":"2024-03-26T03:22:33.409889+0000","last_fullsized":"2024-03-26T03:22:33.409889+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'43","last_scrub_stamp":"2024-03-26T03:22:11.351575+0000","last_deep_scrub":"474'43","last_deep_scrub_stamp":"2024-03-26T03:20:49.325072+0000","last_clean_scrub_stamp":"2024-03-26T03:22:11.351575+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:14.351160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":1069,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410677+0000","last_change":"2024-03-26T03:21:10.350400+0000","last_active":"2024-03-26T03:22:33.410677+0000","last_peered":"2024-03-26T03:22:33.410677+0000","last_clean":"2024-03-26T03:22:33.410677+0000","last_became_active":"2024-03-26T03:20:57.358824+0000","last_became_peered":"2024-03-26T03:20:57.358824+0000","last_unstale":"2024-03-26T03:22:33.410677+0000","last_undegraded":"2024-03-26T03:22:33.410677+0000","last_fullsized":"2024-03-26T03:22:33.410677+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:21:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:10.350308+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:34.618269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":661,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.793412+0000","last_change":"2024-03-26T03:22:47.793436+0000","last_active":"2024-03-26T03:22:47.793412+0000","last_peered":"2024-03-26T03:22:47.793412+0000","last_clean":"2024-03-26T03:22:47.793412+0000","last_became_active":"2024-03-26T03:20:57.354428+0000","last_became_peered":"2024-03-26T03:20:57.354428+0000","last_unstale":"2024-03-26T03:22:47.793412+0000","last_undegraded":"2024-03-26T03:22:47.793412+0000","last_fullsized":"2024-03-26T03:22:47.793412+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:47.793341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:47.793341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.045908+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":1109,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548828+0000","last_change":"2024-03-26T03:22:22.986714+0000","last_active":"2024-03-26T03:22:34.548828+0000","last_peered":"2024-03-26T03:22:34.548828+0000","last_clean":"2024-03-26T03:22:34.548828+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:22:34.548828+0000","last_undegraded":"2024-03-26T03:22:34.548828+0000","last_fullsized":"2024-03-26T03:22:34.548828+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:22.986623+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:22:22.986623+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:46.327164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"446'33","reported_seq":160,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.417141+0000","last_change":"2024-03-26T03:22:25.446976+0000","last_active":"2024-03-26T03:22:33.417141+0000","last_peered":"2024-03-26T03:22:33.417141+0000","last_clean":"2024-03-26T03:22:33.417141+0000","last_became_active":"2024-03-26T03:20:57.355137+0000","last_became_peered":"2024-03-26T03:20:57.355137+0000","last_unstale":"2024-03-26T03:22:33.417141+0000","last_undegraded":"2024-03-26T03:22:33.417141+0000","last_fullsized":"2024-03-26T03:22:33.417141+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'33","last_scrub_stamp":"2024-03-26T03:22:25.446872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.446872+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:40.235392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":843,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409696+0000","last_change":"2024-03-26T03:22:31.539414+0000","last_active":"2024-03-26T03:22:33.409696+0000","last_peered":"2024-03-26T03:22:33.409696+0000","last_clean":"2024-03-26T03:22:33.409696+0000","last_became_active":"2024-03-26T03:21:26.462548+0000","last_became_peered":"2024-03-26T03:21:26.462548+0000","last_unstale":"2024-03-26T03:22:33.409696+0000","last_undegraded":"2024-03-26T03:22:33.409696+0000","last_fullsized":"2024-03-26T03:22:33.409696+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:22:31.539325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.539325+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.560057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":678,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469404+0000","last_change":"2024-03-26T03:21:32.419713+0000","last_active":"2024-03-26T03:22:34.469404+0000","last_peered":"2024-03-26T03:22:34.469404+0000","last_clean":"2024-03-26T03:22:34.469404+0000","last_became_active":"2024-03-26T03:20:57.352698+0000","last_became_peered":"2024-03-26T03:20:57.352698+0000","last_unstale":"2024-03-26T03:22:34.469404+0000","last_undegraded":"2024-03-26T03:22:34.469404+0000","last_fullsized":"2024-03-26T03:22:34.469404+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:32.419625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:32.419625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.198321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549877+0000","last_change":"2024-03-26T03:21:24.087182+0000","last_active":"2024-03-26T03:22:34.549877+0000","last_peered":"2024-03-26T03:22:34.549877+0000","last_clean":"2024-03-26T03:22:34.549877+0000","last_became_active":"2024-03-26T03:20:57.363228+0000","last_became_peered":"2024-03-26T03:20:57.363228+0000","last_unstale":"2024-03-26T03:22:34.549877+0000","last_undegraded":"2024-03-26T03:22:34.549877+0000","last_fullsized":"2024-03-26T03:22:34.549877+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:24.087097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:21:24.087097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:47.157486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":504,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:48.398162+0000","last_change":"2024-03-26T03:22:48.398189+0000","last_active":"2024-03-26T03:22:48.398162+0000","last_peered":"2024-03-26T03:22:48.398162+0000","last_clean":"2024-03-26T03:22:48.398162+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:22:48.398162+0000","last_undegraded":"2024-03-26T03:22:48.398162+0000","last_fullsized":"2024-03-26T03:22:48.398162+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:48.398091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:48.398091+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:07.822795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"446'24","reported_seq":764,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469172+0000","last_change":"2024-03-26T03:22:10.710050+0000","last_active":"2024-03-26T03:22:34.469172+0000","last_peered":"2024-03-26T03:22:34.469172+0000","last_clean":"2024-03-26T03:22:34.469172+0000","last_became_active":"2024-03-26T03:20:57.353003+0000","last_became_peered":"2024-03-26T03:20:57.353003+0000","last_unstale":"2024-03-26T03:22:34.469172+0000","last_undegraded":"2024-03-26T03:22:34.469172+0000","last_fullsized":"2024-03-26T03:22:34.469172+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'24","last_scrub_stamp":"2024-03-26T03:22:10.709951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:10.709951+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:14.913269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1011,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:39.550697+0000","last_change":"2024-03-26T03:22:39.550722+0000","last_active":"2024-03-26T03:22:39.550697+0000","last_peered":"2024-03-26T03:22:39.550697+0000","last_clean":"2024-03-26T03:22:39.550697+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:22:39.550697+0000","last_undegraded":"2024-03-26T03:22:39.550697+0000","last_fullsized":"2024-03-26T03:22:39.550697+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:39.550620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:39.550620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:04.524735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":974,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553863+0000","last_change":"2024-03-26T03:21:50.898485+0000","last_active":"2024-03-26T03:22:33.553863+0000","last_peered":"2024-03-26T03:22:33.553863+0000","last_clean":"2024-03-26T03:22:33.553863+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:22:33.553863+0000","last_undegraded":"2024-03-26T03:22:33.553863+0000","last_fullsized":"2024-03-26T03:22:33.553863+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:50.898381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:50.898381+0000","last_clean_scrub_stamp":"2024-03-26T03:21:50.898381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:17.188243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.467068+0000","last_change":"2024-03-26T03:22:37.467092+0000","last_active":"2024-03-26T03:22:37.467068+0000","last_peered":"2024-03-26T03:22:37.467068+0000","last_clean":"2024-03-26T03:22:37.467068+0000","last_became_active":"2024-03-26T03:20:57.379539+0000","last_became_peered":"2024-03-26T03:20:57.379539+0000","last_unstale":"2024-03-26T03:22:37.467068+0000","last_undegraded":"2024-03-26T03:22:37.467068+0000","last_fullsized":"2024-03-26T03:22:37.467068+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:37.466985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.466985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:44.153349+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":430,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416144+0000","last_change":"2024-03-26T03:22:30.534188+0000","last_active":"2024-03-26T03:22:33.416144+0000","last_peered":"2024-03-26T03:22:33.416144+0000","last_clean":"2024-03-26T03:22:33.416144+0000","last_became_active":"2024-03-26T03:20:15.317985+0000","last_became_peered":"2024-03-26T03:20:15.317985+0000","last_unstale":"2024-03-26T03:22:33.416144+0000","last_undegraded":"2024-03-26T03:22:33.416144+0000","last_fullsized":"2024-03-26T03:22:33.416144+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:30.534097+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:22:30.534097+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:31.820328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"446'49","reported_seq":793,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468814+0000","last_change":"2024-03-26T03:22:23.640850+0000","last_active":"2024-03-26T03:22:34.468814+0000","last_peered":"2024-03-26T03:22:34.468814+0000","last_clean":"2024-03-26T03:22:34.468814+0000","last_became_active":"2024-03-26T03:20:57.353406+0000","last_became_peered":"2024-03-26T03:20:57.353406+0000","last_unstale":"2024-03-26T03:22:34.468814+0000","last_undegraded":"2024-03-26T03:22:34.468814+0000","last_fullsized":"2024-03-26T03:22:34.468814+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'49","last_scrub_stamp":"2024-03-26T03:22:23.640754+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:22:23.640754+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:26.013929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553346+0000","last_change":"2024-03-26T03:22:05.898382+0000","last_active":"2024-03-26T03:22:33.553346+0000","last_peered":"2024-03-26T03:22:33.553346+0000","last_clean":"2024-03-26T03:22:33.553346+0000","last_became_active":"2024-03-26T03:20:57.365743+0000","last_became_peered":"2024-03-26T03:20:57.365743+0000","last_unstale":"2024-03-26T03:22:33.553346+0000","last_undegraded":"2024-03-26T03:22:33.553346+0000","last_fullsized":"2024-03-26T03:22:33.553346+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:05.898268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:22:05.898268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.243361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"471'15","reported_seq":1338,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297668+0000","last_change":"2024-03-26T03:22:37.947982+0000","last_active":"2024-03-26T03:22:42.297668+0000","last_peered":"2024-03-26T03:22:42.297668+0000","last_clean":"2024-03-26T03:22:42.297668+0000","last_became_active":"2024-03-26T03:20:42.255252+0000","last_became_peered":"2024-03-26T03:20:42.255252+0000","last_unstale":"2024-03-26T03:22:42.297668+0000","last_undegraded":"2024-03-26T03:22:42.297668+0000","last_fullsized":"2024-03-26T03:22:42.297668+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'15","last_scrub_stamp":"2024-03-26T03:22:37.947825+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.947825+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:48.429164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.417041+0000","last_change":"2024-03-26T03:22:27.472268+0000","last_active":"2024-03-26T03:22:33.417041+0000","last_peered":"2024-03-26T03:22:33.417041+0000","last_clean":"2024-03-26T03:22:33.417041+0000","last_became_active":"2024-03-26T03:20:57.357744+0000","last_became_peered":"2024-03-26T03:20:57.357744+0000","last_unstale":"2024-03-26T03:22:33.417041+0000","last_undegraded":"2024-03-26T03:22:33.417041+0000","last_fullsized":"2024-03-26T03:22:33.417041+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:27.472152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.472152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.703760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":739,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553231+0000","last_change":"2024-03-26T03:21:46.887888+0000","last_active":"2024-03-26T03:22:33.553231+0000","last_peered":"2024-03-26T03:22:33.553231+0000","last_clean":"2024-03-26T03:22:33.553231+0000","last_became_active":"2024-03-26T03:21:43.566515+0000","last_became_peered":"2024-03-26T03:21:43.566515+0000","last_unstale":"2024-03-26T03:22:33.553231+0000","last_undegraded":"2024-03-26T03:22:33.553231+0000","last_fullsized":"2024-03-26T03:22:33.553231+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:46.887750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.887750+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:05.129158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"475'32","reported_seq":777,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553851+0000","last_change":"2024-03-26T03:22:19.912662+0000","last_active":"2024-03-26T03:22:33.553851+0000","last_peered":"2024-03-26T03:22:33.553851+0000","last_clean":"2024-03-26T03:22:33.553851+0000","last_became_active":"2024-03-26T03:21:43.568013+0000","last_became_peered":"2024-03-26T03:21:43.568013+0000","last_unstale":"2024-03-26T03:22:33.553851+0000","last_undegraded":"2024-03-26T03:22:33.553851+0000","last_fullsized":"2024-03-26T03:22:33.553851+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'32","last_scrub_stamp":"2024-03-26T03:22:19.912549+0000","last_deep_scrub":"475'32","last_deep_scrub_stamp":"2024-03-26T03:20:54.654445+0000","last_clean_scrub_stamp":"2024-03-26T03:22:19.912549+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.469868+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":134,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:50.518175+0000","last_change":"2024-03-26T03:22:50.518215+0000","last_active":"2024-03-26T03:22:50.518175+0000","last_peered":"2024-03-26T03:22:50.518175+0000","last_clean":"2024-03-26T03:22:50.518175+0000","last_became_active":"2024-03-26T03:20:12.746773+0000","last_became_peered":"2024-03-26T03:20:12.746773+0000","last_unstale":"2024-03-26T03:22:50.518175+0000","last_undegraded":"2024-03-26T03:22:50.518175+0000","last_fullsized":"2024-03-26T03:22:50.518175+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:50.518036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:22:50.518036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.225697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"477'12","reported_seq":818,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297608+0000","last_change":"2024-03-26T03:21:15.550286+0000","last_active":"2024-03-26T03:22:42.297608+0000","last_peered":"2024-03-26T03:22:42.297608+0000","last_clean":"2024-03-26T03:22:42.297608+0000","last_became_active":"2024-03-26T03:20:57.362408+0000","last_became_peered":"2024-03-26T03:20:57.362408+0000","last_unstale":"2024-03-26T03:22:42.297608+0000","last_undegraded":"2024-03-26T03:22:42.297608+0000","last_fullsized":"2024-03-26T03:22:42.297608+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'12","last_scrub_stamp":"2024-03-26T03:21:15.550162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:15.550162+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":961,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410834+0000","last_change":"2024-03-26T03:21:34.659309+0000","last_active":"2024-03-26T03:22:33.410834+0000","last_peered":"2024-03-26T03:22:33.410834+0000","last_clean":"2024-03-26T03:22:33.410834+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:22:33.410834+0000","last_undegraded":"2024-03-26T03:22:33.410834+0000","last_fullsized":"2024-03-26T03:22:33.410834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:21:34.659199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:34.659199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:46.637045+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":1098,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553988+0000","last_change":"2024-03-26T03:22:25.985505+0000","last_active":"2024-03-26T03:22:33.553988+0000","last_peered":"2024-03-26T03:22:33.553988+0000","last_clean":"2024-03-26T03:22:33.553988+0000","last_became_active":"2024-03-26T03:20:15.295795+0000","last_became_peered":"2024-03-26T03:20:15.295795+0000","last_unstale":"2024-03-26T03:22:33.553988+0000","last_undegraded":"2024-03-26T03:22:33.553988+0000","last_fullsized":"2024-03-26T03:22:33.553988+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:25.985407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.985407+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:50.668876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"446'25","reported_seq":113,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:35.493636+0000","last_change":"2024-03-26T03:22:35.493666+0000","last_active":"2024-03-26T03:22:35.493636+0000","last_peered":"2024-03-26T03:22:35.493636+0000","last_clean":"2024-03-26T03:22:35.493636+0000","last_became_active":"2024-03-26T03:20:42.258677+0000","last_became_peered":"2024-03-26T03:20:42.258677+0000","last_unstale":"2024-03-26T03:22:35.493636+0000","last_undegraded":"2024-03-26T03:22:35.493636+0000","last_fullsized":"2024-03-26T03:22:35.493636+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'25","last_scrub_stamp":"2024-03-26T03:22:35.493575+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:22:35.493575+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.175400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":968,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.460690+0000","last_change":"2024-03-26T03:21:29.301574+0000","last_active":"2024-03-26T03:22:34.460690+0000","last_peered":"2024-03-26T03:22:34.460690+0000","last_clean":"2024-03-26T03:22:34.460690+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:22:34.460690+0000","last_undegraded":"2024-03-26T03:22:34.460690+0000","last_fullsized":"2024-03-26T03:22:34.460690+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:29.301492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:29.301492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:50.554391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":663,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.560923+0000","last_change":"2024-03-26T03:22:34.560957+0000","last_active":"2024-03-26T03:22:34.560923+0000","last_peered":"2024-03-26T03:22:34.560923+0000","last_clean":"2024-03-26T03:22:34.560923+0000","last_became_active":"2024-03-26T03:20:12.749158+0000","last_became_peered":"2024-03-26T03:20:12.749158+0000","last_unstale":"2024-03-26T03:22:34.560923+0000","last_undegraded":"2024-03-26T03:22:34.560923+0000","last_fullsized":"2024-03-26T03:22:34.560923+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:34.560837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:47.697347+0000","last_clean_scrub_stamp":"2024-03-26T03:22:34.560837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":126,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.415940+0000","last_change":"2024-03-26T03:21:55.318317+0000","last_active":"2024-03-26T03:22:33.415940+0000","last_peered":"2024-03-26T03:22:33.415940+0000","last_clean":"2024-03-26T03:22:33.415940+0000","last_became_active":"2024-03-26T03:20:12.736042+0000","last_became_peered":"2024-03-26T03:20:12.736042+0000","last_unstale":"2024-03-26T03:22:33.415940+0000","last_undegraded":"2024-03-26T03:22:33.415940+0000","last_fullsized":"2024-03-26T03:22:33.415940+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:55.318231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:55.318231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:08.239770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":766,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.819993+0000","last_change":"2024-03-26T03:22:37.820018+0000","last_active":"2024-03-26T03:22:37.819993+0000","last_peered":"2024-03-26T03:22:37.819993+0000","last_clean":"2024-03-26T03:22:37.819993+0000","last_became_active":"2024-03-26T03:20:15.311071+0000","last_became_peered":"2024-03-26T03:20:15.311071+0000","last_unstale":"2024-03-26T03:22:37.819993+0000","last_undegraded":"2024-03-26T03:22:37.819993+0000","last_fullsized":"2024-03-26T03:22:37.819993+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:37.819923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.819923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.347791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"446'27","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410235+0000","last_change":"2024-03-26T03:22:27.152212+0000","last_active":"2024-03-26T03:22:33.410235+0000","last_peered":"2024-03-26T03:22:33.410235+0000","last_clean":"2024-03-26T03:22:33.410235+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:22:33.410235+0000","last_undegraded":"2024-03-26T03:22:33.410235+0000","last_fullsized":"2024-03-26T03:22:33.410235+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'27","last_scrub_stamp":"2024-03-26T03:22:27.152084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.152084+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:40.367990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1010,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.785363+0000","last_change":"2024-03-26T03:22:47.785383+0000","last_active":"2024-03-26T03:22:47.785363+0000","last_peered":"2024-03-26T03:22:47.785363+0000","last_clean":"2024-03-26T03:22:47.785363+0000","last_became_active":"2024-03-26T03:20:12.720143+0000","last_became_peered":"2024-03-26T03:20:12.720143+0000","last_unstale":"2024-03-26T03:22:47.785363+0000","last_undegraded":"2024-03-26T03:22:47.785363+0000","last_fullsized":"2024-03-26T03:22:47.785363+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:47.785304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:47.785304+0000","last_clean_scrub_stamp":"2024-03-26T03:22:47.785304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:14.870263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":952,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548848+0000","last_change":"2024-03-26T03:22:04.988448+0000","last_active":"2024-03-26T03:22:34.548848+0000","last_peered":"2024-03-26T03:22:34.548848+0000","last_clean":"2024-03-26T03:22:34.548848+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:22:34.548848+0000","last_undegraded":"2024-03-26T03:22:34.548848+0000","last_fullsized":"2024-03-26T03:22:34.548848+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:04.988369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:04.988369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.111872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":667,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468350+0000","last_change":"2024-03-26T03:22:08.658538+0000","last_active":"2024-03-26T03:22:34.468350+0000","last_peered":"2024-03-26T03:22:34.468350+0000","last_clean":"2024-03-26T03:22:34.468350+0000","last_became_active":"2024-03-26T03:20:12.745809+0000","last_became_peered":"2024-03-26T03:20:12.745809+0000","last_unstale":"2024-03-26T03:22:34.468350+0000","last_undegraded":"2024-03-26T03:22:34.468350+0000","last_fullsized":"2024-03-26T03:22:34.468350+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:08.658406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:08.658406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.316878+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":675,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549778+0000","last_change":"2024-03-26T03:21:58.207690+0000","last_active":"2024-03-26T03:22:34.549778+0000","last_peered":"2024-03-26T03:22:34.549778+0000","last_clean":"2024-03-26T03:22:34.549778+0000","last_became_active":"2024-03-26T03:21:00.297045+0000","last_became_peered":"2024-03-26T03:21:00.297045+0000","last_unstale":"2024-03-26T03:22:34.549778+0000","last_undegraded":"2024-03-26T03:22:34.549778+0000","last_fullsized":"2024-03-26T03:22:34.549778+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:58.207539+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:21:58.207539+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:17.663822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"465'14","reported_seq":686,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468339+0000","last_change":"2024-03-26T03:21:52.468863+0000","last_active":"2024-03-26T03:22:34.468339+0000","last_peered":"2024-03-26T03:22:34.468339+0000","last_clean":"2024-03-26T03:22:34.468339+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:22:34.468339+0000","last_undegraded":"2024-03-26T03:22:34.468339+0000","last_fullsized":"2024-03-26T03:22:34.468339+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'14","last_scrub_stamp":"2024-03-26T03:21:52.468721+0000","last_deep_scrub":"465'14","last_deep_scrub_stamp":"2024-03-26T03:20:20.501710+0000","last_clean_scrub_stamp":"2024-03-26T03:21:52.468721+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:04.599700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1013,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553992+0000","last_change":"2024-03-26T03:22:13.861284+0000","last_active":"2024-03-26T03:22:33.553992+0000","last_peered":"2024-03-26T03:22:33.553992+0000","last_clean":"2024-03-26T03:22:33.553992+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:22:33.553992+0000","last_undegraded":"2024-03-26T03:22:33.553992+0000","last_fullsized":"2024-03-26T03:22:33.553992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:13.861207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.861207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:18.097192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.460444+0000","last_change":"2024-03-26T03:22:02.352570+0000","last_active":"2024-03-26T03:22:34.460444+0000","last_peered":"2024-03-26T03:22:34.460444+0000","last_clean":"2024-03-26T03:22:34.460444+0000","last_became_active":"2024-03-26T03:20:57.356298+0000","last_became_peered":"2024-03-26T03:20:57.356298+0000","last_unstale":"2024-03-26T03:22:34.460444+0000","last_undegraded":"2024-03-26T03:22:34.460444+0000","last_fullsized":"2024-03-26T03:22:34.460444+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:02.352403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:02.352403+0000","last_clean_scrub_stamp":"2024-03-26T03:22:02.352403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:16.110906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549548+0000","last_change":"2024-03-26T03:22:33.298660+0000","last_active":"2024-03-26T03:22:34.549548+0000","last_peered":"2024-03-26T03:22:34.549548+0000","last_clean":"2024-03-26T03:22:34.549548+0000","last_became_active":"2024-03-26T03:20:57.373433+0000","last_became_peered":"2024-03-26T03:20:57.373433+0000","last_unstale":"2024-03-26T03:22:34.549548+0000","last_undegraded":"2024-03-26T03:22:34.549548+0000","last_fullsized":"2024-03-26T03:22:34.549548+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:33.298546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:33.298546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:52.064969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":695,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.395973+0000","last_change":"2024-03-26T03:22:42.395996+0000","last_active":"2024-03-26T03:22:42.395973+0000","last_peered":"2024-03-26T03:22:42.395973+0000","last_clean":"2024-03-26T03:22:42.395973+0000","last_became_active":"2024-03-26T03:21:26.464254+0000","last_became_peered":"2024-03-26T03:21:26.464254+0000","last_unstale":"2024-03-26T03:22:42.395973+0000","last_undegraded":"2024-03-26T03:22:42.395973+0000","last_fullsized":"2024-03-26T03:22:42.395973+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:42.395913+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:22:42.395913+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.395913+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.577520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"446'39","reported_seq":773,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:43.611535+0000","last_change":"2024-03-26T03:22:43.611573+0000","last_active":"2024-03-26T03:22:43.611535+0000","last_peered":"2024-03-26T03:22:43.611535+0000","last_clean":"2024-03-26T03:22:43.611535+0000","last_became_active":"2024-03-26T03:20:57.359700+0000","last_became_peered":"2024-03-26T03:20:57.359700+0000","last_unstale":"2024-03-26T03:22:43.611535+0000","last_undegraded":"2024-03-26T03:22:43.611535+0000","last_fullsized":"2024-03-26T03:22:43.611535+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:22:43.611473+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:22:43.611473+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:52.597828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":990,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461248+0000","last_change":"2024-03-26T03:21:51.249951+0000","last_active":"2024-03-26T03:22:34.461248+0000","last_peered":"2024-03-26T03:22:34.461248+0000","last_clean":"2024-03-26T03:22:34.461248+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:22:34.461248+0000","last_undegraded":"2024-03-26T03:22:34.461248+0000","last_fullsized":"2024-03-26T03:22:34.461248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:51.249857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:51.249857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:55.256531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549153+0000","last_change":"2024-03-26T03:21:26.056227+0000","last_active":"2024-03-26T03:22:34.549153+0000","last_peered":"2024-03-26T03:22:34.549153+0000","last_clean":"2024-03-26T03:22:34.549153+0000","last_became_active":"2024-03-26T03:20:57.377365+0000","last_became_peered":"2024-03-26T03:20:57.377365+0000","last_unstale":"2024-03-26T03:22:34.549153+0000","last_undegraded":"2024-03-26T03:22:34.549153+0000","last_fullsized":"2024-03-26T03:22:34.549153+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:26.056138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:21:26.056138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:51.770560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":653,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468708+0000","last_change":"2024-03-26T03:21:45.434621+0000","last_active":"2024-03-26T03:22:34.468708+0000","last_peered":"2024-03-26T03:22:34.468708+0000","last_clean":"2024-03-26T03:22:34.468708+0000","last_became_active":"2024-03-26T03:20:12.747304+0000","last_became_peered":"2024-03-26T03:20:12.747304+0000","last_unstale":"2024-03-26T03:22:34.468708+0000","last_undegraded":"2024-03-26T03:22:34.468708+0000","last_fullsized":"2024-03-26T03:22:34.468708+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:45.434520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:45.434520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.328405+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":1001,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553310+0000","last_change":"2024-03-26T03:21:47.914010+0000","last_active":"2024-03-26T03:22:33.553310+0000","last_peered":"2024-03-26T03:22:33.553310+0000","last_clean":"2024-03-26T03:22:33.553310+0000","last_became_active":"2024-03-26T03:20:15.299184+0000","last_became_peered":"2024-03-26T03:20:15.299184+0000","last_unstale":"2024-03-26T03:22:33.553310+0000","last_undegraded":"2024-03-26T03:22:33.553310+0000","last_fullsized":"2024-03-26T03:22:33.553310+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:47.913887+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:21:47.913887+0000","last_clean_scrub_stamp":"2024-03-26T03:21:47.913887+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.198817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"446'36","reported_seq":180,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416677+0000","last_change":"2024-03-26T03:22:20.484470+0000","last_active":"2024-03-26T03:22:33.416677+0000","last_peered":"2024-03-26T03:22:33.416677+0000","last_clean":"2024-03-26T03:22:33.416677+0000","last_became_active":"2024-03-26T03:20:57.358076+0000","last_became_peered":"2024-03-26T03:20:57.358076+0000","last_unstale":"2024-03-26T03:22:33.416677+0000","last_undegraded":"2024-03-26T03:22:33.416677+0000","last_fullsized":"2024-03-26T03:22:33.416677+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:22:20.484360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:20.484360+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.974063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553105+0000","last_change":"2024-03-26T03:21:41.909829+0000","last_active":"2024-03-26T03:22:33.553105+0000","last_peered":"2024-03-26T03:22:33.553105+0000","last_clean":"2024-03-26T03:22:33.553105+0000","last_became_active":"2024-03-26T03:20:57.364349+0000","last_became_peered":"2024-03-26T03:20:57.364349+0000","last_unstale":"2024-03-26T03:22:33.553105+0000","last_undegraded":"2024-03-26T03:22:33.553105+0000","last_fullsized":"2024-03-26T03:22:33.553105+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:41.909728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:21:41.909728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.917095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"477'12","reported_seq":1331,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.297659+0000","last_change":"2024-03-26T03:22:36.548859+0000","last_active":"2024-03-26T03:22:42.297659+0000","last_peered":"2024-03-26T03:22:42.297659+0000","last_clean":"2024-03-26T03:22:42.297659+0000","last_became_active":"2024-03-26T03:20:57.371266+0000","last_became_peered":"2024-03-26T03:20:57.371266+0000","last_unstale":"2024-03-26T03:22:42.297659+0000","last_undegraded":"2024-03-26T03:22:42.297659+0000","last_fullsized":"2024-03-26T03:22:42.297659+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'12","last_scrub_stamp":"2024-03-26T03:22:36.548751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:36.548751+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.227425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":86,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549662+0000","last_change":"2024-03-26T03:21:38.023218+0000","last_active":"2024-03-26T03:22:34.549662+0000","last_peered":"2024-03-26T03:22:34.549662+0000","last_clean":"2024-03-26T03:22:34.549662+0000","last_became_active":"2024-03-26T03:20:57.371714+0000","last_became_peered":"2024-03-26T03:20:57.371714+0000","last_unstale":"2024-03-26T03:22:34.549662+0000","last_undegraded":"2024-03-26T03:22:34.549662+0000","last_fullsized":"2024-03-26T03:22:34.549662+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:21:38.023081+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.023081+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.812426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":979,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409946+0000","last_change":"2024-03-26T03:22:19.243148+0000","last_active":"2024-03-26T03:22:33.409946+0000","last_peered":"2024-03-26T03:22:33.409946+0000","last_clean":"2024-03-26T03:22:33.409946+0000","last_became_active":"2024-03-26T03:21:00.288286+0000","last_became_peered":"2024-03-26T03:21:00.288286+0000","last_unstale":"2024-03-26T03:22:33.409946+0000","last_undegraded":"2024-03-26T03:22:33.409946+0000","last_fullsized":"2024-03-26T03:22:33.409946+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:19.243044+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:22:19.243044+0000","last_clean_scrub_stamp":"2024-03-26T03:22:19.243044+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:39.008270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"507'39","reported_seq":1049,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461424+0000","last_change":"2024-03-26T03:22:13.421938+0000","last_active":"2024-03-26T03:22:34.461424+0000","last_peered":"2024-03-26T03:22:34.461424+0000","last_clean":"2024-03-26T03:22:34.461424+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:22:34.461424+0000","last_undegraded":"2024-03-26T03:22:34.461424+0000","last_fullsized":"2024-03-26T03:22:34.461424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'39","last_scrub_stamp":"2024-03-26T03:22:13.421808+0000","last_deep_scrub":"507'39","last_deep_scrub_stamp":"2024-03-26T03:22:13.421808+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.421808+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:26.103085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"64'1","reported_seq":630,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468670+0000","last_change":"2024-03-26T03:20:58.719921+0000","last_active":"2024-03-26T03:22:34.468670+0000","last_peered":"2024-03-26T03:22:34.468670+0000","last_clean":"2024-03-26T03:22:34.468670+0000","last_became_active":"2024-03-26T03:20:12.746319+0000","last_became_peered":"2024-03-26T03:20:12.746319+0000","last_unstale":"2024-03-26T03:22:34.468670+0000","last_undegraded":"2024-03-26T03:22:34.468670+0000","last_fullsized":"2024-03-26T03:22:34.468670+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:20:58.719791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:20:58.719791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":140,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.448388+0000","last_change":"2024-03-26T03:22:37.448411+0000","last_active":"2024-03-26T03:22:37.448388+0000","last_peered":"2024-03-26T03:22:37.448388+0000","last_clean":"2024-03-26T03:22:37.448388+0000","last_became_active":"2024-03-26T03:20:57.355510+0000","last_became_peered":"2024-03-26T03:20:57.355510+0000","last_unstale":"2024-03-26T03:22:37.448388+0000","last_undegraded":"2024-03-26T03:22:37.448388+0000","last_fullsized":"2024-03-26T03:22:37.448388+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:37.448313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.448313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:02.343970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":982,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549390+0000","last_change":"2024-03-26T03:22:31.174605+0000","last_active":"2024-03-26T03:22:34.549390+0000","last_peered":"2024-03-26T03:22:34.549390+0000","last_clean":"2024-03-26T03:22:34.549390+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:22:34.549390+0000","last_undegraded":"2024-03-26T03:22:34.549390+0000","last_fullsized":"2024-03-26T03:22:34.549390+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:31.174494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.174494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:39.360753+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":883,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468820+0000","last_change":"2024-03-26T03:22:02.691708+0000","last_active":"2024-03-26T03:22:34.468820+0000","last_peered":"2024-03-26T03:22:34.468820+0000","last_clean":"2024-03-26T03:22:34.468820+0000","last_became_active":"2024-03-26T03:20:15.336658+0000","last_became_peered":"2024-03-26T03:20:15.336658+0000","last_unstale":"2024-03-26T03:22:34.468820+0000","last_undegraded":"2024-03-26T03:22:34.468820+0000","last_fullsized":"2024-03-26T03:22:34.468820+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:02.691612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:02.691612+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:12.758244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"446'39","reported_seq":587,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.412675+0000","last_change":"2024-03-26T03:22:25.563778+0000","last_active":"2024-03-26T03:22:33.412675+0000","last_peered":"2024-03-26T03:22:33.412675+0000","last_clean":"2024-03-26T03:22:33.412675+0000","last_became_active":"2024-03-26T03:21:26.470153+0000","last_became_peered":"2024-03-26T03:21:26.470153+0000","last_unstale":"2024-03-26T03:22:33.412675+0000","last_undegraded":"2024-03-26T03:22:33.412675+0000","last_fullsized":"2024-03-26T03:22:33.412675+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:22:25.563648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.563648+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.508537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.530584+0000","last_change":"2024-03-26T03:22:47.530609+0000","last_active":"2024-03-26T03:22:47.530584+0000","last_peered":"2024-03-26T03:22:47.530584+0000","last_clean":"2024-03-26T03:22:47.530584+0000","last_became_active":"2024-03-26T03:20:57.356308+0000","last_became_peered":"2024-03-26T03:20:57.356308+0000","last_unstale":"2024-03-26T03:22:47.530584+0000","last_undegraded":"2024-03-26T03:22:47.530584+0000","last_fullsized":"2024-03-26T03:22:47.530584+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:47.530511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:47.530511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:03.775114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":275,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.460938+0000","last_change":"2024-03-26T03:21:50.202903+0000","last_active":"2024-03-26T03:22:34.460938+0000","last_peered":"2024-03-26T03:22:34.460938+0000","last_clean":"2024-03-26T03:22:34.460938+0000","last_became_active":"2024-03-26T03:20:12.733918+0000","last_became_peered":"2024-03-26T03:20:12.733918+0000","last_unstale":"2024-03-26T03:22:34.460938+0000","last_undegraded":"2024-03-26T03:22:34.460938+0000","last_fullsized":"2024-03-26T03:22:34.460938+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:50.202792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:21:50.202792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:50.547449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":970,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410718+0000","last_change":"2024-03-26T03:21:56.398490+0000","last_active":"2024-03-26T03:22:33.410718+0000","last_peered":"2024-03-26T03:22:33.410718+0000","last_clean":"2024-03-26T03:22:33.410718+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:22:33.410718+0000","last_undegraded":"2024-03-26T03:22:33.410718+0000","last_fullsized":"2024-03-26T03:22:33.410718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:56.398396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:56.398396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:16.749380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":790,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553482+0000","last_change":"2024-03-26T03:21:38.978299+0000","last_active":"2024-03-26T03:22:33.553482+0000","last_peered":"2024-03-26T03:22:33.553482+0000","last_clean":"2024-03-26T03:22:33.553482+0000","last_became_active":"2024-03-26T03:21:00.296006+0000","last_became_peered":"2024-03-26T03:21:00.296006+0000","last_unstale":"2024-03-26T03:22:33.553482+0000","last_undegraded":"2024-03-26T03:22:33.553482+0000","last_fullsized":"2024-03-26T03:22:33.553482+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:21:38.978193+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.978193+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:59.206936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"446'36","reported_seq":728,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468867+0000","last_change":"2024-03-26T03:22:16.553402+0000","last_active":"2024-03-26T03:22:34.468867+0000","last_peered":"2024-03-26T03:22:34.468867+0000","last_clean":"2024-03-26T03:22:34.468867+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:22:34.468867+0000","last_undegraded":"2024-03-26T03:22:34.468867+0000","last_fullsized":"2024-03-26T03:22:34.468867+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:22:16.553295+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:22:16.553295+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.341579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":692,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468824+0000","last_change":"2024-03-26T03:21:47.342963+0000","last_active":"2024-03-26T03:22:34.468824+0000","last_peered":"2024-03-26T03:22:34.468824+0000","last_clean":"2024-03-26T03:22:34.468824+0000","last_became_active":"2024-03-26T03:20:57.353803+0000","last_became_peered":"2024-03-26T03:20:57.353803+0000","last_unstale":"2024-03-26T03:22:34.468824+0000","last_undegraded":"2024-03-26T03:22:34.468824+0000","last_fullsized":"2024-03-26T03:22:34.468824+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:47.342860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:47.342860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:57.521680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":949,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410201+0000","last_change":"2024-03-26T03:22:15.299593+0000","last_active":"2024-03-26T03:22:33.410201+0000","last_peered":"2024-03-26T03:22:33.410201+0000","last_clean":"2024-03-26T03:22:33.410201+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:22:33.410201+0000","last_undegraded":"2024-03-26T03:22:33.410201+0000","last_fullsized":"2024-03-26T03:22:33.410201+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:15.299461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:15.299461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.687673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":642,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469358+0000","last_change":"2024-03-26T03:21:27.728971+0000","last_active":"2024-03-26T03:22:34.469358+0000","last_peered":"2024-03-26T03:22:34.469358+0000","last_clean":"2024-03-26T03:22:34.469358+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:22:34.469358+0000","last_undegraded":"2024-03-26T03:22:34.469358+0000","last_fullsized":"2024-03-26T03:22:34.469358+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:27.728836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:27.728836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"446'39","reported_seq":233,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409879+0000","last_change":"2024-03-26T03:21:28.438386+0000","last_active":"2024-03-26T03:22:33.409879+0000","last_peered":"2024-03-26T03:22:33.409879+0000","last_clean":"2024-03-26T03:22:33.409879+0000","last_became_active":"2024-03-26T03:20:12.732132+0000","last_became_peered":"2024-03-26T03:20:12.732132+0000","last_unstale":"2024-03-26T03:22:33.409879+0000","last_undegraded":"2024-03-26T03:22:33.409879+0000","last_fullsized":"2024-03-26T03:22:33.409879+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:21:28.438285+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:21:28.438285+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:39.296324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:49.618119+0000","last_change":"2024-03-26T03:22:49.618146+0000","last_active":"2024-03-26T03:22:49.618119+0000","last_peered":"2024-03-26T03:22:49.618119+0000","last_clean":"2024-03-26T03:22:49.618119+0000","last_became_active":"2024-03-26T03:20:57.371143+0000","last_became_peered":"2024-03-26T03:20:57.371143+0000","last_unstale":"2024-03-26T03:22:49.618119+0000","last_undegraded":"2024-03-26T03:22:49.618119+0000","last_fullsized":"2024-03-26T03:22:49.618119+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:49.618040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:35.091531+0000","last_clean_scrub_stamp":"2024-03-26T03:22:49.618040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:13.073766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1033,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553458+0000","last_change":"2024-03-26T03:21:28.861641+0000","last_active":"2024-03-26T03:22:33.553458+0000","last_peered":"2024-03-26T03:22:33.553458+0000","last_clean":"2024-03-26T03:22:33.553458+0000","last_became_active":"2024-03-26T03:20:57.363788+0000","last_became_peered":"2024-03-26T03:20:57.363788+0000","last_unstale":"2024-03-26T03:22:33.553458+0000","last_undegraded":"2024-03-26T03:22:33.553458+0000","last_fullsized":"2024-03-26T03:22:33.553458+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:28.861569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:28.861569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:44.640592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416044+0000","last_change":"2024-03-26T03:21:48.281731+0000","last_active":"2024-03-26T03:22:33.416044+0000","last_peered":"2024-03-26T03:22:33.416044+0000","last_clean":"2024-03-26T03:22:33.416044+0000","last_became_active":"2024-03-26T03:20:57.356519+0000","last_became_peered":"2024-03-26T03:20:57.356519+0000","last_unstale":"2024-03-26T03:22:33.416044+0000","last_undegraded":"2024-03-26T03:22:33.416044+0000","last_fullsized":"2024-03-26T03:22:33.416044+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:48.281609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:48.281609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:58.827911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":726,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468843+0000","last_change":"2024-03-26T03:22:07.632347+0000","last_active":"2024-03-26T03:22:34.468843+0000","last_peered":"2024-03-26T03:22:34.468843+0000","last_clean":"2024-03-26T03:22:34.468843+0000","last_became_active":"2024-03-26T03:21:00.293971+0000","last_became_peered":"2024-03-26T03:21:00.293971+0000","last_unstale":"2024-03-26T03:22:34.468843+0000","last_undegraded":"2024-03-26T03:22:34.468843+0000","last_fullsized":"2024-03-26T03:22:34.468843+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:07.632196+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:22:07.632196+0000","last_clean_scrub_stamp":"2024-03-26T03:22:07.632196+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:35.898408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"446'15","reported_seq":193,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410070+0000","last_change":"2024-03-26T03:21:21.420396+0000","last_active":"2024-03-26T03:22:33.410070+0000","last_peered":"2024-03-26T03:22:33.410070+0000","last_clean":"2024-03-26T03:22:33.410070+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:22:33.410070+0000","last_undegraded":"2024-03-26T03:22:33.410070+0000","last_fullsized":"2024-03-26T03:22:33.410070+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'15","last_scrub_stamp":"2024-03-26T03:21:21.420279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:21.420279+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:45.930292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549630+0000","last_change":"2024-03-26T03:21:47.104938+0000","last_active":"2024-03-26T03:22:34.549630+0000","last_peered":"2024-03-26T03:22:34.549630+0000","last_clean":"2024-03-26T03:22:34.549630+0000","last_became_active":"2024-03-26T03:20:57.366367+0000","last_became_peered":"2024-03-26T03:20:57.366367+0000","last_unstale":"2024-03-26T03:22:34.549630+0000","last_undegraded":"2024-03-26T03:22:34.549630+0000","last_fullsized":"2024-03-26T03:22:34.549630+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:47.104834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:21:47.104834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:00.172590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.481268+0000","last_change":"2024-03-26T03:22:42.481314+0000","last_active":"2024-03-26T03:22:42.481268+0000","last_peered":"2024-03-26T03:22:42.481268+0000","last_clean":"2024-03-26T03:22:42.481268+0000","last_became_active":"2024-03-26T03:20:57.374412+0000","last_became_peered":"2024-03-26T03:20:57.374412+0000","last_unstale":"2024-03-26T03:22:42.481268+0000","last_undegraded":"2024-03-26T03:22:42.481268+0000","last_fullsized":"2024-03-26T03:22:42.481268+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:42.481183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:13.096856+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.481183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.660670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":976,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553265+0000","last_change":"2024-03-26T03:22:27.111479+0000","last_active":"2024-03-26T03:22:33.553265+0000","last_peered":"2024-03-26T03:22:33.553265+0000","last_clean":"2024-03-26T03:22:33.553265+0000","last_became_active":"2024-03-26T03:20:42.257114+0000","last_became_peered":"2024-03-26T03:20:42.257114+0000","last_unstale":"2024-03-26T03:22:33.553265+0000","last_undegraded":"2024-03-26T03:22:33.553265+0000","last_fullsized":"2024-03-26T03:22:33.553265+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:27.111376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.111376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:39.428944+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":776,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:45.818139+0000","last_change":"2024-03-26T03:22:45.818177+0000","last_active":"2024-03-26T03:22:45.818139+0000","last_peered":"2024-03-26T03:22:45.818139+0000","last_clean":"2024-03-26T03:22:45.818139+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:22:45.818139+0000","last_undegraded":"2024-03-26T03:22:45.818139+0000","last_fullsized":"2024-03-26T03:22:45.818139+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:45.818062+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:22:45.818062+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:48.559771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"508'32","reported_seq":1027,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409572+0000","last_change":"2024-03-26T03:22:26.183697+0000","last_active":"2024-03-26T03:22:33.409572+0000","last_peered":"2024-03-26T03:22:33.409572+0000","last_clean":"2024-03-26T03:22:33.409572+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:22:33.409572+0000","last_undegraded":"2024-03-26T03:22:33.409572+0000","last_fullsized":"2024-03-26T03:22:33.409572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'32","last_scrub_stamp":"2024-03-26T03:22:26.183569+0000","last_deep_scrub":"508'32","last_deep_scrub_stamp":"2024-03-26T03:22:26.183569+0000","last_clean_scrub_stamp":"2024-03-26T03:22:26.183569+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:43.860787+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":657,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469266+0000","last_change":"2024-03-26T03:21:38.346061+0000","last_active":"2024-03-26T03:22:34.469266+0000","last_peered":"2024-03-26T03:22:34.469266+0000","last_clean":"2024-03-26T03:22:34.469266+0000","last_became_active":"2024-03-26T03:20:12.748421+0000","last_became_peered":"2024-03-26T03:20:12.748421+0000","last_unstale":"2024-03-26T03:22:34.469266+0000","last_undegraded":"2024-03-26T03:22:34.469266+0000","last_fullsized":"2024-03-26T03:22:34.469266+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:38.345955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.345955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:54.930921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":986,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548520+0000","last_change":"2024-03-26T03:22:15.066250+0000","last_active":"2024-03-26T03:22:34.548520+0000","last_peered":"2024-03-26T03:22:34.548520+0000","last_clean":"2024-03-26T03:22:34.548520+0000","last_became_active":"2024-03-26T03:20:57.363793+0000","last_became_peered":"2024-03-26T03:20:57.363793+0000","last_unstale":"2024-03-26T03:22:34.548520+0000","last_undegraded":"2024-03-26T03:22:34.548520+0000","last_fullsized":"2024-03-26T03:22:34.548520+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:15.066162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:15.066162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:27.301815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":968,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410196+0000","last_change":"2024-03-26T03:21:15.448860+0000","last_active":"2024-03-26T03:22:33.410196+0000","last_peered":"2024-03-26T03:22:33.410196+0000","last_clean":"2024-03-26T03:22:33.410196+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:22:33.410196+0000","last_undegraded":"2024-03-26T03:22:33.410196+0000","last_fullsized":"2024-03-26T03:22:33.410196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:15.448753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:21:15.448753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:39.290211+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":506,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549544+0000","last_change":"2024-03-26T03:22:27.330428+0000","last_active":"2024-03-26T03:22:34.549544+0000","last_peered":"2024-03-26T03:22:34.549544+0000","last_clean":"2024-03-26T03:22:34.549544+0000","last_became_active":"2024-03-26T03:21:00.294899+0000","last_became_peered":"2024-03-26T03:21:00.294899+0000","last_unstale":"2024-03-26T03:22:34.549544+0000","last_undegraded":"2024-03-26T03:22:34.549544+0000","last_fullsized":"2024-03-26T03:22:34.549544+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:27.330334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.330334+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:32.042518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"446'64","reported_seq":787,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468623+0000","last_change":"2024-03-26T03:21:37.375730+0000","last_active":"2024-03-26T03:22:34.468623+0000","last_peered":"2024-03-26T03:22:34.468623+0000","last_clean":"2024-03-26T03:22:34.468623+0000","last_became_active":"2024-03-26T03:20:12.748027+0000","last_became_peered":"2024-03-26T03:20:12.748027+0000","last_unstale":"2024-03-26T03:22:34.468623+0000","last_undegraded":"2024-03-26T03:22:34.468623+0000","last_fullsized":"2024-03-26T03:22:34.468623+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'64","last_scrub_stamp":"2024-03-26T03:21:37.375577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.375577+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:40.920423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.417454+0000","last_change":"2024-03-26T03:21:40.351176+0000","last_active":"2024-03-26T03:22:33.417454+0000","last_peered":"2024-03-26T03:22:33.417454+0000","last_clean":"2024-03-26T03:22:33.417454+0000","last_became_active":"2024-03-26T03:20:57.356397+0000","last_became_peered":"2024-03-26T03:20:57.356397+0000","last_unstale":"2024-03-26T03:22:33.417454+0000","last_undegraded":"2024-03-26T03:22:33.417454+0000","last_fullsized":"2024-03-26T03:22:33.417454+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:40.351089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:40.351089+0000","last_clean_scrub_stamp":"2024-03-26T03:21:40.351089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:55.930989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":191,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.282030+0000","last_change":"2024-03-26T03:22:37.282030+0000","last_active":"2024-03-26T03:22:37.282030+0000","last_peered":"2024-03-26T03:22:37.282030+0000","last_clean":"2024-03-26T03:22:37.282030+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:22:37.282030+0000","last_undegraded":"2024-03-26T03:22:37.282030+0000","last_fullsized":"2024-03-26T03:22:37.282030+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:37.281914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.281914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.055497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":648,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468565+0000","last_change":"2024-03-26T03:21:33.485707+0000","last_active":"2024-03-26T03:22:34.468565+0000","last_peered":"2024-03-26T03:22:34.468565+0000","last_clean":"2024-03-26T03:22:34.468565+0000","last_became_active":"2024-03-26T03:20:57.354412+0000","last_became_peered":"2024-03-26T03:20:57.354412+0000","last_unstale":"2024-03-26T03:22:34.468565+0000","last_undegraded":"2024-03-26T03:22:34.468565+0000","last_fullsized":"2024-03-26T03:22:34.468565+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:33.485593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:33.485593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:45.359011+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1508416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1422,"num_read_kb":1417,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1824,"num_read_kb":1549,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":5962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1271,"ondisk_log_size":1271,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103460,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790320,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92954832,"statfs":{"total":95995035648,"available":95185747968,"internally_reserved":0,"allocated":1155072,"data_stored":719985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":808122826},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430504,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848508,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896644,"statfs":{"total":95995035648,"available":95126163456,"internally_reserved":0,"allocated":1036288,"data_stored":686491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":867826118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575169,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1123268,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92621884,"statfs":{"total":95995035648,"available":94844809216,"internally_reserved":0,"allocated":913408,"data_stored":458389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816760,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92928392,"statfs":{"total":95995035648,"available":95158673408,"internally_reserved":0,"allocated":1228800,"data_stored":785740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411643,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912488,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92832664,"statfs":{"total":95995035648,"available":95060647936,"internally_reserved":0,"allocated":1146880,"data_stored":720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175080,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477248,"kb_used_data":1208,"kb_used_omap":4,"kb_used_meta":476027,"kb_avail":93267904,"statfs":{"total":95995035648,"available":95506333696,"internally_reserved":0,"allocated":1236992,"data_stored":784788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":487452323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986482,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942776,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92802376,"statfs":{"total":95995035648,"available":95029633024,"internally_reserved":0,"allocated":901120,"data_stored":457206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692142,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015212,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":1014078,"kb_avail":92729940,"statfs":{"total":95995035648,"available":94955458560,"internally_reserved":0,"allocated":1150976,"data_stored":719445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1038416325},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:22:54.504 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:22:54.761 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:54.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[29666]: 2.14 scrub starts 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[29666]: 2.14 scrub ok 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[29666]: 6.d deep-scrub starts 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[29666]: 6.d deep-scrub ok 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[37920]: 2.14 scrub starts 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[37920]: 2.14 scrub ok 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[37920]: 6.d deep-scrub starts 2024-03-26T03:22:54.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:54 smithi066 ceph-mon[37920]: 6.d deep-scrub ok 2024-03-26T03:22:54.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:54 smithi192 ceph-mon[33110]: 2.14 scrub starts 2024-03-26T03:22:54.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:54 smithi192 ceph-mon[33110]: 2.14 scrub ok 2024-03-26T03:22:54.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:54 smithi192 ceph-mon[33110]: 6.d deep-scrub starts 2024-03-26T03:22:54.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:54 smithi192 ceph-mon[33110]: 6.d deep-scrub ok 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: from='client.36296 v1:172.21.15.66:0/1281223343' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: 5.6 scrub starts 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: 5.6 scrub ok 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: 2.f scrub starts 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: 2.f scrub ok 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: 2.7 scrub starts 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: 2.7 scrub ok 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: pgmap v1130: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:55.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: from='client.36296 v1:172.21.15.66:0/1281223343' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: 5.6 scrub starts 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: 5.6 scrub ok 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: 2.f scrub starts 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: 2.f scrub ok 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: 2.7 scrub starts 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: 2.7 scrub ok 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: pgmap v1130: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:55.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:55 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:55.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: from='client.36296 v1:172.21.15.66:0/1281223343' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: 5.6 scrub starts 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: 5.6 scrub ok 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: 2.f scrub starts 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: 2.f scrub ok 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: 2.7 scrub starts 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: 2.7 scrub ok 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: pgmap v1130: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:55.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:55 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:22:56.823 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:56 smithi066 ceph-mon[29666]: 6.14 deep-scrub starts 2024-03-26T03:22:56.824 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:56 smithi066 ceph-mon[29666]: 6.14 deep-scrub ok 2024-03-26T03:22:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:56 smithi066 ceph-mon[37920]: 6.14 deep-scrub starts 2024-03-26T03:22:56.824 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:56 smithi066 ceph-mon[37920]: 6.14 deep-scrub ok 2024-03-26T03:22:56.829 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-26T03:22:56.832 INFO:teuthology.orchestra.run.smithi066.stderr:dumped all 2024-03-26T03:22:56.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:56 smithi192 ceph-mon[33110]: 6.14 deep-scrub starts 2024-03-26T03:22:56.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:56 smithi192 ceph-mon[33110]: 6.14 deep-scrub ok 2024-03-26T03:22:57.610 INFO:teuthology.orchestra.run.smithi066.stdout:{"pg_ready":true,"pg_map":{"version":1131,"stamp":"2024-03-26T03:22:56.517173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3275,"num_read_kb":2995,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":1519626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5359,"ondisk_log_size":5359,"up":486,"acting":486,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":486,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6930740,"kb_used_data":8564,"kb_used_omap":15,"kb_used_meta":6922096,"kb_avail":743030476,"statfs":{"total":767960285184,"available":760863207424,"internally_reserved":0,"allocated":8769536,"data_stored":5332468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15604,"internal_metadata":7088227084},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003036"},"pg_stats":[{"pgid":"1.d","version":"82'128","reported_seq":385,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:35.381605+0000","last_change":"2024-03-26T03:22:35.381632+0000","last_active":"2024-03-26T03:22:35.381605+0000","last_peered":"2024-03-26T03:22:35.381605+0000","last_clean":"2024-03-26T03:22:35.381605+0000","last_became_active":"2024-03-26T03:21:00.295100+0000","last_became_peered":"2024-03-26T03:21:00.295100+0000","last_unstale":"2024-03-26T03:22:35.381605+0000","last_undegraded":"2024-03-26T03:22:35.381605+0000","last_fullsized":"2024-03-26T03:22:35.381605+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:35.381529+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:22:35.381529+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.354050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"477'11","reported_seq":110,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298598+0000","last_change":"2024-03-26T03:21:51.123948+0000","last_active":"2024-03-26T03:22:47.298598+0000","last_peered":"2024-03-26T03:22:47.298598+0000","last_clean":"2024-03-26T03:22:47.298598+0000","last_became_active":"2024-03-26T03:20:57.376505+0000","last_became_peered":"2024-03-26T03:20:57.376505+0000","last_unstale":"2024-03-26T03:22:47.298598+0000","last_undegraded":"2024-03-26T03:22:47.298598+0000","last_fullsized":"2024-03-26T03:22:47.298598+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'11","last_scrub_stamp":"2024-03-26T03:21:51.123834+0000","last_deep_scrub":"112'9","last_deep_scrub_stamp":"2024-03-26T03:04:48.638002+0000","last_clean_scrub_stamp":"2024-03-26T03:21:51.123834+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:05.400320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"446'51","reported_seq":1148,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553058+0000","last_change":"2024-03-26T03:21:20.956466+0000","last_active":"2024-03-26T03:22:33.553058+0000","last_peered":"2024-03-26T03:22:33.553058+0000","last_clean":"2024-03-26T03:22:33.553058+0000","last_became_active":"2024-03-26T03:20:57.364284+0000","last_became_peered":"2024-03-26T03:20:57.364284+0000","last_unstale":"2024-03-26T03:22:33.553058+0000","last_undegraded":"2024-03-26T03:22:33.553058+0000","last_fullsized":"2024-03-26T03:22:33.553058+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'51","last_scrub_stamp":"2024-03-26T03:21:20.956352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:20.956352+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":617,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.411010+0000","last_change":"2024-03-26T03:22:24.597541+0000","last_active":"2024-03-26T03:22:33.411010+0000","last_peered":"2024-03-26T03:22:33.411010+0000","last_clean":"2024-03-26T03:22:33.411010+0000","last_became_active":"2024-03-26T03:21:26.466719+0000","last_became_peered":"2024-03-26T03:21:26.466719+0000","last_unstale":"2024-03-26T03:22:33.411010+0000","last_undegraded":"2024-03-26T03:22:33.411010+0000","last_fullsized":"2024-03-26T03:22:33.411010+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:24.597458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:24.597458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.424636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":734,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553304+0000","last_change":"2024-03-26T03:21:53.893295+0000","last_active":"2024-03-26T03:22:33.553304+0000","last_peered":"2024-03-26T03:22:33.553304+0000","last_clean":"2024-03-26T03:22:33.553304+0000","last_became_active":"2024-03-26T03:21:43.568404+0000","last_became_peered":"2024-03-26T03:21:43.568404+0000","last_unstale":"2024-03-26T03:22:33.553304+0000","last_undegraded":"2024-03-26T03:22:33.553304+0000","last_fullsized":"2024-03-26T03:22:33.553304+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:53.893197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:47.274511+0000","last_clean_scrub_stamp":"2024-03-26T03:21:53.893197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:08.415875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"82'128","reported_seq":498,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410468+0000","last_change":"2024-03-26T03:21:43.446862+0000","last_active":"2024-03-26T03:22:33.410468+0000","last_peered":"2024-03-26T03:22:33.410468+0000","last_clean":"2024-03-26T03:22:33.410468+0000","last_became_active":"2024-03-26T03:21:00.290504+0000","last_became_peered":"2024-03-26T03:21:00.290504+0000","last_unstale":"2024-03-26T03:22:33.410468+0000","last_undegraded":"2024-03-26T03:22:33.410468+0000","last_fullsized":"2024-03-26T03:22:33.410468+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:43.446783+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:19:16.437810+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.446783+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:45.619393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"446'32","reported_seq":138,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416904+0000","last_change":"2024-03-26T03:21:59.340085+0000","last_active":"2024-03-26T03:22:33.416904+0000","last_peered":"2024-03-26T03:22:33.416904+0000","last_clean":"2024-03-26T03:22:33.416904+0000","last_became_active":"2024-03-26T03:20:12.725584+0000","last_became_peered":"2024-03-26T03:20:12.725584+0000","last_unstale":"2024-03-26T03:22:33.416904+0000","last_undegraded":"2024-03-26T03:22:33.416904+0000","last_fullsized":"2024-03-26T03:22:33.416904+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'32","last_scrub_stamp":"2024-03-26T03:21:59.339988+0000","last_deep_scrub":"231'26","last_deep_scrub_stamp":"2024-03-26T03:11:11.587327+0000","last_clean_scrub_stamp":"2024-03-26T03:21:59.339988+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:06.819009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":980,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549562+0000","last_change":"2024-03-26T03:21:58.937860+0000","last_active":"2024-03-26T03:22:34.549562+0000","last_peered":"2024-03-26T03:22:34.549562+0000","last_clean":"2024-03-26T03:22:34.549562+0000","last_became_active":"2024-03-26T03:17:56.570713+0000","last_became_peered":"2024-03-26T03:17:56.570713+0000","last_unstale":"2024-03-26T03:22:34.549562+0000","last_undegraded":"2024-03-26T03:22:34.549562+0000","last_fullsized":"2024-03-26T03:22:34.549562+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:58.937775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:59.710985+0000","last_clean_scrub_stamp":"2024-03-26T03:21:58.937775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.625992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":609,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:41.400718+0000","last_change":"2024-03-26T03:22:41.400759+0000","last_active":"2024-03-26T03:22:41.400718+0000","last_peered":"2024-03-26T03:22:41.400718+0000","last_clean":"2024-03-26T03:22:41.400718+0000","last_became_active":"2024-03-26T03:21:26.466644+0000","last_became_peered":"2024-03-26T03:21:26.466644+0000","last_unstale":"2024-03-26T03:22:41.400718+0000","last_undegraded":"2024-03-26T03:22:41.400718+0000","last_fullsized":"2024-03-26T03:22:41.400718+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:41.400646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:41.400646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:09.036294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:43.612985+0000","last_change":"2024-03-26T03:22:43.613009+0000","last_active":"2024-03-26T03:22:43.612985+0000","last_peered":"2024-03-26T03:22:43.612985+0000","last_clean":"2024-03-26T03:22:43.612985+0000","last_became_active":"2024-03-26T03:20:57.353859+0000","last_became_peered":"2024-03-26T03:20:57.353859+0000","last_unstale":"2024-03-26T03:22:43.612985+0000","last_undegraded":"2024-03-26T03:22:43.612985+0000","last_fullsized":"2024-03-26T03:22:43.612985+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:43.612922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:48.304319+0000","last_clean_scrub_stamp":"2024-03-26T03:22:43.612922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:13.063334+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"82'128","reported_seq":1076,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:38.564775+0000","last_change":"2024-03-26T03:22:38.564826+0000","last_active":"2024-03-26T03:22:38.564775+0000","last_peered":"2024-03-26T03:22:38.564775+0000","last_clean":"2024-03-26T03:22:38.564775+0000","last_became_active":"2024-03-26T03:10:44.083747+0000","last_became_peered":"2024-03-26T03:10:44.083747+0000","last_unstale":"2024-03-26T03:22:38.564775+0000","last_undegraded":"2024-03-26T03:22:38.564775+0000","last_fullsized":"2024-03-26T03:22:38.564775+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:38.564662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:38.564662+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.250010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"446'57","reported_seq":1163,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.460821+0000","last_change":"2024-03-26T03:21:37.186655+0000","last_active":"2024-03-26T03:22:34.460821+0000","last_peered":"2024-03-26T03:22:34.460821+0000","last_clean":"2024-03-26T03:22:34.460821+0000","last_became_active":"2024-03-26T03:20:57.371493+0000","last_became_peered":"2024-03-26T03:20:57.371493+0000","last_unstale":"2024-03-26T03:22:34.460821+0000","last_undegraded":"2024-03-26T03:22:34.460821+0000","last_fullsized":"2024-03-26T03:22:34.460821+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'57","last_scrub_stamp":"2024-03-26T03:21:37.186539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.186539+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:57.985345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":77,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":823,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.412457+0000","last_change":"2024-03-26T03:22:07.632621+0000","last_active":"2024-03-26T03:22:33.412457+0000","last_peered":"2024-03-26T03:22:33.412457+0000","last_clean":"2024-03-26T03:22:33.412457+0000","last_became_active":"2024-03-26T03:21:26.463723+0000","last_became_peered":"2024-03-26T03:21:26.463723+0000","last_unstale":"2024-03-26T03:22:33.412457+0000","last_undegraded":"2024-03-26T03:22:33.412457+0000","last_fullsized":"2024-03-26T03:22:33.412457+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:07.632516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:47.600600+0000","last_clean_scrub_stamp":"2024-03-26T03:22:07.632516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.434772+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:49.331509+0000","last_change":"2024-03-26T03:22:49.331531+0000","last_active":"2024-03-26T03:22:49.331509+0000","last_peered":"2024-03-26T03:22:49.331509+0000","last_clean":"2024-03-26T03:22:49.331509+0000","last_became_active":"2024-03-26T03:20:12.719811+0000","last_became_peered":"2024-03-26T03:20:12.719811+0000","last_unstale":"2024-03-26T03:22:49.331509+0000","last_undegraded":"2024-03-26T03:22:49.331509+0000","last_fullsized":"2024-03-26T03:22:49.331509+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:49.331438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:16.301182+0000","last_clean_scrub_stamp":"2024-03-26T03:22:49.331438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.235938+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549133+0000","last_change":"2024-03-26T03:22:29.333875+0000","last_active":"2024-03-26T03:22:34.549133+0000","last_peered":"2024-03-26T03:22:34.549133+0000","last_clean":"2024-03-26T03:22:34.549133+0000","last_became_active":"2024-03-26T03:20:57.359873+0000","last_became_peered":"2024-03-26T03:20:57.359873+0000","last_unstale":"2024-03-26T03:22:34.549133+0000","last_undegraded":"2024-03-26T03:22:34.549133+0000","last_fullsized":"2024-03-26T03:22:34.549133+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:29.333767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:29.333767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:33.278689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"446'53","reported_seq":230,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.331265+0000","last_change":"2024-03-26T03:22:42.331310+0000","last_active":"2024-03-26T03:22:42.331265+0000","last_peered":"2024-03-26T03:22:42.331265+0000","last_clean":"2024-03-26T03:22:42.331265+0000","last_became_active":"2024-03-26T03:17:56.571203+0000","last_became_peered":"2024-03-26T03:17:56.571203+0000","last_unstale":"2024-03-26T03:22:42.331265+0000","last_undegraded":"2024-03-26T03:22:42.331265+0000","last_fullsized":"2024-03-26T03:22:42.331265+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'53","last_scrub_stamp":"2024-03-26T03:22:42.331184+0000","last_deep_scrub":"153'25","last_deep_scrub_stamp":"2024-03-26T03:03:57.378721+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.331184+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:44.444720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549516+0000","last_change":"2024-03-26T03:22:12.147474+0000","last_active":"2024-03-26T03:22:34.549516+0000","last_peered":"2024-03-26T03:22:34.549516+0000","last_clean":"2024-03-26T03:22:34.549516+0000","last_became_active":"2024-03-26T03:20:57.375755+0000","last_became_peered":"2024-03-26T03:20:57.375755+0000","last_unstale":"2024-03-26T03:22:34.549516+0000","last_undegraded":"2024-03-26T03:22:34.549516+0000","last_fullsized":"2024-03-26T03:22:34.549516+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:12.147358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:12.147358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.758913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":987,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549200+0000","last_change":"2024-03-26T03:22:19.054387+0000","last_active":"2024-03-26T03:22:34.549200+0000","last_peered":"2024-03-26T03:22:34.549200+0000","last_clean":"2024-03-26T03:22:34.549200+0000","last_became_active":"2024-03-26T03:20:12.720725+0000","last_became_peered":"2024-03-26T03:20:12.720725+0000","last_unstale":"2024-03-26T03:22:34.549200+0000","last_undegraded":"2024-03-26T03:22:34.549200+0000","last_fullsized":"2024-03-26T03:22:34.549200+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:19.054291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:45.397023+0000","last_clean_scrub_stamp":"2024-03-26T03:22:19.054291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:46.668028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":605,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.690729+0000","last_change":"2024-03-26T03:22:34.690752+0000","last_active":"2024-03-26T03:22:34.690729+0000","last_peered":"2024-03-26T03:22:34.690729+0000","last_clean":"2024-03-26T03:22:34.690729+0000","last_became_active":"2024-03-26T03:21:26.467423+0000","last_became_peered":"2024-03-26T03:21:26.467423+0000","last_unstale":"2024-03-26T03:22:34.690729+0000","last_undegraded":"2024-03-26T03:22:34.690729+0000","last_fullsized":"2024-03-26T03:22:34.690729+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:34.690650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:34.690650+0000","last_clean_scrub_stamp":"2024-03-26T03:22:34.690650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:48.469907+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'128","reported_seq":360,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553158+0000","last_change":"2024-03-26T03:21:43.919009+0000","last_active":"2024-03-26T03:22:33.553158+0000","last_peered":"2024-03-26T03:22:33.553158+0000","last_clean":"2024-03-26T03:22:33.553158+0000","last_became_active":"2024-03-26T03:21:43.565798+0000","last_became_peered":"2024-03-26T03:21:43.565798+0000","last_unstale":"2024-03-26T03:22:33.553158+0000","last_undegraded":"2024-03-26T03:22:33.553158+0000","last_fullsized":"2024-03-26T03:22:33.553158+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:43.918932+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:21:27.137087+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.918932+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:02.901198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"82'128","reported_seq":569,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549135+0000","last_change":"2024-03-26T03:22:15.192691+0000","last_active":"2024-03-26T03:22:34.549135+0000","last_peered":"2024-03-26T03:22:34.549135+0000","last_clean":"2024-03-26T03:22:34.549135+0000","last_became_active":"2024-03-26T03:21:00.293608+0000","last_became_peered":"2024-03-26T03:21:00.293608+0000","last_unstale":"2024-03-26T03:22:34.549135+0000","last_undegraded":"2024-03-26T03:22:34.549135+0000","last_fullsized":"2024-03-26T03:22:34.549135+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:15.192599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:15.192599+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:18.491992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"446'30","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410218+0000","last_change":"2024-03-26T03:22:18.293766+0000","last_active":"2024-03-26T03:22:33.410218+0000","last_peered":"2024-03-26T03:22:33.410218+0000","last_clean":"2024-03-26T03:22:33.410218+0000","last_became_active":"2024-03-26T03:14:21.597744+0000","last_became_peered":"2024-03-26T03:14:21.597744+0000","last_unstale":"2024-03-26T03:22:33.410218+0000","last_undegraded":"2024-03-26T03:22:33.410218+0000","last_fullsized":"2024-03-26T03:22:33.410218+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'30","last_scrub_stamp":"2024-03-26T03:22:18.293639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:18.293639+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:27.339943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":973,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548762+0000","last_change":"2024-03-26T03:21:42.933870+0000","last_active":"2024-03-26T03:22:34.548762+0000","last_peered":"2024-03-26T03:22:34.548762+0000","last_clean":"2024-03-26T03:22:34.548762+0000","last_became_active":"2024-03-26T03:20:57.364110+0000","last_became_peered":"2024-03-26T03:20:57.364110+0000","last_unstale":"2024-03-26T03:22:34.548762+0000","last_undegraded":"2024-03-26T03:22:34.548762+0000","last_fullsized":"2024-03-26T03:22:34.548762+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:42.933794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:42.933794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.532956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"490'14","reported_seq":1053,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298496+0000","last_change":"2024-03-26T03:21:37.303210+0000","last_active":"2024-03-26T03:22:47.298496+0000","last_peered":"2024-03-26T03:22:47.298496+0000","last_clean":"2024-03-26T03:22:47.298496+0000","last_became_active":"2024-03-26T03:21:26.466292+0000","last_became_peered":"2024-03-26T03:21:26.466292+0000","last_unstale":"2024-03-26T03:22:47.298496+0000","last_undegraded":"2024-03-26T03:22:47.298496+0000","last_fullsized":"2024-03-26T03:22:47.298496+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"490'14","last_scrub_stamp":"2024-03-26T03:21:37.303105+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-26T03:08:05.956667+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.303105+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:05.462102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":191,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410502+0000","last_change":"2024-03-26T03:22:23.251448+0000","last_active":"2024-03-26T03:22:33.410502+0000","last_peered":"2024-03-26T03:22:33.410502+0000","last_clean":"2024-03-26T03:22:33.410502+0000","last_became_active":"2024-03-26T03:17:56.573839+0000","last_became_peered":"2024-03-26T03:17:56.573839+0000","last_unstale":"2024-03-26T03:22:33.410502+0000","last_undegraded":"2024-03-26T03:22:33.410502+0000","last_fullsized":"2024-03-26T03:22:33.410502+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:23.251349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:23.251349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.640251+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'128","reported_seq":340,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553135+0000","last_change":"2024-03-26T03:21:44.890578+0000","last_active":"2024-03-26T03:22:33.553135+0000","last_peered":"2024-03-26T03:22:33.553135+0000","last_clean":"2024-03-26T03:22:33.553135+0000","last_became_active":"2024-03-26T03:21:43.566139+0000","last_became_peered":"2024-03-26T03:21:43.566139+0000","last_unstale":"2024-03-26T03:22:33.553135+0000","last_undegraded":"2024-03-26T03:22:33.553135+0000","last_fullsized":"2024-03-26T03:22:33.553135+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:44.890494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:44.890494+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.294185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"446'39","reported_seq":1048,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549244+0000","last_change":"2024-03-26T03:22:10.066546+0000","last_active":"2024-03-26T03:22:34.549244+0000","last_peered":"2024-03-26T03:22:34.549244+0000","last_clean":"2024-03-26T03:22:34.549244+0000","last_became_active":"2024-03-26T03:14:21.580126+0000","last_became_peered":"2024-03-26T03:14:21.580126+0000","last_unstale":"2024-03-26T03:22:34.549244+0000","last_undegraded":"2024-03-26T03:22:34.549244+0000","last_fullsized":"2024-03-26T03:22:34.549244+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:22:10.066431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:10.066431+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:34.409681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":973,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548524+0000","last_change":"2024-03-26T03:21:52.021391+0000","last_active":"2024-03-26T03:22:34.548524+0000","last_peered":"2024-03-26T03:22:34.548524+0000","last_clean":"2024-03-26T03:22:34.548524+0000","last_became_active":"2024-03-26T03:20:57.364582+0000","last_became_peered":"2024-03-26T03:20:57.364582+0000","last_unstale":"2024-03-26T03:22:34.548524+0000","last_undegraded":"2024-03-26T03:22:34.548524+0000","last_fullsized":"2024-03-26T03:22:34.548524+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:52.021309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:52.021309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.642241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"460'13","reported_seq":267,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298528+0000","last_change":"2024-03-26T03:21:46.484839+0000","last_active":"2024-03-26T03:22:47.298528+0000","last_peered":"2024-03-26T03:22:47.298528+0000","last_clean":"2024-03-26T03:22:47.298528+0000","last_became_active":"2024-03-26T03:20:12.733250+0000","last_became_peered":"2024-03-26T03:20:12.733250+0000","last_unstale":"2024-03-26T03:22:47.298528+0000","last_undegraded":"2024-03-26T03:22:47.298528+0000","last_fullsized":"2024-03-26T03:22:47.298528+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'13","last_scrub_stamp":"2024-03-26T03:21:46.484733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.484733+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:08.987183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":718,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:54.133598+0000","last_change":"2024-03-26T03:22:54.133625+0000","last_active":"2024-03-26T03:22:54.133598+0000","last_peered":"2024-03-26T03:22:54.133598+0000","last_clean":"2024-03-26T03:22:54.133598+0000","last_became_active":"2024-03-26T03:21:43.857406+0000","last_became_peered":"2024-03-26T03:21:43.857406+0000","last_unstale":"2024-03-26T03:22:54.133598+0000","last_undegraded":"2024-03-26T03:22:54.133598+0000","last_fullsized":"2024-03-26T03:22:54.133598+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:54.133516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:54.133516+0000","last_clean_scrub_stamp":"2024-03-26T03:22:54.133516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:03.031889+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"82'128","reported_seq":1117,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:55.789382+0000","last_change":"2024-03-26T03:22:55.789408+0000","last_active":"2024-03-26T03:22:55.789382+0000","last_peered":"2024-03-26T03:22:55.789382+0000","last_clean":"2024-03-26T03:22:55.789382+0000","last_became_active":"2024-03-26T03:20:15.312092+0000","last_became_peered":"2024-03-26T03:20:15.312092+0000","last_unstale":"2024-03-26T03:22:55.789382+0000","last_undegraded":"2024-03-26T03:22:55.789382+0000","last_fullsized":"2024-03-26T03:22:55.789382+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:55.789310+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:22:55.789310+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:13.952000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"446'39","reported_seq":1038,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410673+0000","last_change":"2024-03-26T03:21:16.400009+0000","last_active":"2024-03-26T03:22:33.410673+0000","last_peered":"2024-03-26T03:22:33.410673+0000","last_clean":"2024-03-26T03:22:33.410673+0000","last_became_active":"2024-03-26T02:58:58.926433+0000","last_became_peered":"2024-03-26T02:58:58.926433+0000","last_unstale":"2024-03-26T03:22:33.410673+0000","last_undegraded":"2024-03-26T03:22:33.410673+0000","last_fullsized":"2024-03-26T03:22:33.410673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:21:16.399895+0000","last_deep_scrub":"446'39","last_deep_scrub_stamp":"2024-03-26T03:21:16.399895+0000","last_clean_scrub_stamp":"2024-03-26T03:21:16.399895+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:41.176504+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549379+0000","last_change":"2024-03-26T03:21:46.062451+0000","last_active":"2024-03-26T03:22:34.549379+0000","last_peered":"2024-03-26T03:22:34.549379+0000","last_clean":"2024-03-26T03:22:34.549379+0000","last_became_active":"2024-03-26T03:20:57.369214+0000","last_became_peered":"2024-03-26T03:20:57.369214+0000","last_unstale":"2024-03-26T03:22:34.549379+0000","last_undegraded":"2024-03-26T03:22:34.549379+0000","last_fullsized":"2024-03-26T03:22:34.549379+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:46.062330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.062330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:00.923161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":793,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410824+0000","last_change":"2024-03-26T03:22:21.270329+0000","last_active":"2024-03-26T03:22:33.410824+0000","last_peered":"2024-03-26T03:22:33.410824+0000","last_clean":"2024-03-26T03:22:33.410824+0000","last_became_active":"2024-03-26T03:20:57.356625+0000","last_became_peered":"2024-03-26T03:20:57.356625+0000","last_unstale":"2024-03-26T03:22:33.410824+0000","last_undegraded":"2024-03-26T03:22:33.410824+0000","last_fullsized":"2024-03-26T03:22:33.410824+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:21.270217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:21.270217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.790519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":608,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.560797+0000","last_change":"2024-03-26T03:22:42.560832+0000","last_active":"2024-03-26T03:22:42.560797+0000","last_peered":"2024-03-26T03:22:42.560797+0000","last_clean":"2024-03-26T03:22:42.560797+0000","last_became_active":"2024-03-26T03:14:21.591533+0000","last_became_peered":"2024-03-26T03:14:21.591533+0000","last_unstale":"2024-03-26T03:22:42.560797+0000","last_undegraded":"2024-03-26T03:22:42.560797+0000","last_fullsized":"2024-03-26T03:22:42.560797+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:42.560729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:15.672090+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.560729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:12.481917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"332'11","reported_seq":830,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298528+0000","last_change":"2024-03-26T03:21:01.701749+0000","last_active":"2024-03-26T03:22:47.298528+0000","last_peered":"2024-03-26T03:22:47.298528+0000","last_clean":"2024-03-26T03:22:47.298528+0000","last_became_active":"2024-03-26T03:14:21.586171+0000","last_became_peered":"2024-03-26T03:14:21.586171+0000","last_unstale":"2024-03-26T03:22:47.298528+0000","last_undegraded":"2024-03-26T03:22:47.298528+0000","last_fullsized":"2024-03-26T03:22:47.298528+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'11","last_scrub_stamp":"2024-03-26T03:21:01.701636+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-26T03:00:37.456826+0000","last_clean_scrub_stamp":"2024-03-26T03:21:01.701636+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"446'48","reported_seq":1061,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409845+0000","last_change":"2024-03-26T03:22:17.335701+0000","last_active":"2024-03-26T03:22:33.409845+0000","last_peered":"2024-03-26T03:22:33.409845+0000","last_clean":"2024-03-26T03:22:33.409845+0000","last_became_active":"2024-03-26T02:58:58.926910+0000","last_became_peered":"2024-03-26T02:58:58.926910+0000","last_unstale":"2024-03-26T03:22:33.409845+0000","last_undegraded":"2024-03-26T03:22:33.409845+0000","last_fullsized":"2024-03-26T03:22:33.409845+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'48","last_scrub_stamp":"2024-03-26T03:22:17.335576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:17.335576+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.479184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1012,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:54.430345+0000","last_change":"2024-03-26T03:22:54.430367+0000","last_active":"2024-03-26T03:22:54.430345+0000","last_peered":"2024-03-26T03:22:54.430345+0000","last_clean":"2024-03-26T03:22:54.430345+0000","last_became_active":"2024-03-26T03:20:57.357348+0000","last_became_peered":"2024-03-26T03:20:57.357348+0000","last_unstale":"2024-03-26T03:22:54.430345+0000","last_undegraded":"2024-03-26T03:22:54.430345+0000","last_fullsized":"2024-03-26T03:22:54.430345+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:54.430280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:19.268683+0000","last_clean_scrub_stamp":"2024-03-26T03:22:54.430280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:06.448799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549535+0000","last_change":"2024-03-26T03:21:43.163634+0000","last_active":"2024-03-26T03:22:34.549535+0000","last_peered":"2024-03-26T03:22:34.549535+0000","last_clean":"2024-03-26T03:22:34.549535+0000","last_became_active":"2024-03-26T03:20:57.362070+0000","last_became_peered":"2024-03-26T03:20:57.362070+0000","last_unstale":"2024-03-26T03:22:34.549535+0000","last_undegraded":"2024-03-26T03:22:34.549535+0000","last_fullsized":"2024-03-26T03:22:34.549535+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:43.163526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:43.163526+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.163526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:00.782062+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"82'128","reported_seq":789,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469315+0000","last_change":"2024-03-26T03:22:25.641863+0000","last_active":"2024-03-26T03:22:34.469315+0000","last_peered":"2024-03-26T03:22:34.469315+0000","last_clean":"2024-03-26T03:22:34.469315+0000","last_became_active":"2024-03-26T03:21:00.294201+0000","last_became_peered":"2024-03-26T03:21:00.294201+0000","last_unstale":"2024-03-26T03:22:34.469315+0000","last_undegraded":"2024-03-26T03:22:34.469315+0000","last_fullsized":"2024-03-26T03:22:34.469315+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:25.641776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.641776+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.062856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":969,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410003+0000","last_change":"2024-03-26T03:22:13.330407+0000","last_active":"2024-03-26T03:22:33.410003+0000","last_peered":"2024-03-26T03:22:33.410003+0000","last_clean":"2024-03-26T03:22:33.410003+0000","last_became_active":"2024-03-26T03:20:57.355658+0000","last_became_peered":"2024-03-26T03:20:57.355658+0000","last_unstale":"2024-03-26T03:22:33.410003+0000","last_undegraded":"2024-03-26T03:22:33.410003+0000","last_fullsized":"2024-03-26T03:22:33.410003+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:13.330308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.330308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:16.417609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"446'66","reported_seq":875,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468456+0000","last_change":"2024-03-26T03:22:22.616080+0000","last_active":"2024-03-26T03:22:34.468456+0000","last_peered":"2024-03-26T03:22:34.468456+0000","last_clean":"2024-03-26T03:22:34.468456+0000","last_became_active":"2024-03-26T03:20:57.364272+0000","last_became_peered":"2024-03-26T03:20:57.364272+0000","last_unstale":"2024-03-26T03:22:34.468456+0000","last_undegraded":"2024-03-26T03:22:34.468456+0000","last_fullsized":"2024-03-26T03:22:34.468456+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'66","last_scrub_stamp":"2024-03-26T03:22:22.615941+0000","last_deep_scrub":"84'42","last_deep_scrub_stamp":"2024-03-26T03:00:08.772486+0000","last_clean_scrub_stamp":"2024-03-26T03:22:22.615941+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.127529+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'2","reported_seq":994,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:41.845584+0000","last_change":"2024-03-26T03:22:41.845584+0000","last_active":"2024-03-26T03:22:41.845584+0000","last_peered":"2024-03-26T03:22:41.845584+0000","last_clean":"2024-03-26T03:22:41.845584+0000","last_became_active":"2024-03-26T02:58:56.931016+0000","last_became_peered":"2024-03-26T02:58:56.931016+0000","last_unstale":"2024-03-26T03:22:41.845584+0000","last_undegraded":"2024-03-26T03:22:41.845584+0000","last_fullsized":"2024-03-26T03:22:41.845584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'2","last_scrub_stamp":"2024-03-26T03:22:41.845497+0000","last_deep_scrub":"88'2","last_deep_scrub_stamp":"2024-03-26T03:22:41.845497+0000","last_clean_scrub_stamp":"2024-03-26T03:22:41.845497+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:43.894102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":183,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409809+0000","last_change":"2024-03-26T03:21:39.470235+0000","last_active":"2024-03-26T03:22:33.409809+0000","last_peered":"2024-03-26T03:22:33.409809+0000","last_clean":"2024-03-26T03:22:33.409809+0000","last_became_active":"2024-03-26T03:17:56.566986+0000","last_became_peered":"2024-03-26T03:17:56.566986+0000","last_unstale":"2024-03-26T03:22:33.409809+0000","last_undegraded":"2024-03-26T03:22:33.409809+0000","last_fullsized":"2024-03-26T03:22:33.409809+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:39.470125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:39.470125+0000","last_clean_scrub_stamp":"2024-03-26T03:21:39.470125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:43.620499+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'128","reported_seq":1083,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410714+0000","last_change":"2024-03-26T03:21:37.466092+0000","last_active":"2024-03-26T03:22:33.410714+0000","last_peered":"2024-03-26T03:22:33.410714+0000","last_clean":"2024-03-26T03:22:33.410714+0000","last_became_active":"2024-03-26T03:13:15.495100+0000","last_became_peered":"2024-03-26T03:13:15.495100+0000","last_unstale":"2024-03-26T03:22:33.410714+0000","last_undegraded":"2024-03-26T03:22:33.410714+0000","last_fullsized":"2024-03-26T03:22:33.410714+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:37.466001+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:21:37.466001+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.466001+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:49.706819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1004,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.523429+0000","last_change":"2024-03-26T03:22:42.523457+0000","last_active":"2024-03-26T03:22:42.523429+0000","last_peered":"2024-03-26T03:22:42.523429+0000","last_clean":"2024-03-26T03:22:42.523429+0000","last_became_active":"2024-03-26T03:20:12.719873+0000","last_became_peered":"2024-03-26T03:20:12.719873+0000","last_unstale":"2024-03-26T03:22:42.523429+0000","last_undegraded":"2024-03-26T03:22:42.523429+0000","last_fullsized":"2024-03-26T03:22:42.523429+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:42.523355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.523355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:54.434095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"446'45","reported_seq":172,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416774+0000","last_change":"2024-03-26T03:22:13.361000+0000","last_active":"2024-03-26T03:22:33.416774+0000","last_peered":"2024-03-26T03:22:33.416774+0000","last_clean":"2024-03-26T03:22:33.416774+0000","last_became_active":"2024-03-26T03:20:57.356509+0000","last_became_peered":"2024-03-26T03:20:57.356509+0000","last_unstale":"2024-03-26T03:22:33.416774+0000","last_undegraded":"2024-03-26T03:22:33.416774+0000","last_fullsized":"2024-03-26T03:22:33.416774+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'45","last_scrub_stamp":"2024-03-26T03:22:13.360829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.360829+0000","objects_scrubbed":9,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:14.432214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":76,"num_read_kb":61,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410623+0000","last_change":"2024-03-26T03:21:43.445246+0000","last_active":"2024-03-26T03:22:33.410623+0000","last_peered":"2024-03-26T03:22:33.410623+0000","last_clean":"2024-03-26T03:22:33.410623+0000","last_became_active":"2024-03-26T03:21:42.741369+0000","last_became_peered":"2024-03-26T03:21:42.741369+0000","last_unstale":"2024-03-26T03:22:33.410623+0000","last_undegraded":"2024-03-26T03:22:33.410623+0000","last_fullsized":"2024-03-26T03:22:33.410623+0000","mapping_epoch":494,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:43.445121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:43.445121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.474043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":975,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548714+0000","last_change":"2024-03-26T03:21:29.932255+0000","last_active":"2024-03-26T03:22:34.548714+0000","last_peered":"2024-03-26T03:22:34.548714+0000","last_clean":"2024-03-26T03:22:34.548714+0000","last_became_active":"2024-03-26T02:59:03.206696+0000","last_became_peered":"2024-03-26T02:59:03.206696+0000","last_unstale":"2024-03-26T03:22:34.548714+0000","last_undegraded":"2024-03-26T03:22:34.548714+0000","last_fullsized":"2024-03-26T03:22:34.548714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:29.932140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:16.222744+0000","last_clean_scrub_stamp":"2024-03-26T03:21:29.932140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.824323+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'128","reported_seq":1078,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461260+0000","last_change":"2024-03-26T03:22:12.414426+0000","last_active":"2024-03-26T03:22:34.461260+0000","last_peered":"2024-03-26T03:22:34.461260+0000","last_clean":"2024-03-26T03:22:34.461260+0000","last_became_active":"2024-03-26T03:20:15.306151+0000","last_became_peered":"2024-03-26T03:20:15.306151+0000","last_unstale":"2024-03-26T03:22:34.461260+0000","last_undegraded":"2024-03-26T03:22:34.461260+0000","last_fullsized":"2024-03-26T03:22:34.461260+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:12.414324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:12.414324+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:36.107527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1019,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:36.946676+0000","last_change":"2024-03-26T03:22:36.946676+0000","last_active":"2024-03-26T03:22:36.946676+0000","last_peered":"2024-03-26T03:22:36.946676+0000","last_clean":"2024-03-26T03:22:36.946676+0000","last_became_active":"2024-03-26T03:20:12.732621+0000","last_became_peered":"2024-03-26T03:20:12.732621+0000","last_unstale":"2024-03-26T03:22:36.946676+0000","last_undegraded":"2024-03-26T03:22:36.946676+0000","last_fullsized":"2024-03-26T03:22:36.946676+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:36.946581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:36.946581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.948567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"446'42","reported_seq":140,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416782+0000","last_change":"2024-03-26T03:22:26.443959+0000","last_active":"2024-03-26T03:22:33.416782+0000","last_peered":"2024-03-26T03:22:33.416782+0000","last_clean":"2024-03-26T03:22:33.416782+0000","last_became_active":"2024-03-26T03:20:12.734007+0000","last_became_peered":"2024-03-26T03:20:12.734007+0000","last_unstale":"2024-03-26T03:22:33.416782+0000","last_undegraded":"2024-03-26T03:22:33.416782+0000","last_fullsized":"2024-03-26T03:22:33.416782+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'42","last_scrub_stamp":"2024-03-26T03:22:26.443851+0000","last_deep_scrub":"230'30","last_deep_scrub_stamp":"2024-03-26T03:11:48.575462+0000","last_clean_scrub_stamp":"2024-03-26T03:22:26.443851+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.007652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548956+0000","last_change":"2024-03-26T03:22:31.323174+0000","last_active":"2024-03-26T03:22:34.548956+0000","last_peered":"2024-03-26T03:22:34.548956+0000","last_clean":"2024-03-26T03:22:34.548956+0000","last_became_active":"2024-03-26T03:20:57.367476+0000","last_became_peered":"2024-03-26T03:20:57.367476+0000","last_unstale":"2024-03-26T03:22:34.548956+0000","last_undegraded":"2024-03-26T03:22:34.548956+0000","last_fullsized":"2024-03-26T03:22:34.548956+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:31.323076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:31.323076+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.323076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:56.126169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":126,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416301+0000","last_change":"2024-03-26T03:22:22.437982+0000","last_active":"2024-03-26T03:22:33.416301+0000","last_peered":"2024-03-26T03:22:33.416301+0000","last_clean":"2024-03-26T03:22:33.416301+0000","last_became_active":"2024-03-26T03:20:12.745764+0000","last_became_peered":"2024-03-26T03:20:12.745764+0000","last_unstale":"2024-03-26T03:22:33.416301+0000","last_undegraded":"2024-03-26T03:22:33.416301+0000","last_fullsized":"2024-03-26T03:22:33.416301+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:22.437874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:32.315559+0000","last_clean_scrub_stamp":"2024-03-26T03:22:22.437874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:41.213886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096562899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":957,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549214+0000","last_change":"2024-03-26T03:21:48.072831+0000","last_active":"2024-03-26T03:22:34.549214+0000","last_peered":"2024-03-26T03:22:34.549214+0000","last_clean":"2024-03-26T03:22:34.549214+0000","last_became_active":"2024-03-26T03:14:21.587947+0000","last_became_peered":"2024-03-26T03:14:21.587947+0000","last_unstale":"2024-03-26T03:22:34.549214+0000","last_undegraded":"2024-03-26T03:22:34.549214+0000","last_fullsized":"2024-03-26T03:22:34.549214+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:48.072751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:48.072751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:58.802616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"82'128","reported_seq":1097,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:51.803647+0000","last_change":"2024-03-26T03:22:51.803680+0000","last_active":"2024-03-26T03:22:51.803647+0000","last_peered":"2024-03-26T03:22:51.803647+0000","last_clean":"2024-03-26T03:22:51.803647+0000","last_became_active":"2024-03-26T03:17:59.602930+0000","last_became_peered":"2024-03-26T03:17:59.602930+0000","last_unstale":"2024-03-26T03:22:51.803647+0000","last_undegraded":"2024-03-26T03:22:51.803647+0000","last_fullsized":"2024-03-26T03:22:51.803647+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:51.803559+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:22:51.803559+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.589377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549771+0000","last_change":"2024-03-26T03:21:49.077589+0000","last_active":"2024-03-26T03:22:34.549771+0000","last_peered":"2024-03-26T03:22:34.549771+0000","last_clean":"2024-03-26T03:22:34.549771+0000","last_became_active":"2024-03-26T03:20:57.370336+0000","last_became_peered":"2024-03-26T03:20:57.370336+0000","last_unstale":"2024-03-26T03:22:34.549771+0000","last_undegraded":"2024-03-26T03:22:34.549771+0000","last_fullsized":"2024-03-26T03:22:34.549771+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:49.077483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:49.077483+0000","last_clean_scrub_stamp":"2024-03-26T03:21:49.077483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:10.283581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"477'43","reported_seq":1415,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298590+0000","last_change":"2024-03-26T03:22:09.369344+0000","last_active":"2024-03-26T03:22:47.298590+0000","last_peered":"2024-03-26T03:22:47.298590+0000","last_clean":"2024-03-26T03:22:47.298590+0000","last_became_active":"2024-03-26T03:20:57.372520+0000","last_became_peered":"2024-03-26T03:20:57.372520+0000","last_unstale":"2024-03-26T03:22:47.298590+0000","last_undegraded":"2024-03-26T03:22:47.298590+0000","last_fullsized":"2024-03-26T03:22:47.298590+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'43","last_scrub_stamp":"2024-03-26T03:22:09.369202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:09.369202+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:35.501851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"82'128","reported_seq":952,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548872+0000","last_change":"2024-03-26T03:21:29.084072+0000","last_active":"2024-03-26T03:22:34.548872+0000","last_peered":"2024-03-26T03:22:34.548872+0000","last_clean":"2024-03-26T03:22:34.548872+0000","last_became_active":"2024-03-26T03:21:00.294086+0000","last_became_peered":"2024-03-26T03:21:00.294086+0000","last_unstale":"2024-03-26T03:22:34.548872+0000","last_undegraded":"2024-03-26T03:22:34.548872+0000","last_fullsized":"2024-03-26T03:22:34.548872+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:29.083945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:29.083945+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:52.694410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1041,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553531+0000","last_change":"2024-03-26T03:22:16.900629+0000","last_active":"2024-03-26T03:22:33.553531+0000","last_peered":"2024-03-26T03:22:33.553531+0000","last_clean":"2024-03-26T03:22:33.553531+0000","last_became_active":"2024-03-26T03:20:57.363509+0000","last_became_peered":"2024-03-26T03:20:57.363509+0000","last_unstale":"2024-03-26T03:22:33.553531+0000","last_undegraded":"2024-03-26T03:22:33.553531+0000","last_fullsized":"2024-03-26T03:22:33.553531+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:16.900533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:05:40.720544+0000","last_clean_scrub_stamp":"2024-03-26T03:22:16.900533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:42.851427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"68'2","reported_seq":663,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468555+0000","last_change":"2024-03-26T03:21:22.638381+0000","last_active":"2024-03-26T03:22:34.468555+0000","last_peered":"2024-03-26T03:22:34.468555+0000","last_clean":"2024-03-26T03:22:34.468555+0000","last_became_active":"2024-03-26T03:14:21.656346+0000","last_became_peered":"2024-03-26T03:14:21.656346+0000","last_unstale":"2024-03-26T03:22:34.468555+0000","last_undegraded":"2024-03-26T03:22:34.468555+0000","last_fullsized":"2024-03-26T03:22:34.468555+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-26T03:21:22.638292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:22.638292+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:44.181572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0032310059999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":972,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:41.038852+0000","last_change":"2024-03-26T03:22:41.038852+0000","last_active":"2024-03-26T03:22:41.038852+0000","last_peered":"2024-03-26T03:22:41.038852+0000","last_clean":"2024-03-26T03:22:41.038852+0000","last_became_active":"2024-03-26T03:14:21.596294+0000","last_became_peered":"2024-03-26T03:14:21.596294+0000","last_unstale":"2024-03-26T03:22:41.038852+0000","last_undegraded":"2024-03-26T03:22:41.038852+0000","last_fullsized":"2024-03-26T03:22:41.038852+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:41.038751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:41.038751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.977874+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":879,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:54.319522+0000","last_change":"2024-03-26T03:22:54.319546+0000","last_active":"2024-03-26T03:22:54.319522+0000","last_peered":"2024-03-26T03:22:54.319522+0000","last_clean":"2024-03-26T03:22:54.319522+0000","last_became_active":"2024-03-26T03:21:26.458997+0000","last_became_peered":"2024-03-26T03:21:26.458997+0000","last_unstale":"2024-03-26T03:22:54.319522+0000","last_undegraded":"2024-03-26T03:22:54.319522+0000","last_fullsized":"2024-03-26T03:22:54.319522+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:54.319424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:16:52.721247+0000","last_clean_scrub_stamp":"2024-03-26T03:22:54.319424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:58.026180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"446'27","reported_seq":1091,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549444+0000","last_change":"2024-03-26T03:21:34.875552+0000","last_active":"2024-03-26T03:22:34.549444+0000","last_peered":"2024-03-26T03:22:34.549444+0000","last_clean":"2024-03-26T03:22:34.549444+0000","last_became_active":"2024-03-26T03:20:12.726379+0000","last_became_peered":"2024-03-26T03:20:12.726379+0000","last_unstale":"2024-03-26T03:22:34.549444+0000","last_undegraded":"2024-03-26T03:22:34.549444+0000","last_fullsized":"2024-03-26T03:22:34.549444+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'27","last_scrub_stamp":"2024-03-26T03:21:34.875458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:34.875458+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":791,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410255+0000","last_change":"2024-03-26T03:21:48.362289+0000","last_active":"2024-03-26T03:22:33.410255+0000","last_peered":"2024-03-26T03:22:33.410255+0000","last_clean":"2024-03-26T03:22:33.410255+0000","last_became_active":"2024-03-26T03:21:26.466136+0000","last_became_peered":"2024-03-26T03:21:26.466136+0000","last_unstale":"2024-03-26T03:22:33.410255+0000","last_undegraded":"2024-03-26T03:22:33.410255+0000","last_fullsized":"2024-03-26T03:22:33.410255+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:48.362208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:48.362208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:01.692281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"72'5","reported_seq":2429,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:54.646668+0000","last_change":"2024-03-26T03:21:41.967044+0000","last_active":"2024-03-26T03:22:54.646668+0000","last_peered":"2024-03-26T03:22:54.646668+0000","last_clean":"2024-03-26T03:22:54.646668+0000","last_became_active":"2024-03-26T02:58:56.930943+0000","last_became_peered":"2024-03-26T02:58:56.930943+0000","last_unstale":"2024-03-26T03:22:54.646668+0000","last_undegraded":"2024-03-26T03:22:54.646668+0000","last_fullsized":"2024-03-26T03:22:54.646668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-26T03:21:41.966885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:41.966885+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.693689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073466500000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1427,"num_read_kb":1422,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:45.555213+0000","last_change":"2024-03-26T03:22:45.555255+0000","last_active":"2024-03-26T03:22:45.555213+0000","last_peered":"2024-03-26T03:22:45.555213+0000","last_clean":"2024-03-26T03:22:45.555213+0000","last_became_active":"2024-03-26T03:20:57.362304+0000","last_became_peered":"2024-03-26T03:20:57.362304+0000","last_unstale":"2024-03-26T03:22:45.555213+0000","last_undegraded":"2024-03-26T03:22:45.555213+0000","last_fullsized":"2024-03-26T03:22:45.555213+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:45.555093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:41.268157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:45.555093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:46.190863+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"82'128","reported_seq":1073,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549404+0000","last_change":"2024-03-26T03:21:54.144030+0000","last_active":"2024-03-26T03:22:34.549404+0000","last_peered":"2024-03-26T03:22:34.549404+0000","last_clean":"2024-03-26T03:22:34.549404+0000","last_became_active":"2024-03-26T03:21:00.290993+0000","last_became_peered":"2024-03-26T03:21:00.290993+0000","last_unstale":"2024-03-26T03:22:34.549404+0000","last_undegraded":"2024-03-26T03:22:34.549404+0000","last_fullsized":"2024-03-26T03:22:34.549404+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:54.143892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:54.143892+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:58.637015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":96,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468213+0000","last_change":"2024-03-26T03:22:05.700859+0000","last_active":"2024-03-26T03:22:34.468213+0000","last_peered":"2024-03-26T03:22:34.468213+0000","last_clean":"2024-03-26T03:22:34.468213+0000","last_became_active":"2024-03-26T03:20:42.255520+0000","last_became_peered":"2024-03-26T03:20:42.255520+0000","last_unstale":"2024-03-26T03:22:34.468213+0000","last_undegraded":"2024-03-26T03:22:34.468213+0000","last_fullsized":"2024-03-26T03:22:34.468213+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:05.700775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:05.700775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:35.202375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"446'71","reported_seq":1376,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298500+0000","last_change":"2024-03-26T03:21:59.973704+0000","last_active":"2024-03-26T03:22:47.298500+0000","last_peered":"2024-03-26T03:22:47.298500+0000","last_clean":"2024-03-26T03:22:47.298500+0000","last_became_active":"2024-03-26T03:14:21.587656+0000","last_became_peered":"2024-03-26T03:14:21.587656+0000","last_unstale":"2024-03-26T03:22:47.298500+0000","last_undegraded":"2024-03-26T03:22:47.298500+0000","last_fullsized":"2024-03-26T03:22:47.298500+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'71","last_scrub_stamp":"2024-03-26T03:21:59.973561+0000","last_deep_scrub":"86'45","last_deep_scrub_stamp":"2024-03-26T03:00:13.375080+0000","last_clean_scrub_stamp":"2024-03-26T03:21:59.973561+0000","objects_scrubbed":11,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:17.204881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":259,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553451+0000","last_change":"2024-03-26T03:22:25.002868+0000","last_active":"2024-03-26T03:22:33.553451+0000","last_peered":"2024-03-26T03:22:33.553451+0000","last_clean":"2024-03-26T03:22:33.553451+0000","last_became_active":"2024-03-26T03:21:43.857186+0000","last_became_peered":"2024-03-26T03:21:43.857186+0000","last_unstale":"2024-03-26T03:22:33.553451+0000","last_undegraded":"2024-03-26T03:22:33.553451+0000","last_fullsized":"2024-03-26T03:22:33.553451+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:25.002761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.002761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:36.907401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":680,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469404+0000","last_change":"2024-03-26T03:21:46.379636+0000","last_active":"2024-03-26T03:22:34.469404+0000","last_peered":"2024-03-26T03:22:34.469404+0000","last_clean":"2024-03-26T03:22:34.469404+0000","last_became_active":"2024-03-26T03:20:57.353414+0000","last_became_peered":"2024-03-26T03:20:57.353414+0000","last_unstale":"2024-03-26T03:22:34.469404+0000","last_undegraded":"2024-03-26T03:22:34.469404+0000","last_fullsized":"2024-03-26T03:22:34.469404+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:46.379554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:19:12.537519+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.379554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:52.441795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'128","reported_seq":341,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553133+0000","last_change":"2024-03-26T03:21:45.876199+0000","last_active":"2024-03-26T03:22:33.553133+0000","last_peered":"2024-03-26T03:22:33.553133+0000","last_clean":"2024-03-26T03:22:33.553133+0000","last_became_active":"2024-03-26T03:21:43.566527+0000","last_became_peered":"2024-03-26T03:21:43.566527+0000","last_unstale":"2024-03-26T03:22:33.553133+0000","last_undegraded":"2024-03-26T03:22:33.553133+0000","last_fullsized":"2024-03-26T03:22:33.553133+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:45.876100+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:21:45.876100+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.015082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"460'13","reported_seq":164,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298504+0000","last_change":"2024-03-26T03:22:01.354367+0000","last_active":"2024-03-26T03:22:47.298504+0000","last_peered":"2024-03-26T03:22:47.298504+0000","last_clean":"2024-03-26T03:22:47.298504+0000","last_became_active":"2024-03-26T03:20:12.737843+0000","last_became_peered":"2024-03-26T03:20:12.737843+0000","last_unstale":"2024-03-26T03:22:47.298504+0000","last_undegraded":"2024-03-26T03:22:47.298504+0000","last_fullsized":"2024-03-26T03:22:47.298504+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'13","last_scrub_stamp":"2024-03-26T03:22:01.354278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:01.354278+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:15.852785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"446'42","reported_seq":1086,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548736+0000","last_change":"2024-03-26T03:22:31.944057+0000","last_active":"2024-03-26T03:22:34.548736+0000","last_peered":"2024-03-26T03:22:34.548736+0000","last_clean":"2024-03-26T03:22:34.548736+0000","last_became_active":"2024-03-26T03:17:56.568844+0000","last_became_peered":"2024-03-26T03:17:56.568844+0000","last_unstale":"2024-03-26T03:22:34.548736+0000","last_undegraded":"2024-03-26T03:22:34.548736+0000","last_fullsized":"2024-03-26T03:22:34.548736+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'42","last_scrub_stamp":"2024-03-26T03:22:31.943952+0000","last_deep_scrub":"446'42","last_deep_scrub_stamp":"2024-03-26T03:22:31.943952+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.943952+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:50.597560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1016,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:38.785021+0000","last_change":"2024-03-26T03:22:38.785045+0000","last_active":"2024-03-26T03:22:38.785021+0000","last_peered":"2024-03-26T03:22:38.785021+0000","last_clean":"2024-03-26T03:22:38.785021+0000","last_became_active":"2024-03-26T03:20:57.364467+0000","last_became_peered":"2024-03-26T03:20:57.364467+0000","last_unstale":"2024-03-26T03:22:38.785021+0000","last_undegraded":"2024-03-26T03:22:38.785021+0000","last_fullsized":"2024-03-26T03:22:38.785021+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:38.784961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:38.784961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.147376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":126,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.415978+0000","last_change":"2024-03-26T03:22:10.276162+0000","last_active":"2024-03-26T03:22:33.415978+0000","last_peered":"2024-03-26T03:22:33.415978+0000","last_clean":"2024-03-26T03:22:33.415978+0000","last_became_active":"2024-03-26T03:20:12.746518+0000","last_became_peered":"2024-03-26T03:20:12.746518+0000","last_unstale":"2024-03-26T03:22:33.415978+0000","last_undegraded":"2024-03-26T03:22:33.415978+0000","last_fullsized":"2024-03-26T03:22:33.415978+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:10.276075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:03:53.959237+0000","last_clean_scrub_stamp":"2024-03-26T03:22:10.276075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.374414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"82'128","reported_seq":1079,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461133+0000","last_change":"2024-03-26T03:22:14.388219+0000","last_active":"2024-03-26T03:22:34.461133+0000","last_peered":"2024-03-26T03:22:34.461133+0000","last_clean":"2024-03-26T03:22:34.461133+0000","last_became_active":"2024-03-26T03:21:00.294477+0000","last_became_peered":"2024-03-26T03:21:00.294477+0000","last_unstale":"2024-03-26T03:22:34.461133+0000","last_undegraded":"2024-03-26T03:22:34.461133+0000","last_fullsized":"2024-03-26T03:22:34.461133+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:14.388136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:14.388136+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:15.088059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"82'128","reported_seq":799,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468909+0000","last_change":"2024-03-26T03:21:50.413946+0000","last_active":"2024-03-26T03:22:34.468909+0000","last_peered":"2024-03-26T03:22:34.468909+0000","last_clean":"2024-03-26T03:22:34.468909+0000","last_became_active":"2024-03-26T03:20:15.308673+0000","last_became_peered":"2024-03-26T03:20:15.308673+0000","last_unstale":"2024-03-26T03:22:34.468909+0000","last_undegraded":"2024-03-26T03:22:34.468909+0000","last_fullsized":"2024-03-26T03:22:34.468909+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:50.413841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:50.413841+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:10.104423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":816,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409814+0000","last_change":"2024-03-26T03:22:05.568746+0000","last_active":"2024-03-26T03:22:33.409814+0000","last_peered":"2024-03-26T03:22:33.409814+0000","last_clean":"2024-03-26T03:22:33.409814+0000","last_became_active":"2024-03-26T03:21:26.464824+0000","last_became_peered":"2024-03-26T03:21:26.464824+0000","last_unstale":"2024-03-26T03:22:33.409814+0000","last_undegraded":"2024-03-26T03:22:33.409814+0000","last_fullsized":"2024-03-26T03:22:33.409814+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:05.568661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:05.568661+0000","last_clean_scrub_stamp":"2024-03-26T03:22:05.568661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:29.644568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"474'43","reported_seq":1058,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409958+0000","last_change":"2024-03-26T03:22:11.351736+0000","last_active":"2024-03-26T03:22:33.409958+0000","last_peered":"2024-03-26T03:22:33.409958+0000","last_clean":"2024-03-26T03:22:33.409958+0000","last_became_active":"2024-03-26T03:14:21.599848+0000","last_became_peered":"2024-03-26T03:14:21.599848+0000","last_unstale":"2024-03-26T03:22:33.409958+0000","last_undegraded":"2024-03-26T03:22:33.409958+0000","last_fullsized":"2024-03-26T03:22:33.409958+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'43","last_scrub_stamp":"2024-03-26T03:22:11.351575+0000","last_deep_scrub":"474'43","last_deep_scrub_stamp":"2024-03-26T03:20:49.325072+0000","last_clean_scrub_stamp":"2024-03-26T03:22:11.351575+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:14.351160+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"64'3","reported_seq":1077,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:54.363834+0000","last_change":"2024-03-26T03:22:54.363871+0000","last_active":"2024-03-26T03:22:54.363834+0000","last_peered":"2024-03-26T03:22:54.363834+0000","last_clean":"2024-03-26T03:22:54.363834+0000","last_became_active":"2024-03-26T03:20:57.358824+0000","last_became_peered":"2024-03-26T03:20:57.358824+0000","last_unstale":"2024-03-26T03:22:54.363834+0000","last_undegraded":"2024-03-26T03:22:54.363834+0000","last_fullsized":"2024-03-26T03:22:54.363834+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'3","last_scrub_stamp":"2024-03-26T03:22:54.363764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:54.363764+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:20.911892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":661,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.793412+0000","last_change":"2024-03-26T03:22:47.793436+0000","last_active":"2024-03-26T03:22:47.793412+0000","last_peered":"2024-03-26T03:22:47.793412+0000","last_clean":"2024-03-26T03:22:47.793412+0000","last_became_active":"2024-03-26T03:20:57.354428+0000","last_became_peered":"2024-03-26T03:20:57.354428+0000","last_unstale":"2024-03-26T03:22:47.793412+0000","last_undegraded":"2024-03-26T03:22:47.793412+0000","last_fullsized":"2024-03-26T03:22:47.793412+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:47.793341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:47.793341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:55.045908+0000","scrub_duration":171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"82'128","reported_seq":1109,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548828+0000","last_change":"2024-03-26T03:22:22.986714+0000","last_active":"2024-03-26T03:22:34.548828+0000","last_peered":"2024-03-26T03:22:34.548828+0000","last_clean":"2024-03-26T03:22:34.548828+0000","last_became_active":"2024-03-26T03:17:59.607165+0000","last_became_peered":"2024-03-26T03:17:59.607165+0000","last_unstale":"2024-03-26T03:22:34.548828+0000","last_undegraded":"2024-03-26T03:22:34.548828+0000","last_fullsized":"2024-03-26T03:22:34.548828+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:22.986623+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:28.248568+0000","last_clean_scrub_stamp":"2024-03-26T03:22:22.986623+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:46.327164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"446'33","reported_seq":160,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.417141+0000","last_change":"2024-03-26T03:22:25.446976+0000","last_active":"2024-03-26T03:22:33.417141+0000","last_peered":"2024-03-26T03:22:33.417141+0000","last_clean":"2024-03-26T03:22:33.417141+0000","last_became_active":"2024-03-26T03:20:57.355137+0000","last_became_peered":"2024-03-26T03:20:57.355137+0000","last_unstale":"2024-03-26T03:22:33.417141+0000","last_undegraded":"2024-03-26T03:22:33.417141+0000","last_fullsized":"2024-03-26T03:22:33.417141+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'33","last_scrub_stamp":"2024-03-26T03:22:25.446872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.446872+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:40.235392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"64'1","reported_seq":843,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409696+0000","last_change":"2024-03-26T03:22:31.539414+0000","last_active":"2024-03-26T03:22:33.409696+0000","last_peered":"2024-03-26T03:22:33.409696+0000","last_clean":"2024-03-26T03:22:33.409696+0000","last_became_active":"2024-03-26T03:21:26.462548+0000","last_became_peered":"2024-03-26T03:21:26.462548+0000","last_unstale":"2024-03-26T03:22:33.409696+0000","last_undegraded":"2024-03-26T03:22:33.409696+0000","last_fullsized":"2024-03-26T03:22:33.409696+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"64'1","last_scrub_stamp":"2024-03-26T03:22:31.539325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.539325+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:38.560057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":678,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469404+0000","last_change":"2024-03-26T03:21:32.419713+0000","last_active":"2024-03-26T03:22:34.469404+0000","last_peered":"2024-03-26T03:22:34.469404+0000","last_clean":"2024-03-26T03:22:34.469404+0000","last_became_active":"2024-03-26T03:20:57.352698+0000","last_became_peered":"2024-03-26T03:20:57.352698+0000","last_unstale":"2024-03-26T03:22:34.469404+0000","last_undegraded":"2024-03-26T03:22:34.469404+0000","last_fullsized":"2024-03-26T03:22:34.469404+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:32.419625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:32.419625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.198321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549877+0000","last_change":"2024-03-26T03:21:24.087182+0000","last_active":"2024-03-26T03:22:34.549877+0000","last_peered":"2024-03-26T03:22:34.549877+0000","last_clean":"2024-03-26T03:22:34.549877+0000","last_became_active":"2024-03-26T03:20:57.363228+0000","last_became_peered":"2024-03-26T03:20:57.363228+0000","last_unstale":"2024-03-26T03:22:34.549877+0000","last_undegraded":"2024-03-26T03:22:34.549877+0000","last_fullsized":"2024-03-26T03:22:34.549877+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:24.087097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:08:08.941442+0000","last_clean_scrub_stamp":"2024-03-26T03:21:24.087097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:47.157486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"82'128","reported_seq":504,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:48.398162+0000","last_change":"2024-03-26T03:22:48.398189+0000","last_active":"2024-03-26T03:22:48.398162+0000","last_peered":"2024-03-26T03:22:48.398162+0000","last_clean":"2024-03-26T03:22:48.398162+0000","last_became_active":"2024-03-26T03:17:44.278017+0000","last_became_peered":"2024-03-26T03:17:44.278017+0000","last_unstale":"2024-03-26T03:22:48.398162+0000","last_undegraded":"2024-03-26T03:22:48.398162+0000","last_fullsized":"2024-03-26T03:22:48.398162+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:48.398091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:48.398091+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:07.822795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"446'24","reported_seq":764,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469172+0000","last_change":"2024-03-26T03:22:10.710050+0000","last_active":"2024-03-26T03:22:34.469172+0000","last_peered":"2024-03-26T03:22:34.469172+0000","last_clean":"2024-03-26T03:22:34.469172+0000","last_became_active":"2024-03-26T03:20:57.353003+0000","last_became_peered":"2024-03-26T03:20:57.353003+0000","last_unstale":"2024-03-26T03:22:34.469172+0000","last_undegraded":"2024-03-26T03:22:34.469172+0000","last_fullsized":"2024-03-26T03:22:34.469172+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'24","last_scrub_stamp":"2024-03-26T03:22:10.709951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:10.709951+0000","objects_scrubbed":4,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:14.913269+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1011,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:39.550697+0000","last_change":"2024-03-26T03:22:39.550722+0000","last_active":"2024-03-26T03:22:39.550697+0000","last_peered":"2024-03-26T03:22:39.550697+0000","last_clean":"2024-03-26T03:22:39.550697+0000","last_became_active":"2024-03-26T03:17:56.570158+0000","last_became_peered":"2024-03-26T03:17:56.570158+0000","last_unstale":"2024-03-26T03:22:39.550697+0000","last_undegraded":"2024-03-26T03:22:39.550697+0000","last_fullsized":"2024-03-26T03:22:39.550697+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:39.550620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:39.550620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:04.524735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":974,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553863+0000","last_change":"2024-03-26T03:21:50.898485+0000","last_active":"2024-03-26T03:22:33.553863+0000","last_peered":"2024-03-26T03:22:33.553863+0000","last_clean":"2024-03-26T03:22:33.553863+0000","last_became_active":"2024-03-26T03:14:21.592372+0000","last_became_peered":"2024-03-26T03:14:21.592372+0000","last_unstale":"2024-03-26T03:22:33.553863+0000","last_undegraded":"2024-03-26T03:22:33.553863+0000","last_fullsized":"2024-03-26T03:22:33.553863+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:50.898381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:50.898381+0000","last_clean_scrub_stamp":"2024-03-26T03:21:50.898381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:17.188243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.467068+0000","last_change":"2024-03-26T03:22:37.467092+0000","last_active":"2024-03-26T03:22:37.467068+0000","last_peered":"2024-03-26T03:22:37.467068+0000","last_clean":"2024-03-26T03:22:37.467068+0000","last_became_active":"2024-03-26T03:20:57.379539+0000","last_became_peered":"2024-03-26T03:20:57.379539+0000","last_unstale":"2024-03-26T03:22:37.467068+0000","last_undegraded":"2024-03-26T03:22:37.467068+0000","last_fullsized":"2024-03-26T03:22:37.467068+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:37.466985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.466985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:44.153349+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"82'128","reported_seq":430,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416144+0000","last_change":"2024-03-26T03:22:30.534188+0000","last_active":"2024-03-26T03:22:33.416144+0000","last_peered":"2024-03-26T03:22:33.416144+0000","last_clean":"2024-03-26T03:22:33.416144+0000","last_became_active":"2024-03-26T03:20:15.317985+0000","last_became_peered":"2024-03-26T03:20:15.317985+0000","last_unstale":"2024-03-26T03:22:33.416144+0000","last_undegraded":"2024-03-26T03:22:33.416144+0000","last_fullsized":"2024-03-26T03:22:33.416144+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:30.534097+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:10:04.204409+0000","last_clean_scrub_stamp":"2024-03-26T03:22:30.534097+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:31.820328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"446'49","reported_seq":793,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468814+0000","last_change":"2024-03-26T03:22:23.640850+0000","last_active":"2024-03-26T03:22:34.468814+0000","last_peered":"2024-03-26T03:22:34.468814+0000","last_clean":"2024-03-26T03:22:34.468814+0000","last_became_active":"2024-03-26T03:20:57.353406+0000","last_became_peered":"2024-03-26T03:20:57.353406+0000","last_unstale":"2024-03-26T03:22:34.468814+0000","last_undegraded":"2024-03-26T03:22:34.468814+0000","last_fullsized":"2024-03-26T03:22:34.468814+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'49","last_scrub_stamp":"2024-03-26T03:22:23.640754+0000","last_deep_scrub":"231'39","last_deep_scrub_stamp":"2024-03-26T03:16:49.904637+0000","last_clean_scrub_stamp":"2024-03-26T03:22:23.640754+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:26.013929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553346+0000","last_change":"2024-03-26T03:22:05.898382+0000","last_active":"2024-03-26T03:22:33.553346+0000","last_peered":"2024-03-26T03:22:33.553346+0000","last_clean":"2024-03-26T03:22:33.553346+0000","last_became_active":"2024-03-26T03:20:57.365743+0000","last_became_peered":"2024-03-26T03:20:57.365743+0000","last_unstale":"2024-03-26T03:22:33.553346+0000","last_undegraded":"2024-03-26T03:22:33.553346+0000","last_fullsized":"2024-03-26T03:22:33.553346+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:05.898268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.320691+0000","last_clean_scrub_stamp":"2024-03-26T03:22:05.898268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.243361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"471'15","reported_seq":1339,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298532+0000","last_change":"2024-03-26T03:22:37.947982+0000","last_active":"2024-03-26T03:22:47.298532+0000","last_peered":"2024-03-26T03:22:47.298532+0000","last_clean":"2024-03-26T03:22:47.298532+0000","last_became_active":"2024-03-26T03:20:42.255252+0000","last_became_peered":"2024-03-26T03:20:42.255252+0000","last_unstale":"2024-03-26T03:22:47.298532+0000","last_undegraded":"2024-03-26T03:22:47.298532+0000","last_fullsized":"2024-03-26T03:22:47.298532+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'15","last_scrub_stamp":"2024-03-26T03:22:37.947825+0000","last_deep_scrub":"201'9","last_deep_scrub_stamp":"2024-03-26T03:06:36.601857+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.947825+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:48.429164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.417041+0000","last_change":"2024-03-26T03:22:27.472268+0000","last_active":"2024-03-26T03:22:33.417041+0000","last_peered":"2024-03-26T03:22:33.417041+0000","last_clean":"2024-03-26T03:22:33.417041+0000","last_became_active":"2024-03-26T03:20:57.357744+0000","last_became_peered":"2024-03-26T03:20:57.357744+0000","last_unstale":"2024-03-26T03:22:33.417041+0000","last_undegraded":"2024-03-26T03:22:33.417041+0000","last_fullsized":"2024-03-26T03:22:33.417041+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:27.472152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.472152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.703760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"82'128","reported_seq":739,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553231+0000","last_change":"2024-03-26T03:21:46.887888+0000","last_active":"2024-03-26T03:22:33.553231+0000","last_peered":"2024-03-26T03:22:33.553231+0000","last_clean":"2024-03-26T03:22:33.553231+0000","last_became_active":"2024-03-26T03:21:43.566515+0000","last_became_peered":"2024-03-26T03:21:43.566515+0000","last_unstale":"2024-03-26T03:22:33.553231+0000","last_undegraded":"2024-03-26T03:22:33.553231+0000","last_fullsized":"2024-03-26T03:22:33.553231+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:46.887750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:21:46.887750+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:05.129158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"475'32","reported_seq":777,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553851+0000","last_change":"2024-03-26T03:22:19.912662+0000","last_active":"2024-03-26T03:22:33.553851+0000","last_peered":"2024-03-26T03:22:33.553851+0000","last_clean":"2024-03-26T03:22:33.553851+0000","last_became_active":"2024-03-26T03:21:43.568013+0000","last_became_peered":"2024-03-26T03:21:43.568013+0000","last_unstale":"2024-03-26T03:22:33.553851+0000","last_undegraded":"2024-03-26T03:22:33.553851+0000","last_fullsized":"2024-03-26T03:22:33.553851+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'32","last_scrub_stamp":"2024-03-26T03:22:19.912549+0000","last_deep_scrub":"475'32","last_deep_scrub_stamp":"2024-03-26T03:20:54.654445+0000","last_clean_scrub_stamp":"2024-03-26T03:22:19.912549+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.469868+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":134,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:50.518175+0000","last_change":"2024-03-26T03:22:50.518215+0000","last_active":"2024-03-26T03:22:50.518175+0000","last_peered":"2024-03-26T03:22:50.518175+0000","last_clean":"2024-03-26T03:22:50.518175+0000","last_became_active":"2024-03-26T03:20:12.746773+0000","last_became_peered":"2024-03-26T03:20:12.746773+0000","last_unstale":"2024-03-26T03:22:50.518175+0000","last_undegraded":"2024-03-26T03:22:50.518175+0000","last_fullsized":"2024-03-26T03:22:50.518175+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:50.518036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:46.666837+0000","last_clean_scrub_stamp":"2024-03-26T03:22:50.518036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.225697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"477'12","reported_seq":819,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298652+0000","last_change":"2024-03-26T03:21:15.550286+0000","last_active":"2024-03-26T03:22:47.298652+0000","last_peered":"2024-03-26T03:22:47.298652+0000","last_clean":"2024-03-26T03:22:47.298652+0000","last_became_active":"2024-03-26T03:20:57.362408+0000","last_became_peered":"2024-03-26T03:20:57.362408+0000","last_unstale":"2024-03-26T03:22:47.298652+0000","last_undegraded":"2024-03-26T03:22:47.298652+0000","last_fullsized":"2024-03-26T03:22:47.298652+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'12","last_scrub_stamp":"2024-03-26T03:21:15.550162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:21:15.550162+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"71'1","reported_seq":961,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410834+0000","last_change":"2024-03-26T03:21:34.659309+0000","last_active":"2024-03-26T03:22:33.410834+0000","last_peered":"2024-03-26T03:22:33.410834+0000","last_clean":"2024-03-26T03:22:33.410834+0000","last_became_active":"2024-03-26T02:59:02.944652+0000","last_became_peered":"2024-03-26T02:59:02.944652+0000","last_unstale":"2024-03-26T03:22:33.410834+0000","last_undegraded":"2024-03-26T03:22:33.410834+0000","last_fullsized":"2024-03-26T03:22:33.410834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'1","last_scrub_stamp":"2024-03-26T03:21:34.659199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:34.659199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:46.637045+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"82'128","reported_seq":1098,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553988+0000","last_change":"2024-03-26T03:22:25.985505+0000","last_active":"2024-03-26T03:22:33.553988+0000","last_peered":"2024-03-26T03:22:33.553988+0000","last_clean":"2024-03-26T03:22:33.553988+0000","last_became_active":"2024-03-26T03:20:15.295795+0000","last_became_peered":"2024-03-26T03:20:15.295795+0000","last_unstale":"2024-03-26T03:22:33.553988+0000","last_undegraded":"2024-03-26T03:22:33.553988+0000","last_fullsized":"2024-03-26T03:22:33.553988+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:25.985407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.985407+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:50.668876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"446'25","reported_seq":113,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:35.493636+0000","last_change":"2024-03-26T03:22:35.493666+0000","last_active":"2024-03-26T03:22:35.493636+0000","last_peered":"2024-03-26T03:22:35.493636+0000","last_clean":"2024-03-26T03:22:35.493636+0000","last_became_active":"2024-03-26T03:20:42.258677+0000","last_became_peered":"2024-03-26T03:20:42.258677+0000","last_unstale":"2024-03-26T03:22:35.493636+0000","last_undegraded":"2024-03-26T03:22:35.493636+0000","last_fullsized":"2024-03-26T03:22:35.493636+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'25","last_scrub_stamp":"2024-03-26T03:22:35.493575+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-26T03:11:46.227482+0000","last_clean_scrub_stamp":"2024-03-26T03:22:35.493575+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.175400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":976,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:53.460527+0000","last_change":"2024-03-26T03:22:53.460554+0000","last_active":"2024-03-26T03:22:53.460527+0000","last_peered":"2024-03-26T03:22:53.460527+0000","last_clean":"2024-03-26T03:22:53.460527+0000","last_became_active":"2024-03-26T03:14:21.588178+0000","last_became_peered":"2024-03-26T03:14:21.588178+0000","last_unstale":"2024-03-26T03:22:53.460527+0000","last_undegraded":"2024-03-26T03:22:53.460527+0000","last_fullsized":"2024-03-26T03:22:53.460527+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:53.460440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:53.460440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:07.723628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":663,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.560923+0000","last_change":"2024-03-26T03:22:34.560957+0000","last_active":"2024-03-26T03:22:34.560923+0000","last_peered":"2024-03-26T03:22:34.560923+0000","last_clean":"2024-03-26T03:22:34.560923+0000","last_became_active":"2024-03-26T03:20:12.749158+0000","last_became_peered":"2024-03-26T03:20:12.749158+0000","last_unstale":"2024-03-26T03:22:34.560923+0000","last_undegraded":"2024-03-26T03:22:34.560923+0000","last_fullsized":"2024-03-26T03:22:34.560923+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:34.560837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:20:47.697347+0000","last_clean_scrub_stamp":"2024-03-26T03:22:34.560837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.676849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":126,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.415940+0000","last_change":"2024-03-26T03:21:55.318317+0000","last_active":"2024-03-26T03:22:33.415940+0000","last_peered":"2024-03-26T03:22:33.415940+0000","last_clean":"2024-03-26T03:22:33.415940+0000","last_became_active":"2024-03-26T03:20:12.736042+0000","last_became_peered":"2024-03-26T03:20:12.736042+0000","last_unstale":"2024-03-26T03:22:33.415940+0000","last_undegraded":"2024-03-26T03:22:33.415940+0000","last_fullsized":"2024-03-26T03:22:33.415940+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:55.318231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:55.318231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:08.239770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"82'128","reported_seq":766,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.819993+0000","last_change":"2024-03-26T03:22:37.820018+0000","last_active":"2024-03-26T03:22:37.819993+0000","last_peered":"2024-03-26T03:22:37.819993+0000","last_clean":"2024-03-26T03:22:37.819993+0000","last_became_active":"2024-03-26T03:20:15.311071+0000","last_became_peered":"2024-03-26T03:20:15.311071+0000","last_unstale":"2024-03-26T03:22:37.819993+0000","last_undegraded":"2024-03-26T03:22:37.819993+0000","last_fullsized":"2024-03-26T03:22:37.819993+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:37.819923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.819923+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.347791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"446'27","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410235+0000","last_change":"2024-03-26T03:22:27.152212+0000","last_active":"2024-03-26T03:22:33.410235+0000","last_peered":"2024-03-26T03:22:33.410235+0000","last_clean":"2024-03-26T03:22:33.410235+0000","last_became_active":"2024-03-26T02:58:58.925522+0000","last_became_peered":"2024-03-26T02:58:58.925522+0000","last_unstale":"2024-03-26T03:22:33.410235+0000","last_undegraded":"2024-03-26T03:22:33.410235+0000","last_fullsized":"2024-03-26T03:22:33.410235+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'27","last_scrub_stamp":"2024-03-26T03:22:27.152084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.152084+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:40.367990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1010,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.785363+0000","last_change":"2024-03-26T03:22:47.785383+0000","last_active":"2024-03-26T03:22:47.785363+0000","last_peered":"2024-03-26T03:22:47.785363+0000","last_clean":"2024-03-26T03:22:47.785363+0000","last_became_active":"2024-03-26T03:20:12.720143+0000","last_became_peered":"2024-03-26T03:20:12.720143+0000","last_unstale":"2024-03-26T03:22:47.785363+0000","last_undegraded":"2024-03-26T03:22:47.785363+0000","last_fullsized":"2024-03-26T03:22:47.785363+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:47.785304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:47.785304+0000","last_clean_scrub_stamp":"2024-03-26T03:22:47.785304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:14.870263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":952,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548848+0000","last_change":"2024-03-26T03:22:04.988448+0000","last_active":"2024-03-26T03:22:34.548848+0000","last_peered":"2024-03-26T03:22:34.548848+0000","last_clean":"2024-03-26T03:22:34.548848+0000","last_became_active":"2024-03-26T03:14:21.665690+0000","last_became_peered":"2024-03-26T03:14:21.665690+0000","last_unstale":"2024-03-26T03:22:34.548848+0000","last_undegraded":"2024-03-26T03:22:34.548848+0000","last_fullsized":"2024-03-26T03:22:34.548848+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:04.988369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:04.988369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:11.111872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":667,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468350+0000","last_change":"2024-03-26T03:22:08.658538+0000","last_active":"2024-03-26T03:22:34.468350+0000","last_peered":"2024-03-26T03:22:34.468350+0000","last_clean":"2024-03-26T03:22:34.468350+0000","last_became_active":"2024-03-26T03:20:12.745809+0000","last_became_peered":"2024-03-26T03:20:12.745809+0000","last_unstale":"2024-03-26T03:22:34.468350+0000","last_undegraded":"2024-03-26T03:22:34.468350+0000","last_fullsized":"2024-03-26T03:22:34.468350+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:08.658406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:08.658406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:09.316878+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"82'128","reported_seq":675,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549778+0000","last_change":"2024-03-26T03:21:58.207690+0000","last_active":"2024-03-26T03:22:34.549778+0000","last_peered":"2024-03-26T03:22:34.549778+0000","last_clean":"2024-03-26T03:22:34.549778+0000","last_became_active":"2024-03-26T03:21:00.297045+0000","last_became_peered":"2024-03-26T03:21:00.297045+0000","last_unstale":"2024-03-26T03:22:34.549778+0000","last_undegraded":"2024-03-26T03:22:34.549778+0000","last_fullsized":"2024-03-26T03:22:34.549778+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:58.207539+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:12:36.136805+0000","last_clean_scrub_stamp":"2024-03-26T03:21:58.207539+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:17.663822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"465'14","reported_seq":686,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468339+0000","last_change":"2024-03-26T03:21:52.468863+0000","last_active":"2024-03-26T03:22:34.468339+0000","last_peered":"2024-03-26T03:22:34.468339+0000","last_clean":"2024-03-26T03:22:34.468339+0000","last_became_active":"2024-03-26T03:14:21.602211+0000","last_became_peered":"2024-03-26T03:14:21.602211+0000","last_unstale":"2024-03-26T03:22:34.468339+0000","last_undegraded":"2024-03-26T03:22:34.468339+0000","last_fullsized":"2024-03-26T03:22:34.468339+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'14","last_scrub_stamp":"2024-03-26T03:21:52.468721+0000","last_deep_scrub":"465'14","last_deep_scrub_stamp":"2024-03-26T03:20:20.501710+0000","last_clean_scrub_stamp":"2024-03-26T03:21:52.468721+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:04.599700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1013,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553992+0000","last_change":"2024-03-26T03:22:13.861284+0000","last_active":"2024-03-26T03:22:33.553992+0000","last_peered":"2024-03-26T03:22:33.553992+0000","last_clean":"2024-03-26T03:22:33.553992+0000","last_became_active":"2024-03-26T02:58:56.923369+0000","last_became_peered":"2024-03-26T02:58:56.923369+0000","last_unstale":"2024-03-26T03:22:33.553992+0000","last_undegraded":"2024-03-26T03:22:33.553992+0000","last_fullsized":"2024-03-26T03:22:33.553992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:13.861207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:18.868661+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.861207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:18.097192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.460444+0000","last_change":"2024-03-26T03:22:02.352570+0000","last_active":"2024-03-26T03:22:34.460444+0000","last_peered":"2024-03-26T03:22:34.460444+0000","last_clean":"2024-03-26T03:22:34.460444+0000","last_became_active":"2024-03-26T03:20:57.356298+0000","last_became_peered":"2024-03-26T03:20:57.356298+0000","last_unstale":"2024-03-26T03:22:34.460444+0000","last_undegraded":"2024-03-26T03:22:34.460444+0000","last_fullsized":"2024-03-26T03:22:34.460444+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:02.352403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:22:02.352403+0000","last_clean_scrub_stamp":"2024-03-26T03:22:02.352403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:16.110906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549548+0000","last_change":"2024-03-26T03:22:33.298660+0000","last_active":"2024-03-26T03:22:34.549548+0000","last_peered":"2024-03-26T03:22:34.549548+0000","last_clean":"2024-03-26T03:22:34.549548+0000","last_became_active":"2024-03-26T03:20:57.373433+0000","last_became_peered":"2024-03-26T03:20:57.373433+0000","last_unstale":"2024-03-26T03:22:34.549548+0000","last_undegraded":"2024-03-26T03:22:34.549548+0000","last_fullsized":"2024-03-26T03:22:34.549548+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:33.298546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:33.298546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:52.064969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"82'128","reported_seq":695,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.395973+0000","last_change":"2024-03-26T03:22:42.395996+0000","last_active":"2024-03-26T03:22:42.395973+0000","last_peered":"2024-03-26T03:22:42.395973+0000","last_clean":"2024-03-26T03:22:42.395973+0000","last_became_active":"2024-03-26T03:21:26.464254+0000","last_became_peered":"2024-03-26T03:21:26.464254+0000","last_unstale":"2024-03-26T03:22:42.395973+0000","last_undegraded":"2024-03-26T03:22:42.395973+0000","last_fullsized":"2024-03-26T03:22:42.395973+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:42.395913+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:22:42.395913+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.395913+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.577520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"446'39","reported_seq":773,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:43.611535+0000","last_change":"2024-03-26T03:22:43.611573+0000","last_active":"2024-03-26T03:22:43.611535+0000","last_peered":"2024-03-26T03:22:43.611535+0000","last_clean":"2024-03-26T03:22:43.611535+0000","last_became_active":"2024-03-26T03:20:57.359700+0000","last_became_peered":"2024-03-26T03:20:57.359700+0000","last_unstale":"2024-03-26T03:22:43.611535+0000","last_undegraded":"2024-03-26T03:22:43.611535+0000","last_fullsized":"2024-03-26T03:22:43.611535+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:22:43.611473+0000","last_deep_scrub":"231'31","last_deep_scrub_stamp":"2024-03-26T03:16:56.991031+0000","last_clean_scrub_stamp":"2024-03-26T03:22:43.611473+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:52.597828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":990,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461248+0000","last_change":"2024-03-26T03:21:51.249951+0000","last_active":"2024-03-26T03:22:34.461248+0000","last_peered":"2024-03-26T03:22:34.461248+0000","last_clean":"2024-03-26T03:22:34.461248+0000","last_became_active":"2024-03-26T02:58:56.924324+0000","last_became_peered":"2024-03-26T02:58:56.924324+0000","last_unstale":"2024-03-26T03:22:34.461248+0000","last_undegraded":"2024-03-26T03:22:34.461248+0000","last_fullsized":"2024-03-26T03:22:34.461248+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:51.249857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:51.249857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:55.256531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549153+0000","last_change":"2024-03-26T03:21:26.056227+0000","last_active":"2024-03-26T03:22:34.549153+0000","last_peered":"2024-03-26T03:22:34.549153+0000","last_clean":"2024-03-26T03:22:34.549153+0000","last_became_active":"2024-03-26T03:20:57.377365+0000","last_became_peered":"2024-03-26T03:20:57.377365+0000","last_unstale":"2024-03-26T03:22:34.549153+0000","last_undegraded":"2024-03-26T03:22:34.549153+0000","last_fullsized":"2024-03-26T03:22:34.549153+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:26.056138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:04:40.594788+0000","last_clean_scrub_stamp":"2024-03-26T03:21:26.056138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:51.770560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":653,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468708+0000","last_change":"2024-03-26T03:21:45.434621+0000","last_active":"2024-03-26T03:22:34.468708+0000","last_peered":"2024-03-26T03:22:34.468708+0000","last_clean":"2024-03-26T03:22:34.468708+0000","last_became_active":"2024-03-26T03:20:12.747304+0000","last_became_peered":"2024-03-26T03:20:12.747304+0000","last_unstale":"2024-03-26T03:22:34.468708+0000","last_undegraded":"2024-03-26T03:22:34.468708+0000","last_fullsized":"2024-03-26T03:22:34.468708+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:45.434520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:45.434520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.328405+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"82'128","reported_seq":1001,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553310+0000","last_change":"2024-03-26T03:21:47.914010+0000","last_active":"2024-03-26T03:22:33.553310+0000","last_peered":"2024-03-26T03:22:33.553310+0000","last_clean":"2024-03-26T03:22:33.553310+0000","last_became_active":"2024-03-26T03:20:15.299184+0000","last_became_peered":"2024-03-26T03:20:15.299184+0000","last_unstale":"2024-03-26T03:22:33.553310+0000","last_undegraded":"2024-03-26T03:22:33.553310+0000","last_fullsized":"2024-03-26T03:22:33.553310+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:21:47.913887+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:21:47.913887+0000","last_clean_scrub_stamp":"2024-03-26T03:21:47.913887+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.198817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"446'36","reported_seq":180,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416677+0000","last_change":"2024-03-26T03:22:20.484470+0000","last_active":"2024-03-26T03:22:33.416677+0000","last_peered":"2024-03-26T03:22:33.416677+0000","last_clean":"2024-03-26T03:22:33.416677+0000","last_became_active":"2024-03-26T03:20:57.358076+0000","last_became_peered":"2024-03-26T03:20:57.358076+0000","last_unstale":"2024-03-26T03:22:33.416677+0000","last_undegraded":"2024-03-26T03:22:33.416677+0000","last_fullsized":"2024-03-26T03:22:33.416677+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:22:20.484360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:20.484360+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.974063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553105+0000","last_change":"2024-03-26T03:21:41.909829+0000","last_active":"2024-03-26T03:22:33.553105+0000","last_peered":"2024-03-26T03:22:33.553105+0000","last_clean":"2024-03-26T03:22:33.553105+0000","last_became_active":"2024-03-26T03:20:57.364349+0000","last_became_peered":"2024-03-26T03:20:57.364349+0000","last_unstale":"2024-03-26T03:22:33.553105+0000","last_undegraded":"2024-03-26T03:22:33.553105+0000","last_fullsized":"2024-03-26T03:22:33.553105+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:41.909728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:00:18.957349+0000","last_clean_scrub_stamp":"2024-03-26T03:21:41.909728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:48.917095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"477'12","reported_seq":1332,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.298574+0000","last_change":"2024-03-26T03:22:36.548859+0000","last_active":"2024-03-26T03:22:47.298574+0000","last_peered":"2024-03-26T03:22:47.298574+0000","last_clean":"2024-03-26T03:22:47.298574+0000","last_became_active":"2024-03-26T03:20:57.371266+0000","last_became_peered":"2024-03-26T03:20:57.371266+0000","last_unstale":"2024-03-26T03:22:47.298574+0000","last_undegraded":"2024-03-26T03:22:47.298574+0000","last_fullsized":"2024-03-26T03:22:47.298574+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"477'12","last_scrub_stamp":"2024-03-26T03:22:36.548751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:36.548751+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:49.227425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"288'2","reported_seq":86,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549662+0000","last_change":"2024-03-26T03:21:38.023218+0000","last_active":"2024-03-26T03:22:34.549662+0000","last_peered":"2024-03-26T03:22:34.549662+0000","last_clean":"2024-03-26T03:22:34.549662+0000","last_became_active":"2024-03-26T03:20:57.371714+0000","last_became_peered":"2024-03-26T03:20:57.371714+0000","last_unstale":"2024-03-26T03:22:34.549662+0000","last_undegraded":"2024-03-26T03:22:34.549662+0000","last_fullsized":"2024-03-26T03:22:34.549662+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'2","last_scrub_stamp":"2024-03-26T03:21:38.023081+0000","last_deep_scrub":"288'2","last_deep_scrub_stamp":"2024-03-26T03:12:34.141917+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.023081+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:07.812426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"82'128","reported_seq":979,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409946+0000","last_change":"2024-03-26T03:22:19.243148+0000","last_active":"2024-03-26T03:22:33.409946+0000","last_peered":"2024-03-26T03:22:33.409946+0000","last_clean":"2024-03-26T03:22:33.409946+0000","last_became_active":"2024-03-26T03:21:00.288286+0000","last_became_peered":"2024-03-26T03:21:00.288286+0000","last_unstale":"2024-03-26T03:22:33.409946+0000","last_undegraded":"2024-03-26T03:22:33.409946+0000","last_fullsized":"2024-03-26T03:22:33.409946+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:19.243044+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:22:19.243044+0000","last_clean_scrub_stamp":"2024-03-26T03:22:19.243044+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:39.008270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"507'39","reported_seq":1049,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.461424+0000","last_change":"2024-03-26T03:22:13.421938+0000","last_active":"2024-03-26T03:22:34.461424+0000","last_peered":"2024-03-26T03:22:34.461424+0000","last_clean":"2024-03-26T03:22:34.461424+0000","last_became_active":"2024-03-26T02:58:58.928837+0000","last_became_peered":"2024-03-26T02:58:58.928837+0000","last_unstale":"2024-03-26T03:22:34.461424+0000","last_undegraded":"2024-03-26T03:22:34.461424+0000","last_fullsized":"2024-03-26T03:22:34.461424+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'39","last_scrub_stamp":"2024-03-26T03:22:13.421808+0000","last_deep_scrub":"507'39","last_deep_scrub_stamp":"2024-03-26T03:22:13.421808+0000","last_clean_scrub_stamp":"2024-03-26T03:22:13.421808+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:26.103085+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"509'2","reported_seq":639,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:50.632251+0000","last_change":"2024-03-26T03:22:50.632302+0000","last_active":"2024-03-26T03:22:50.632251+0000","last_peered":"2024-03-26T03:22:50.632251+0000","last_clean":"2024-03-26T03:22:50.632251+0000","last_became_active":"2024-03-26T03:20:12.746319+0000","last_became_peered":"2024-03-26T03:20:12.746319+0000","last_unstale":"2024-03-26T03:22:50.632251+0000","last_undegraded":"2024-03-26T03:22:50.632251+0000","last_fullsized":"2024-03-26T03:22:50.632251+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"509'2","last_scrub_stamp":"2024-03-26T03:22:50.632155+0000","last_deep_scrub":"509'2","last_deep_scrub_stamp":"2024-03-26T03:22:50.632155+0000","last_clean_scrub_stamp":"2024-03-26T03:22:50.632155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:10.399072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":140,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.448388+0000","last_change":"2024-03-26T03:22:37.448411+0000","last_active":"2024-03-26T03:22:37.448388+0000","last_peered":"2024-03-26T03:22:37.448388+0000","last_clean":"2024-03-26T03:22:37.448388+0000","last_became_active":"2024-03-26T03:20:57.355510+0000","last_became_peered":"2024-03-26T03:20:57.355510+0000","last_unstale":"2024-03-26T03:22:37.448388+0000","last_undegraded":"2024-03-26T03:22:37.448388+0000","last_fullsized":"2024-03-26T03:22:37.448388+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:37.448313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:10:20.281044+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.448313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:02.343970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":982,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549390+0000","last_change":"2024-03-26T03:22:31.174605+0000","last_active":"2024-03-26T03:22:34.549390+0000","last_peered":"2024-03-26T03:22:34.549390+0000","last_clean":"2024-03-26T03:22:34.549390+0000","last_became_active":"2024-03-26T03:17:56.570375+0000","last_became_peered":"2024-03-26T03:17:56.570375+0000","last_unstale":"2024-03-26T03:22:34.549390+0000","last_undegraded":"2024-03-26T03:22:34.549390+0000","last_fullsized":"2024-03-26T03:22:34.549390+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:31.174494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:31.174494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:39.360753+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"82'128","reported_seq":883,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468820+0000","last_change":"2024-03-26T03:22:02.691708+0000","last_active":"2024-03-26T03:22:34.468820+0000","last_peered":"2024-03-26T03:22:34.468820+0000","last_clean":"2024-03-26T03:22:34.468820+0000","last_became_active":"2024-03-26T03:20:15.336658+0000","last_became_peered":"2024-03-26T03:20:15.336658+0000","last_unstale":"2024-03-26T03:22:34.468820+0000","last_undegraded":"2024-03-26T03:22:34.468820+0000","last_fullsized":"2024-03-26T03:22:34.468820+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:02.691612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:02.691612+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:12.758244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"446'39","reported_seq":587,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.412675+0000","last_change":"2024-03-26T03:22:25.563778+0000","last_active":"2024-03-26T03:22:33.412675+0000","last_peered":"2024-03-26T03:22:33.412675+0000","last_clean":"2024-03-26T03:22:33.412675+0000","last_became_active":"2024-03-26T03:21:26.470153+0000","last_became_peered":"2024-03-26T03:21:26.470153+0000","last_unstale":"2024-03-26T03:22:33.412675+0000","last_undegraded":"2024-03-26T03:22:33.412675+0000","last_fullsized":"2024-03-26T03:22:33.412675+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":490,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:22:25.563648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:22:25.563648+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:45.508537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:47.530584+0000","last_change":"2024-03-26T03:22:47.530609+0000","last_active":"2024-03-26T03:22:47.530584+0000","last_peered":"2024-03-26T03:22:47.530584+0000","last_clean":"2024-03-26T03:22:47.530584+0000","last_became_active":"2024-03-26T03:20:57.356308+0000","last_became_peered":"2024-03-26T03:20:57.356308+0000","last_unstale":"2024-03-26T03:22:47.530584+0000","last_undegraded":"2024-03-26T03:22:47.530584+0000","last_fullsized":"2024-03-26T03:22:47.530584+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:47.530511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:22:47.530511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:03.775114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":283,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:52.504616+0000","last_change":"2024-03-26T03:22:52.504642+0000","last_active":"2024-03-26T03:22:52.504616+0000","last_peered":"2024-03-26T03:22:52.504616+0000","last_clean":"2024-03-26T03:22:52.504616+0000","last_became_active":"2024-03-26T03:20:12.733918+0000","last_became_peered":"2024-03-26T03:20:12.733918+0000","last_unstale":"2024-03-26T03:22:52.504616+0000","last_undegraded":"2024-03-26T03:22:52.504616+0000","last_fullsized":"2024-03-26T03:22:52.504616+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:52.504539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:09:14.479135+0000","last_clean_scrub_stamp":"2024-03-26T03:22:52.504539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:57.199038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":970,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410718+0000","last_change":"2024-03-26T03:21:56.398490+0000","last_active":"2024-03-26T03:22:33.410718+0000","last_peered":"2024-03-26T03:22:33.410718+0000","last_clean":"2024-03-26T03:22:33.410718+0000","last_became_active":"2024-03-26T02:59:03.206235+0000","last_became_peered":"2024-03-26T02:59:03.206235+0000","last_unstale":"2024-03-26T03:22:33.410718+0000","last_undegraded":"2024-03-26T03:22:33.410718+0000","last_fullsized":"2024-03-26T03:22:33.410718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:56.398396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:56.398396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:16.749380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"399'129","reported_seq":790,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553482+0000","last_change":"2024-03-26T03:21:38.978299+0000","last_active":"2024-03-26T03:22:33.553482+0000","last_peered":"2024-03-26T03:22:33.553482+0000","last_clean":"2024-03-26T03:22:33.553482+0000","last_became_active":"2024-03-26T03:21:00.296006+0000","last_became_peered":"2024-03-26T03:21:00.296006+0000","last_unstale":"2024-03-26T03:22:33.553482+0000","last_undegraded":"2024-03-26T03:22:33.553482+0000","last_fullsized":"2024-03-26T03:22:33.553482+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'129","last_scrub_stamp":"2024-03-26T03:21:38.978193+0000","last_deep_scrub":"399'129","last_deep_scrub_stamp":"2024-03-26T03:16:46.720272+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.978193+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:59.206936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"446'36","reported_seq":728,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468867+0000","last_change":"2024-03-26T03:22:16.553402+0000","last_active":"2024-03-26T03:22:34.468867+0000","last_peered":"2024-03-26T03:22:34.468867+0000","last_clean":"2024-03-26T03:22:34.468867+0000","last_became_active":"2024-03-26T03:14:21.582694+0000","last_became_peered":"2024-03-26T03:14:21.582694+0000","last_unstale":"2024-03-26T03:22:34.468867+0000","last_undegraded":"2024-03-26T03:22:34.468867+0000","last_fullsized":"2024-03-26T03:22:34.468867+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'36","last_scrub_stamp":"2024-03-26T03:22:16.553295+0000","last_deep_scrub":"399'30","last_deep_scrub_stamp":"2024-03-26T03:16:48.957049+0000","last_clean_scrub_stamp":"2024-03-26T03:22:16.553295+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:37.341579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":692,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468824+0000","last_change":"2024-03-26T03:21:47.342963+0000","last_active":"2024-03-26T03:22:34.468824+0000","last_peered":"2024-03-26T03:22:34.468824+0000","last_clean":"2024-03-26T03:22:34.468824+0000","last_became_active":"2024-03-26T03:20:57.353803+0000","last_became_peered":"2024-03-26T03:20:57.353803+0000","last_unstale":"2024-03-26T03:22:34.468824+0000","last_undegraded":"2024-03-26T03:22:34.468824+0000","last_fullsized":"2024-03-26T03:22:34.468824+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:47.342860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:47.342860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:57.521680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":949,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410201+0000","last_change":"2024-03-26T03:22:15.299593+0000","last_active":"2024-03-26T03:22:33.410201+0000","last_peered":"2024-03-26T03:22:33.410201+0000","last_clean":"2024-03-26T03:22:33.410201+0000","last_became_active":"2024-03-26T03:14:21.596996+0000","last_became_peered":"2024-03-26T03:14:21.596996+0000","last_unstale":"2024-03-26T03:22:33.410201+0000","last_undegraded":"2024-03-26T03:22:33.410201+0000","last_fullsized":"2024-03-26T03:22:33.410201+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:15.299461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:15.299461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:22.687673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":642,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469358+0000","last_change":"2024-03-26T03:21:27.728971+0000","last_active":"2024-03-26T03:22:34.469358+0000","last_peered":"2024-03-26T03:22:34.469358+0000","last_clean":"2024-03-26T03:22:34.469358+0000","last_became_active":"2024-03-26T03:14:21.602787+0000","last_became_peered":"2024-03-26T03:14:21.602787+0000","last_unstale":"2024-03-26T03:22:34.469358+0000","last_undegraded":"2024-03-26T03:22:34.469358+0000","last_fullsized":"2024-03-26T03:22:34.469358+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:27.728836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:27.728836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"446'39","reported_seq":233,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409879+0000","last_change":"2024-03-26T03:21:28.438386+0000","last_active":"2024-03-26T03:22:33.409879+0000","last_peered":"2024-03-26T03:22:33.409879+0000","last_clean":"2024-03-26T03:22:33.409879+0000","last_became_active":"2024-03-26T03:20:12.732132+0000","last_became_peered":"2024-03-26T03:20:12.732132+0000","last_unstale":"2024-03-26T03:22:33.409879+0000","last_undegraded":"2024-03-26T03:22:33.409879+0000","last_fullsized":"2024-03-26T03:22:33.409879+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'39","last_scrub_stamp":"2024-03-26T03:21:28.438285+0000","last_deep_scrub":"188'27","last_deep_scrub_stamp":"2024-03-26T03:05:37.637787+0000","last_clean_scrub_stamp":"2024-03-26T03:21:28.438285+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:39.296324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:49.618119+0000","last_change":"2024-03-26T03:22:49.618146+0000","last_active":"2024-03-26T03:22:49.618119+0000","last_peered":"2024-03-26T03:22:49.618119+0000","last_clean":"2024-03-26T03:22:49.618119+0000","last_became_active":"2024-03-26T03:20:57.371143+0000","last_became_peered":"2024-03-26T03:20:57.371143+0000","last_unstale":"2024-03-26T03:22:49.618119+0000","last_undegraded":"2024-03-26T03:22:49.618119+0000","last_fullsized":"2024-03-26T03:22:49.618119+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:49.618040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:35.091531+0000","last_clean_scrub_stamp":"2024-03-26T03:22:49.618040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:13.073766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1041,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:52.072811+0000","last_change":"2024-03-26T03:22:52.072835+0000","last_active":"2024-03-26T03:22:52.072811+0000","last_peered":"2024-03-26T03:22:52.072811+0000","last_clean":"2024-03-26T03:22:52.072811+0000","last_became_active":"2024-03-26T03:20:57.363788+0000","last_became_peered":"2024-03-26T03:20:57.363788+0000","last_unstale":"2024-03-26T03:22:52.072811+0000","last_undegraded":"2024-03-26T03:22:52.072811+0000","last_fullsized":"2024-03-26T03:22:52.072811+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:52.072707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:52.072707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:24:09.662171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.416044+0000","last_change":"2024-03-26T03:21:48.281731+0000","last_active":"2024-03-26T03:22:33.416044+0000","last_peered":"2024-03-26T03:22:33.416044+0000","last_clean":"2024-03-26T03:22:33.416044+0000","last_became_active":"2024-03-26T03:20:57.356519+0000","last_became_peered":"2024-03-26T03:20:57.356519+0000","last_unstale":"2024-03-26T03:22:33.416044+0000","last_undegraded":"2024-03-26T03:22:33.416044+0000","last_fullsized":"2024-03-26T03:22:33.416044+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:48.281609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:48.281609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:58.827911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"82'128","reported_seq":726,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468843+0000","last_change":"2024-03-26T03:22:07.632347+0000","last_active":"2024-03-26T03:22:34.468843+0000","last_peered":"2024-03-26T03:22:34.468843+0000","last_clean":"2024-03-26T03:22:34.468843+0000","last_became_active":"2024-03-26T03:21:00.293971+0000","last_became_peered":"2024-03-26T03:21:00.293971+0000","last_unstale":"2024-03-26T03:22:34.468843+0000","last_undegraded":"2024-03-26T03:22:34.468843+0000","last_fullsized":"2024-03-26T03:22:34.468843+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:07.632196+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:22:07.632196+0000","last_clean_scrub_stamp":"2024-03-26T03:22:07.632196+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:35.898408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"446'15","reported_seq":193,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410070+0000","last_change":"2024-03-26T03:21:21.420396+0000","last_active":"2024-03-26T03:22:33.410070+0000","last_peered":"2024-03-26T03:22:33.410070+0000","last_clean":"2024-03-26T03:22:33.410070+0000","last_became_active":"2024-03-26T03:17:56.574015+0000","last_became_peered":"2024-03-26T03:17:56.574015+0000","last_unstale":"2024-03-26T03:22:33.410070+0000","last_undegraded":"2024-03-26T03:22:33.410070+0000","last_fullsized":"2024-03-26T03:22:33.410070+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'15","last_scrub_stamp":"2024-03-26T03:21:21.420279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:21.420279+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:45.930292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":82,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549630+0000","last_change":"2024-03-26T03:21:47.104938+0000","last_active":"2024-03-26T03:22:34.549630+0000","last_peered":"2024-03-26T03:22:34.549630+0000","last_clean":"2024-03-26T03:22:34.549630+0000","last_became_active":"2024-03-26T03:20:57.366367+0000","last_became_peered":"2024-03-26T03:20:57.366367+0000","last_unstale":"2024-03-26T03:22:34.549630+0000","last_undegraded":"2024-03-26T03:22:34.549630+0000","last_fullsized":"2024-03-26T03:22:34.549630+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:47.104834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:06:26.605507+0000","last_clean_scrub_stamp":"2024-03-26T03:21:47.104834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:00.172590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":90,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:42.481268+0000","last_change":"2024-03-26T03:22:42.481314+0000","last_active":"2024-03-26T03:22:42.481268+0000","last_peered":"2024-03-26T03:22:42.481268+0000","last_clean":"2024-03-26T03:22:42.481268+0000","last_became_active":"2024-03-26T03:20:57.374412+0000","last_became_peered":"2024-03-26T03:20:57.374412+0000","last_unstale":"2024-03-26T03:22:42.481268+0000","last_undegraded":"2024-03-26T03:22:42.481268+0000","last_fullsized":"2024-03-26T03:22:42.481268+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:42.481183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:13.096856+0000","last_clean_scrub_stamp":"2024-03-26T03:22:42.481183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:53.660670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":976,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.553265+0000","last_change":"2024-03-26T03:22:27.111479+0000","last_active":"2024-03-26T03:22:33.553265+0000","last_peered":"2024-03-26T03:22:33.553265+0000","last_clean":"2024-03-26T03:22:33.553265+0000","last_became_active":"2024-03-26T03:20:42.257114+0000","last_became_peered":"2024-03-26T03:20:42.257114+0000","last_unstale":"2024-03-26T03:22:33.553265+0000","last_undegraded":"2024-03-26T03:22:33.553265+0000","last_fullsized":"2024-03-26T03:22:33.553265+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:27.111376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.111376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:39.428944+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1d","version":"82'128","reported_seq":776,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:45.818139+0000","last_change":"2024-03-26T03:22:45.818177+0000","last_active":"2024-03-26T03:22:45.818139+0000","last_peered":"2024-03-26T03:22:45.818139+0000","last_clean":"2024-03-26T03:22:45.818139+0000","last_became_active":"2024-03-26T03:17:59.609812+0000","last_became_peered":"2024-03-26T03:17:59.609812+0000","last_unstale":"2024-03-26T03:22:45.818139+0000","last_undegraded":"2024-03-26T03:22:45.818139+0000","last_fullsized":"2024-03-26T03:22:45.818139+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:45.818062+0000","last_deep_scrub":"82'128","last_deep_scrub_stamp":"2024-03-26T03:06:35.171810+0000","last_clean_scrub_stamp":"2024-03-26T03:22:45.818062+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:48.559771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"508'32","reported_seq":1027,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.409572+0000","last_change":"2024-03-26T03:22:26.183697+0000","last_active":"2024-03-26T03:22:33.409572+0000","last_peered":"2024-03-26T03:22:33.409572+0000","last_clean":"2024-03-26T03:22:33.409572+0000","last_became_active":"2024-03-26T02:58:58.926641+0000","last_became_peered":"2024-03-26T02:58:58.926641+0000","last_unstale":"2024-03-26T03:22:33.409572+0000","last_undegraded":"2024-03-26T03:22:33.409572+0000","last_fullsized":"2024-03-26T03:22:33.409572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'32","last_scrub_stamp":"2024-03-26T03:22:26.183569+0000","last_deep_scrub":"508'32","last_deep_scrub_stamp":"2024-03-26T03:22:26.183569+0000","last_clean_scrub_stamp":"2024-03-26T03:22:26.183569+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:43.860787+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":657,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.469266+0000","last_change":"2024-03-26T03:21:38.346061+0000","last_active":"2024-03-26T03:22:34.469266+0000","last_peered":"2024-03-26T03:22:34.469266+0000","last_clean":"2024-03-26T03:22:34.469266+0000","last_became_active":"2024-03-26T03:20:12.748421+0000","last_became_peered":"2024-03-26T03:20:12.748421+0000","last_unstale":"2024-03-26T03:22:34.469266+0000","last_undegraded":"2024-03-26T03:22:34.469266+0000","last_fullsized":"2024-03-26T03:22:34.469266+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:38.345955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:55.892771+0000","last_clean_scrub_stamp":"2024-03-26T03:21:38.345955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:54.930921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":986,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.548520+0000","last_change":"2024-03-26T03:22:15.066250+0000","last_active":"2024-03-26T03:22:34.548520+0000","last_peered":"2024-03-26T03:22:34.548520+0000","last_clean":"2024-03-26T03:22:34.548520+0000","last_became_active":"2024-03-26T03:20:57.363793+0000","last_became_peered":"2024-03-26T03:20:57.363793+0000","last_unstale":"2024-03-26T03:22:34.548520+0000","last_undegraded":"2024-03-26T03:22:34.548520+0000","last_fullsized":"2024-03-26T03:22:34.548520+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:15.066162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:15.066162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:27.301815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":968,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.410196+0000","last_change":"2024-03-26T03:21:15.448860+0000","last_active":"2024-03-26T03:22:33.410196+0000","last_peered":"2024-03-26T03:22:33.410196+0000","last_clean":"2024-03-26T03:22:33.410196+0000","last_became_active":"2024-03-26T02:59:02.944629+0000","last_became_peered":"2024-03-26T02:59:02.944629+0000","last_unstale":"2024-03-26T03:22:33.410196+0000","last_undegraded":"2024-03-26T03:22:33.410196+0000","last_fullsized":"2024-03-26T03:22:33.410196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:15.448753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:11:46.936200+0000","last_clean_scrub_stamp":"2024-03-26T03:21:15.448753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:39.290211+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"82'128","reported_seq":506,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.549544+0000","last_change":"2024-03-26T03:22:27.330428+0000","last_active":"2024-03-26T03:22:34.549544+0000","last_peered":"2024-03-26T03:22:34.549544+0000","last_clean":"2024-03-26T03:22:34.549544+0000","last_became_active":"2024-03-26T03:21:00.294899+0000","last_became_peered":"2024-03-26T03:21:00.294899+0000","last_unstale":"2024-03-26T03:22:34.549544+0000","last_undegraded":"2024-03-26T03:22:34.549544+0000","last_fullsized":"2024-03-26T03:22:34.549544+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":303,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"82'128","last_scrub_stamp":"2024-03-26T03:22:27.330334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:56:44.163760+0000","last_clean_scrub_stamp":"2024-03-26T03:22:27.330334+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:32.042518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"446'64","reported_seq":787,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468623+0000","last_change":"2024-03-26T03:21:37.375730+0000","last_active":"2024-03-26T03:22:34.468623+0000","last_peered":"2024-03-26T03:22:34.468623+0000","last_clean":"2024-03-26T03:22:34.468623+0000","last_became_active":"2024-03-26T03:20:12.748027+0000","last_became_peered":"2024-03-26T03:20:12.748027+0000","last_unstale":"2024-03-26T03:22:34.468623+0000","last_undegraded":"2024-03-26T03:22:34.468623+0000","last_fullsized":"2024-03-26T03:22:34.468623+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'64","last_scrub_stamp":"2024-03-26T03:21:37.375577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:57.908400+0000","last_clean_scrub_stamp":"2024-03-26T03:21:37.375577+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:40.920423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":132,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:33.417454+0000","last_change":"2024-03-26T03:21:40.351176+0000","last_active":"2024-03-26T03:22:33.417454+0000","last_peered":"2024-03-26T03:22:33.417454+0000","last_clean":"2024-03-26T03:22:33.417454+0000","last_became_active":"2024-03-26T03:20:57.356397+0000","last_became_peered":"2024-03-26T03:20:57.356397+0000","last_unstale":"2024-03-26T03:22:33.417454+0000","last_undegraded":"2024-03-26T03:22:33.417454+0000","last_fullsized":"2024-03-26T03:22:33.417454+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:40.351089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:21:40.351089+0000","last_clean_scrub_stamp":"2024-03-26T03:21:40.351089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:55.930989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":191,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:37.282030+0000","last_change":"2024-03-26T03:22:37.282030+0000","last_active":"2024-03-26T03:22:37.282030+0000","last_peered":"2024-03-26T03:22:37.282030+0000","last_clean":"2024-03-26T03:22:37.282030+0000","last_became_active":"2024-03-26T03:17:56.571882+0000","last_became_peered":"2024-03-26T03:17:56.571882+0000","last_unstale":"2024-03-26T03:22:37.282030+0000","last_undegraded":"2024-03-26T03:22:37.282030+0000","last_fullsized":"2024-03-26T03:22:37.282030+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:22:37.281914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:58:59.914157+0000","last_clean_scrub_stamp":"2024-03-26T03:22:37.281914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:23:51.055497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":648,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-03-26T03:22:34.468565+0000","last_change":"2024-03-26T03:21:33.485707+0000","last_active":"2024-03-26T03:22:34.468565+0000","last_peered":"2024-03-26T03:22:34.468565+0000","last_clean":"2024-03-26T03:22:34.468565+0000","last_became_active":"2024-03-26T03:20:57.354412+0000","last_became_peered":"2024-03-26T03:20:57.354412+0000","last_unstale":"2024-03-26T03:22:34.468565+0000","last_undegraded":"2024-03-26T03:22:34.468565+0000","last_fullsized":"2024-03-26T03:22:34.468565+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:21:33.485593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:59:01.918211+0000","last_clean_scrub_stamp":"2024-03-26T03:21:33.485593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:22:45.359011+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1508416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3969,"ondisk_log_size":3969,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1427,"num_read_kb":1422,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1824,"num_read_kb":1549,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":5962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1271,"ondisk_log_size":1271,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103461,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790320,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92954832,"statfs":{"total":95995035648,"available":95185747968,"internally_reserved":0,"allocated":1155072,"data_stored":719985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":808122826},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430505,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848508,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896644,"statfs":{"total":95995035648,"available":95126163456,"internally_reserved":0,"allocated":1036288,"data_stored":686491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":867826118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575170,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1123268,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92621884,"statfs":{"total":95995035648,"available":94844809216,"internally_reserved":0,"allocated":913408,"data_stored":458389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1149303237},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738695,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820920,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92924232,"statfs":{"total":95995035648,"available":95154413568,"internally_reserved":0,"allocated":1228800,"data_stored":785740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":839383493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411644,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912488,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":92832664,"statfs":{"total":95995035648,"available":95060647936,"internally_reserved":0,"allocated":1146880,"data_stored":720424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":933231045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":331,"seq":1421634175081,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477248,"kb_used_data":1208,"kb_used_omap":4,"kb_used_meta":476027,"kb_avail":93267904,"statfs":{"total":95995035648,"available":95506333696,"internally_reserved":0,"allocated":1236992,"data_stored":784788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":487452323},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986483,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942776,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":92802376,"statfs":{"total":95995035648,"available":95029633024,"internally_reserved":0,"allocated":901120,"data_stored":457206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":964491717},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692143,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015212,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":1014078,"kb_avail":92729940,"statfs":{"total":95995035648,"available":94955458560,"internally_reserved":0,"allocated":1150976,"data_stored":719445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1038416325},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:22:58.261 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T03:22:58.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: 1.b scrub starts 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: 1.b scrub ok 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: 2.17 scrub starts 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: 2.17 scrub ok 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: pgmap v1131: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: 2.18 scrub starts 2024-03-26T03:22:58.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:57 smithi192 ceph-mon[33110]: 2.18 scrub ok 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: 1.b scrub starts 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: 1.b scrub ok 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: 2.17 scrub starts 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: 2.17 scrub ok 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: pgmap v1131: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: 2.18 scrub starts 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[37920]: 2.18 scrub ok 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: 1.b scrub starts 2024-03-26T03:22:58.264 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: 1.b scrub ok 2024-03-26T03:22:58.265 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: 2.17 scrub starts 2024-03-26T03:22:58.265 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: 2.17 scrub ok 2024-03-26T03:22:58.265 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: pgmap v1131: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:22:58.265 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: 2.18 scrub starts 2024-03-26T03:22:58.265 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:57 smithi066 ceph-mon[29666]: 2.18 scrub ok 2024-03-26T03:22:58.538 INFO:teuthology.orchestra.run.smithi066.stderr:Inferring config /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:22:58.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[37920]: from='client.16455 v1:172.21.15.66:0/3324132261' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:22:58.608 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[37920]: 5.0 scrub starts 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[37920]: 5.0 scrub ok 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[37920]: 2.1f scrub starts 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[37920]: 2.1f scrub ok 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[29666]: from='client.16455 v1:172.21.15.66:0/3324132261' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[29666]: 5.0 scrub starts 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[29666]: 5.0 scrub ok 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[29666]: 2.1f scrub starts 2024-03-26T03:22:58.609 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:58 smithi066 ceph-mon[29666]: 2.1f scrub ok 2024-03-26T03:22:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:58 smithi192 ceph-mon[33110]: from='client.16455 v1:172.21.15.66:0/3324132261' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:22:58.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:58 smithi192 ceph-mon[33110]: 5.0 scrub starts 2024-03-26T03:22:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:58 smithi192 ceph-mon[33110]: 5.0 scrub ok 2024-03-26T03:22:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:58 smithi192 ceph-mon[33110]: 2.1f scrub starts 2024-03-26T03:22:58.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:58 smithi192 ceph-mon[33110]: 2.1f scrub ok 2024-03-26T03:22:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: 6.1 scrub starts 2024-03-26T03:22:59.858 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: 6.1 scrub ok 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: 6.1a scrub starts 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: 6.1a scrub ok 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: pgmap v1132: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: 6.1f scrub starts 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[29666]: 6.1f scrub ok 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: 6.1 scrub starts 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: 6.1 scrub ok 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: 6.1a scrub starts 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: 6.1a scrub ok 2024-03-26T03:22:59.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: pgmap v1132: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:59.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: 6.1f scrub starts 2024-03-26T03:22:59.860 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:22:59 smithi066 ceph-mon[37920]: 6.1f scrub ok 2024-03-26T03:22:59.987 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: 6.1 scrub starts 2024-03-26T03:22:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: 6.1 scrub ok 2024-03-26T03:22:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: 6.1a scrub starts 2024-03-26T03:22:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: 6.1a scrub ok 2024-03-26T03:22:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: pgmap v1132: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:22:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: 6.1f scrub starts 2024-03-26T03:22:59.988 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:22:59 smithi192 ceph-mon[33110]: 6.1f scrub ok 2024-03-26T03:23:00.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug there is no tcmu-runner data available 2024-03-26T03:23:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-mon[33110]: 4.7 scrub starts 2024-03-26T03:23:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-mon[33110]: 4.7 scrub ok 2024-03-26T03:23:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-mon[33110]: 4.8 scrub starts 2024-03-26T03:23:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-mon[33110]: 4.8 scrub ok 2024-03-26T03:23:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-mon[33110]: from='client.? v1:172.21.15.66:0/3276336235' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T03:23:00.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:00 smithi192 ceph-mon[33110]: from='client.36308 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T03:23:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[37920]: 4.7 scrub starts 2024-03-26T03:23:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[37920]: 4.7 scrub ok 2024-03-26T03:23:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[37920]: 4.8 scrub starts 2024-03-26T03:23:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[37920]: 4.8 scrub ok 2024-03-26T03:23:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[37920]: from='client.? v1:172.21.15.66:0/3276336235' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T03:23:00.859 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[37920]: from='client.36308 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[29666]: 4.7 scrub starts 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[29666]: 4.7 scrub ok 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[29666]: 4.8 scrub starts 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[29666]: 4.8 scrub ok 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[29666]: from='client.? v1:172.21.15.66:0/3276336235' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-mon[29666]: from='client.36308 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:00] ENGINE Bus STOPPING 2024-03-26T03:23:00.860 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:23:00.861 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:00] ENGINE Bus STOPPED 2024-03-26T03:23:00.861 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:00] ENGINE Bus STARTING 2024-03-26T03:23:00.861 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:00] ENGINE Serving on http://:::9283 2024-03-26T03:23:00.861 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:00 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:00] ENGINE Bus STARTED 2024-03-26T03:23:01.164 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T03:23:01.175 INFO:tasks.cephadm:Teardown begin 2024-03-26T03:23:01.176 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:23:01.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:23:01.319 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T03:23:01.319 DEBUG:teuthology.orchestra.run.smithi066:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T03:23:01.362 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T03:23:01.387 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T03:23:01.388 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T03:23:01.388 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 systemd[1]: Stopping Ceph mon.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: 4.1a scrub starts 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: 4.1a scrub ok 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: pgmap v1133: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: from='client.36308 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:23:01.782 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[29666]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: 4.1a scrub starts 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: 4.1a scrub ok 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: pgmap v1133: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: from='client.36308 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:23:01.783 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-mon[37920]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:23:01.989 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: 4.1a scrub starts 2024-03-26T03:23:01.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: 4.1a scrub ok 2024-03-26T03:23:01.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: from='client.24472 v1:172.21.15.192:0/3736020072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:23:01.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: pgmap v1133: 162 pgs: 162 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:23:01.990 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: from='client.36308 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T03:23:01.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:23:01.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:23:01.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:23:01.991 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:01 smithi192 ceph-mon[33110]: from='mgr.34481 v1:172.21.15.66:0/1166815880' entity='mgr.y' 2024-03-26T03:23:02.108 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29662]: 2024-03-26T03:23:01.780+0000 7f94ac9e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a[29662]: 2024-03-26T03:23:01.780+0000 7f94ac9e7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:01 smithi066 podman[203323]: 2024-03-26 03:23:01.832007626 +0000 UTC m=+0.132982056 container died 858570b43b8513babaeb250bfa0924b4121f20ba00b9b7794b6ae7bac41c2ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:01] ENGINE Bus STOPPING 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:01] ENGINE Bus STOPPED 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:01] ENGINE Bus STARTING 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:01] ENGINE Serving on http://:::9283 2024-03-26T03:23:02.109 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:01 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: [26/Mar/2024:03:23:01] ENGINE Bus STARTED 2024-03-26T03:23:02.575 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:02 smithi066 podman[203323]: 2024-03-26 03:23:02.298203143 +0000 UTC m=+0.599177570 container cleanup 858570b43b8513babaeb250bfa0924b4121f20ba00b9b7794b6ae7bac41c2ccf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T03:23:02.575 INFO:journalctl@ceph.mon.a.smithi066.stdout:Mar 26 03:23:02 smithi066 bash[203323]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-a 2024-03-26T03:23:02.719 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T03:23:02.780 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:02.780 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T03:23:02.780 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T03:23:02.780 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.c 2024-03-26T03:23:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:02 smithi066 systemd[1]: Stopping Ceph mon.c for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c[37914]: 2024-03-26T03:23:03.069+0000 7f2696229700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:03.109 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c[37914]: 2024-03-26T03:23:03.069+0000 7f2696229700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T03:23:03.462 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:03 smithi066 podman[203555]: 2024-03-26 03:23:03.171880843 +0000 UTC m=+0.201441320 container died 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:23:03.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:03 smithi066 podman[203555]: 2024-03-26 03:23:03.644944875 +0000 UTC m=+0.674505360 container cleanup 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:23:03.832 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:03 smithi066 bash[203555]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c 2024-03-26T03:23:04.069 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T03:23:04.097 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:03 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: ::ffff:172.21.15.192 - - [26/Mar/2024:03:23:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:23:04.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:03 smithi066 podman[203570]: 2024-03-26 03:23:03.930049377 +0000 UTC m=+0.747471104 container remove 323935c049ed4587f717533193c8fbf14ba6c9d8bffa14a30ccfc5404525421e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-c, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T03:23:04.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:04 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-26T03:23:04.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:04 smithi066 systemd[1]: Stopped Ceph mon.c for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:23:04.097 INFO:journalctl@ceph.mon.c.smithi066.stdout:Mar 26 03:23:04 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.c.service: Consumed 22.124s CPU time. 2024-03-26T03:23:04.110 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:04.110 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T03:23:04.110 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T03:23:04.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.b 2024-03-26T03:23:04.455 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:04 smithi192 systemd[1]: Stopping Ceph mon.b for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:04.737 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:04 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-b[33084]: 2024-03-26T03:23:04.453+0000 7faaf3b27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:04 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-b[33084]: 2024-03-26T03:23:04.453+0000 7faaf3b27700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T03:23:04.738 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:04 smithi192 podman[158025]: 2024-03-26 03:23:04.689354753 +0000 UTC m=+0.333638520 container died 61c0eb3397e33e3e2373581017cf854eb5a6343cf348be0737d9f8a864809110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T03:23:05.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:05 smithi192 podman[158025]: 2024-03-26 03:23:05.196382865 +0000 UTC m=+0.840666685 container cleanup 61c0eb3397e33e3e2373581017cf854eb5a6343cf348be0737d9f8a864809110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-b, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T03:23:05.440 INFO:journalctl@ceph.mon.b.smithi192.stdout:Mar 26 03:23:05 smithi192 bash[158025]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mon-b 2024-03-26T03:23:05.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T03:23:05.613 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:05.613 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T03:23:05.613 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T03:23:05.614 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y 2024-03-26T03:23:05.925 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:05 smithi066 systemd[1]: Stopping Ceph mgr.y for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:06.293 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:05 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T03:23:05.923+0000 7fa3223df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:06.293 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:05 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y[29897]: 2024-03-26T03:23:05.923+0000 7fa3223df700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T03:23:06.293 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:05 smithi066 podman[203676]: 2024-03-26 03:23:05.955771388 +0000 UTC m=+0.141488759 container died 7a4b80071d2cf72e1065c09ee5f269612374cd3f1577929825c377306d416d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T03:23:06.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:06 smithi066 podman[203676]: 2024-03-26 03:23:06.458669707 +0000 UTC m=+0.644387066 container cleanup 7a4b80071d2cf72e1065c09ee5f269612374cd3f1577929825c377306d416d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T03:23:06.608 INFO:journalctl@ceph.mgr.y.smithi066.stdout:Mar 26 03:23:06 smithi066 bash[203676]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-y 2024-03-26T03:23:06.850 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T03:23:06.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:06.893 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T03:23:06.893 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T03:23:06.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.x 2024-03-26T03:23:07.216 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 03:23:06 smithi192 systemd[1]: Stopping Ceph mgr.x for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:07.477 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 03:23:07 smithi192 podman[158147]: 2024-03-26 03:23:07.215506531 +0000 UTC m=+0.130636379 container died 43fd2080f308581c77d9aadc559189df18b207b512fb9d61f4b2667a11db4a4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:23:07.737 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 03:23:07 smithi192 podman[158147]: 2024-03-26 03:23:07.643361077 +0000 UTC m=+0.558490943 container cleanup 43fd2080f308581c77d9aadc559189df18b207b512fb9d61f4b2667a11db4a4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:23:07.738 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Mar 26 03:23:07 smithi192 bash[158147]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-mgr-x 2024-03-26T03:23:08.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T03:23:08.139 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:08.139 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T03:23:08.139 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T03:23:08.139 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.0 2024-03-26T03:23:08.481 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:08 smithi066 systemd[1]: Stopping Ceph osd.0 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:08.858 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:08 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0[41834]: 2024-03-26T03:23:08.478+0000 7fa0c8474700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:08.859 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:08 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0[41834]: 2024-03-26T03:23:08.478+0000 7fa0c8474700 -1 osd.0 509 *** Got signal Terminated *** 2024-03-26T03:23:08.859 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:08 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0[41834]: 2024-03-26T03:23:08.478+0000 7fa0c8474700 -1 osd.0 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:14.374 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:14 smithi066 podman[203795]: 2024-03-26 03:23:14.108326031 +0000 UTC m=+5.769220709 container died 067a374a0958702d9f9d0aaaa992c1da4f0d272bf08342b2bb7a0ddc0eca6391 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T03:23:14.858 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:14 smithi066 podman[203795]: 2024-03-26 03:23:14.572277157 +0000 UTC m=+6.233171834 container cleanup 067a374a0958702d9f9d0aaaa992c1da4f0d272bf08342b2bb7a0ddc0eca6391 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2) 2024-03-26T03:23:14.858 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:14 smithi066 bash[203795]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0 2024-03-26T03:23:15.190 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:14 smithi066 podman[203812]: 2024-03-26 03:23:14.890605634 +0000 UTC m=+0.780093071 container remove 067a374a0958702d9f9d0aaaa992c1da4f0d272bf08342b2bb7a0ddc0eca6391 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:23:15.608 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:15 smithi066 podman[203904]: 2024-03-26 03:23:15.093522756 +0000 UTC m=+0.018557632 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:23:15.608 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:15 smithi066 podman[203904]: 2024-03-26 03:23:15.213941814 +0000 UTC m=+0.138976675 container create e85f183c5d2daf195b96b4d657dfd9addffeb2942c0f4562a621ded93a2c8581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T03:23:15.608 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:15 smithi066 podman[203904]: 2024-03-26 03:23:15.436125043 +0000 UTC m=+0.361159911 container init e85f183c5d2daf195b96b4d657dfd9addffeb2942c0f4562a621ded93a2c8581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T03:23:15.609 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:15 smithi066 podman[203904]: 2024-03-26 03:23:15.439943333 +0000 UTC m=+0.364978199 container start e85f183c5d2daf195b96b4d657dfd9addffeb2942c0f4562a621ded93a2c8581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:23:16.108 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:15 smithi066 podman[203904]: 2024-03-26 03:23:15.610735244 +0000 UTC m=+0.535770120 container attach e85f183c5d2daf195b96b4d657dfd9addffeb2942c0f4562a621ded93a2c8581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T03:23:16.831 INFO:journalctl@ceph.osd.0.smithi066.stdout:Mar 26 03:23:16 smithi066 podman[203904]: 2024-03-26 03:23:16.493354395 +0000 UTC m=+1.418389250 container died e85f183c5d2daf195b96b4d657dfd9addffeb2942c0f4562a621ded93a2c8581 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T03:23:17.110 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T03:23:17.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:17.158 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T03:23:17.158 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T03:23:17.158 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.1 2024-03-26T03:23:17.358 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:17 smithi066 systemd[1]: Stopping Ceph osd.1 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:17.859 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:17 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1[47622]: 2024-03-26T03:23:17.458+0000 7fd86ae49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:17.859 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:17 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1[47622]: 2024-03-26T03:23:17.458+0000 7fd86ae49700 -1 osd.1 509 *** Got signal Terminated *** 2024-03-26T03:23:17.859 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:17 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1[47622]: 2024-03-26T03:23:17.458+0000 7fd86ae49700 -1 osd.1 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:23.219 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:22 smithi066 podman[204010]: 2024-03-26 03:23:22.924073614 +0000 UTC m=+5.568791572 container died cfa80a88bc89a4e70eecdc1496d52ed9cc78f7c03165f9720e507ab8177867bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T03:23:23.536 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:23 smithi066 podman[204010]: 2024-03-26 03:23:23.30090998 +0000 UTC m=+5.945627940 container cleanup cfa80a88bc89a4e70eecdc1496d52ed9cc78f7c03165f9720e507ab8177867bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T03:23:23.536 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:23 smithi066 bash[204010]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1 2024-03-26T03:23:23.836 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:23 smithi066 podman[204030]: 2024-03-26 03:23:23.535456893 +0000 UTC m=+0.598719911 container remove cfa80a88bc89a4e70eecdc1496d52ed9cc78f7c03165f9720e507ab8177867bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T03:23:24.108 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:23 smithi066 podman[204120]: 2024-03-26 03:23:23.836468528 +0000 UTC m=+0.103428957 container create 590d5cd1abb3c73278f32c699e483a05a91f243320a45ba71a4142f0c8e98031 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T03:23:24.109 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:23 smithi066 podman[204120]: 2024-03-26 03:23:23.7514786 +0000 UTC m=+0.018439062 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:23:24.109 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:24 smithi066 podman[204120]: 2024-03-26 03:23:24.038915174 +0000 UTC m=+0.305875614 container init 590d5cd1abb3c73278f32c699e483a05a91f243320a45ba71a4142f0c8e98031 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T03:23:24.109 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:24 smithi066 podman[204120]: 2024-03-26 03:23:24.04202836 +0000 UTC m=+0.308988795 container start 590d5cd1abb3c73278f32c699e483a05a91f243320a45ba71a4142f0c8e98031 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:23:24.609 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:24 smithi066 podman[204120]: 2024-03-26 03:23:24.113619934 +0000 UTC m=+0.380580367 container attach 590d5cd1abb3c73278f32c699e483a05a91f243320a45ba71a4142f0c8e98031 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T03:23:24.609 INFO:journalctl@ceph.osd.1.smithi066.stdout:Mar 26 03:23:24 smithi066 podman[204120]: 2024-03-26 03:23:24.354494756 +0000 UTC m=+0.621455201 container died 590d5cd1abb3c73278f32c699e483a05a91f243320a45ba71a4142f0c8e98031 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T03:23:25.023 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T03:23:25.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:25.072 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T03:23:25.072 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T03:23:25.072 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.2 2024-03-26T03:23:25.358 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:25 smithi066 systemd[1]: Stopping Ceph osd.2 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:25.858 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:25 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2[53675]: 2024-03-26T03:23:25.381+0000 7f3d74b5f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:25.859 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:25 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2[53675]: 2024-03-26T03:23:25.381+0000 7f3d74b5f700 -1 osd.2 509 *** Got signal Terminated *** 2024-03-26T03:23:25.859 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:25 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2[53675]: 2024-03-26T03:23:25.381+0000 7f3d74b5f700 -1 osd.2 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:31.155 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:30 smithi066 podman[204225]: 2024-03-26 03:23:30.883587156 +0000 UTC m=+5.616525236 container died b36bbb27fe9b8a53d13f751a315eb2db9885740fd9be4144c7444c01e14b505a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T03:23:31.530 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:31 smithi066 podman[204225]: 2024-03-26 03:23:31.328714207 +0000 UTC m=+6.061652269 container cleanup b36bbb27fe9b8a53d13f751a315eb2db9885740fd9be4144c7444c01e14b505a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T03:23:31.530 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:31 smithi066 bash[204225]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2 2024-03-26T03:23:31.858 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:31 smithi066 podman[204241]: 2024-03-26 03:23:31.530151099 +0000 UTC m=+0.643972778 container remove b36bbb27fe9b8a53d13f751a315eb2db9885740fd9be4144c7444c01e14b505a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T03:23:32.275 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:31 smithi066 podman[204331]: 2024-03-26 03:23:31.853781673 +0000 UTC m=+0.018658385 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:23:32.275 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:31 smithi066 podman[204331]: 2024-03-26 03:23:31.956051415 +0000 UTC m=+0.120928170 container create 3b668366ddbb1cba40e745ce546829f57a1196e63aae5d8319ebf29ffa5ccdf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T03:23:32.275 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:32 smithi066 podman[204331]: 2024-03-26 03:23:32.174987922 +0000 UTC m=+0.339864643 container init 3b668366ddbb1cba40e745ce546829f57a1196e63aae5d8319ebf29ffa5ccdf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T03:23:32.275 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:32 smithi066 podman[204331]: 2024-03-26 03:23:32.17809904 +0000 UTC m=+0.342975753 container start 3b668366ddbb1cba40e745ce546829f57a1196e63aae5d8319ebf29ffa5ccdf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T03:23:32.275 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:32 smithi066 podman[204331]: 2024-03-26 03:23:32.274900857 +0000 UTC m=+0.439777586 container attach 3b668366ddbb1cba40e745ce546829f57a1196e63aae5d8319ebf29ffa5ccdf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T03:23:32.608 INFO:journalctl@ceph.osd.2.smithi066.stdout:Mar 26 03:23:32 smithi066 podman[204331]: 2024-03-26 03:23:32.478186333 +0000 UTC m=+0.643063051 container died 3b668366ddbb1cba40e745ce546829f57a1196e63aae5d8319ebf29ffa5ccdf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:23:33.097 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T03:23:33.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:33.144 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T03:23:33.144 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T03:23:33.144 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3 2024-03-26T03:23:33.358 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:33 smithi066 systemd[1]: Stopping Ceph osd.3 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:33.858 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:23:33.473+0000 7f24ea29a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:33.859 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:23:33.473+0000 7f24ea29a700 -1 osd.3 509 *** Got signal Terminated *** 2024-03-26T03:23:33.859 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:33 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:23:33.473+0000 7f24ea29a700 -1 osd.3 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:34.738 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:34 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:34.374+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:35.487 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:35 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:35.402+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:35.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:35 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:35.196+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:35.858 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:35 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:23:35.557+0000 7f24e688e700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.772438+0000 front 2024-03-26T03:23:11.772511+0000 (oldest deadline 2024-03-26T03:23:34.672469+0000) 2024-03-26T03:23:35.990 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:35 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:35.651+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:36.487 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:36 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:36.372+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:36.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:36 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:36.208+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:36.858 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:36 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:23:36.600+0000 7f24e688e700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.772438+0000 front 2024-03-26T03:23:11.772511+0000 (oldest deadline 2024-03-26T03:23:34.672469+0000) 2024-03-26T03:23:36.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:36 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:36.695+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:37.487 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:37.353+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:37.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:37.214+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:37.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:37 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:37.659+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:38.108 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:37 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3[156687]: 2024-03-26T03:23:37.618+0000 7f24e688e700 -1 osd.3 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.772438+0000 front 2024-03-26T03:23:11.772511+0000 (oldest deadline 2024-03-26T03:23:34.672469+0000) 2024-03-26T03:23:38.384 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:38 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:38.180+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:38.384 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:38 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:38.115+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:38.656 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:38 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:38.383+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:38.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:38 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:38.654+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:39.174 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:38 smithi066 podman[204437]: 2024-03-26 03:23:38.903490538 +0000 UTC m=+5.568584273 container died 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-26T03:23:39.487 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:39.334+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:39.488 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:39.135+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:39.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:39.135+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:39.575 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:39 smithi066 podman[204437]: 2024-03-26 03:23:39.348384925 +0000 UTC m=+6.013478665 container cleanup 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T03:23:39.576 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:39 smithi066 bash[204437]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3 2024-03-26T03:23:39.858 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:39 smithi066 podman[204454]: 2024-03-26 03:23:39.574684044 +0000 UTC m=+0.657177677 container remove 1e60122d09d4989359147b6bbc3cdacbec0802ca7ec0298ae0850242a2b56c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T03:23:39.994 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:39 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:39.614+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:40.287 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:39 smithi066 podman[204546]: 2024-03-26 03:23:39.992292887 +0000 UTC m=+0.113330011 container create 8cc0c0da4ba6e6fa51e0feb0cbb389c8ac9988400c11d33343da06c7d551a118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T03:23:40.287 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:39 smithi066 podman[204546]: 2024-03-26 03:23:39.897189568 +0000 UTC m=+0.018226678 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:23:40.287 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:40 smithi066 podman[204546]: 2024-03-26 03:23:40.211472385 +0000 UTC m=+0.332509474 container init 8cc0c0da4ba6e6fa51e0feb0cbb389c8ac9988400c11d33343da06c7d551a118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T03:23:40.287 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:40 smithi066 podman[204546]: 2024-03-26 03:23:40.21476509 +0000 UTC m=+0.335802185 container start 8cc0c0da4ba6e6fa51e0feb0cbb389c8ac9988400c11d33343da06c7d551a118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T03:23:40.287 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:40 smithi066 podman[204546]: 2024-03-26 03:23:40.2865669 +0000 UTC m=+0.407604000 container attach 8cc0c0da4ba6e6fa51e0feb0cbb389c8ac9988400c11d33343da06c7d551a118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:23:40.487 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:40.332+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:40.488 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:40.140+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:40.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:40.108+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:40.608 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:40 smithi066 podman[204546]: 2024-03-26 03:23:40.523783705 +0000 UTC m=+0.644820800 container died 8cc0c0da4ba6e6fa51e0feb0cbb389c8ac9988400c11d33343da06c7d551a118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T03:23:40.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:40 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:40.638+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:41.074 INFO:journalctl@ceph.osd.3.smithi066.stdout:Mar 26 03:23:41 smithi066 podman[204572]: 2024-03-26 03:23:41.056257938 +0000 UTC m=+0.523755488 container remove 8cc0c0da4ba6e6fa51e0feb0cbb389c8ac9988400c11d33343da06c7d551a118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:23:41.079 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T03:23:41.124 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:41.124 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T03:23:41.124 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T03:23:41.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.4 2024-03-26T03:23:41.421 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:41.157+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:41.421 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:41.127+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:41.422 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:41 smithi192 systemd[1]: Stopping Ceph osd.4 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:41.422 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:41.317+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:41.737 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:41.593+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:41.737 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:41.420+0000 7fcd944bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:41.738 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:41.420+0000 7fcd944bb700 -1 osd.4 509 *** Got signal Terminated *** 2024-03-26T03:23:41.738 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:41 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:41.420+0000 7fcd944bb700 -1 osd.4 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:42.361 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:42 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:42.108+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:42.361 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:42 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:42.130+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:42.737 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:42 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:42.360+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:42.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:42 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:42.607+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:43.380 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:43.113+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:43.380 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:43.142+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:43.737 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:43.378+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:43.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:43 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:43.625+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:44.425 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:44.099+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:44.426 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:44.145+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:44.426 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:44.145+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:44.737 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:44.424+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:44.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:44.632+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:44.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:44 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:44.632+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:45.438 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:45.095+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:45.439 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:45.190+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:45.439 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:45.190+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:45.737 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:45.650+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:45.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:45.650+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:45.738 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:45.437+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:10.775147+0000 front 2024-03-26T03:23:10.775129+0000 (oldest deadline 2024-03-26T03:23:34.274927+0000) 2024-03-26T03:23:45.738 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:45 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4[38021]: 2024-03-26T03:23:45.437+0000 7fcd8caa7700 -1 osd.4 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:21.776860+0000 front 2024-03-26T03:23:21.776847+0000 (oldest deadline 2024-03-26T03:23:45.276895+0000) 2024-03-26T03:23:46.391 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:46.091+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:46.391 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:46.206+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:46.391 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:46.206+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:46.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:46.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:46.660 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:46.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:46.660 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:23:46.391Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-26T03:23:46.660 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:23:46.392Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-26T03:23:46.660 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:23:46.394Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-26T03:23:46.660 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:23:46.398Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-26T03:23:46.660 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:23:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:23:46.398Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.66:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.66:8765: connect: connection refused" 2024-03-26T03:23:47.206 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:46 smithi192 podman[158282]: 2024-03-26 03:23:46.926194377 +0000 UTC m=+5.595195439 container died f3861a519119ffe1db662f125c9d114b63645136ade9d6f5e488d25937917656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T03:23:47.206 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:47 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:47.111+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:47.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:47 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:47.230+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:47.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:47 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:47.230+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:47.488 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:47 smithi192 podman[158282]: 2024-03-26 03:23:47.404896731 +0000 UTC m=+6.073897781 container cleanup f3861a519119ffe1db662f125c9d114b63645136ade9d6f5e488d25937917656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T03:23:47.488 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:47 smithi192 bash[158282]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4 2024-03-26T03:23:47.987 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:47 smithi192 podman[158308]: 2024-03-26 03:23:47.706508464 +0000 UTC m=+0.765256517 container remove f3861a519119ffe1db662f125c9d114b63645136ade9d6f5e488d25937917656 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T03:23:47.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:47 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:47.591+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:47.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:47 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:47.591+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:48.271 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:48.127+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:48.272 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:48.127+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:48.272 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:48.242+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:48.272 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:48.242+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:48.272 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:48 smithi192 podman[158405]: 2024-03-26 03:23:47.922606577 +0000 UTC m=+0.018315320 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:23:48.272 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:48 smithi192 podman[158405]: 2024-03-26 03:23:48.040949825 +0000 UTC m=+0.136658541 container create 0da89878742df6231648030328e7a134b8aff36ab8a960dffe002e81080a0488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T03:23:48.272 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:48 smithi192 podman[158405]: 2024-03-26 03:23:48.268389434 +0000 UTC m=+0.364098149 container init 0da89878742df6231648030328e7a134b8aff36ab8a960dffe002e81080a0488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T03:23:48.273 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:48 smithi192 podman[158405]: 2024-03-26 03:23:48.271600043 +0000 UTC m=+0.367308759 container start 0da89878742df6231648030328e7a134b8aff36ab8a960dffe002e81080a0488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:23:48.581 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:48 smithi192 podman[158405]: 2024-03-26 03:23:48.402100114 +0000 UTC m=+0.497808838 container attach 0da89878742df6231648030328e7a134b8aff36ab8a960dffe002e81080a0488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T03:23:48.867 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:48.580+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:48.867 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:48 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:48.580+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:49.172 INFO:journalctl@ceph.osd.4.smithi192.stdout:Mar 26 03:23:48 smithi192 podman[158405]: 2024-03-26 03:23:48.867184543 +0000 UTC m=+0.962893271 container died 0da89878742df6231648030328e7a134b8aff36ab8a960dffe002e81080a0488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-26T03:23:49.173 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:49.096+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:49.173 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:49.096+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:49.465 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:49.241+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:49.465 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:49.241+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:49.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T03:23:49.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:49.536 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T03:23:49.536 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T03:23:49.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.5 2024-03-26T03:23:49.737 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:49 smithi192 systemd[1]: Stopping Ceph osd.5 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:49.737 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:49.572+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:49.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:49.572+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:50.237 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:49.817+0000 7f2ddfba1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:50.238 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:49.817+0000 7f2ddfba1700 -1 osd.5 509 *** Got signal Terminated *** 2024-03-26T03:23:50.238 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:49 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:49.817+0000 7f2ddfba1700 -1 osd.5 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:50.238 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:50.063+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:50.238 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:50.063+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:50.557 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:50.238+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:50.557 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:50.238+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:50.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:50.556+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:50.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:50 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:50.556+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:51.487 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:51.019+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:51.488 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:51.019+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:51.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:51.197+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:51.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:51.197+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:51.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:51.583+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:51.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:51.583+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:52.246 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:51.989+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:52.246 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:51 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:51.989+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:52.624 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:52 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:52.245+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:52.624 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:52 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:52.245+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:52.958 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:52 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:52.623+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:52.958 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:52 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:52.623+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:53.237 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:52 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:52.957+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:53.237 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:52 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:52.957+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:53.581 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:53 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:53.250+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:53.581 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:53 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:53.250+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:53.969 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:53 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:53.580+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:53.969 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:53 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:53.580+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:54.237 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:53 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:53.967+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:11.758824+0000 front 2024-03-26T03:23:11.758874+0000 (oldest deadline 2024-03-26T03:23:37.558912+0000) 2024-03-26T03:23:54.237 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:53 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5[44040]: 2024-03-26T03:23:53.967+0000 7f2dd818d700 -1 osd.5 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:22.259735+0000 front 2024-03-26T03:23:22.259875+0000 (oldest deadline 2024-03-26T03:23:47.559771+0000) 2024-03-26T03:23:54.238 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:54.209+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:54.238 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:54.209+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:54.238 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:54.209+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:23:54.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:54.613+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:54.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:54 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:54.613+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:55.487 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:55 smithi192 podman[158513]: 2024-03-26 03:23:55.283548044 +0000 UTC m=+5.553858477 container died 962680fb9c11629c6525c2f3ca687fc1ef8b02f27356a2e8963169c963705519 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T03:23:55.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:55 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:55.181+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:55.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:55 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:55.181+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:55.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:55 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:55.181+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:23:55.819 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:55 smithi192 podman[158513]: 2024-03-26 03:23:55.617619615 +0000 UTC m=+5.887930088 container cleanup 962680fb9c11629c6525c2f3ca687fc1ef8b02f27356a2e8963169c963705519 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:23:55.819 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:55 smithi192 bash[158513]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5 2024-03-26T03:23:55.819 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:55 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:55.587+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:55.819 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:55 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:55.587+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:55.819 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:55 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:55.587+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:23:56.160 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:55 smithi192 podman[158531]: 2024-03-26 03:23:55.81871551 +0000 UTC m=+0.535449481 container remove 962680fb9c11629c6525c2f3ca687fc1ef8b02f27356a2e8963169c963705519 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T03:23:56.441 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:56 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:56.159+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:56.441 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:56 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:56.159+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:56.441 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:56 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:56.159+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:23:56.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:56 smithi192 podman[158626]: 2024-03-26 03:23:56.134305393 +0000 UTC m=+0.017507174 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:23:56.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:56 smithi192 podman[158626]: 2024-03-26 03:23:56.23680177 +0000 UTC m=+0.120003548 container create 660518cf5cb22796c42f277c5c7a67003ebb376fe3635a5ec0f71868b15a6dee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T03:23:56.442 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:56 smithi192 podman[158626]: 2024-03-26 03:23:56.439140663 +0000 UTC m=+0.322342441 container init 660518cf5cb22796c42f277c5c7a67003ebb376fe3635a5ec0f71868b15a6dee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T03:23:56.731 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:56 smithi192 podman[158626]: 2024-03-26 03:23:56.442336411 +0000 UTC m=+0.325538188 container start 660518cf5cb22796c42f277c5c7a67003ebb376fe3635a5ec0f71868b15a6dee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:23:56.731 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:56 smithi192 podman[158626]: 2024-03-26 03:23:56.5225778 +0000 UTC m=+0.405779595 container attach 660518cf5cb22796c42f277c5c7a67003ebb376fe3635a5ec0f71868b15a6dee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T03:23:56.731 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:56 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:56.620+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:56.731 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:56 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:56.620+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:56.731 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:56 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:56.620+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:23:56.987 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 26 03:23:56 smithi192 podman[158626]: 2024-03-26 03:23:56.730746703 +0000 UTC m=+0.613948487 container died 660518cf5cb22796c42f277c5c7a67003ebb376fe3635a5ec0f71868b15a6dee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD) 2024-03-26T03:23:57.310 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:57.201+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:57.310 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:57.201+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:57.310 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:57.201+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:23:57.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T03:23:57.375 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:23:57.375 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T03:23:57.375 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T03:23:57.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.6 2024-03-26T03:23:57.637 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 systemd[1]: Stopping Ceph osd.6 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:23:57.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:57.636+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:57.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:57.636+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:57.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:57.636+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:23:57.988 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:57.695+0000 7f2f372d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:23:57.988 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:57.695+0000 7f2f372d1700 -1 osd.6 509 *** Got signal Terminated *** 2024-03-26T03:23:57.988 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:57 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:57.695+0000 7f2f372d1700 -1 osd.6 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:23:58.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:58.173+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:58.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:58.173+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:58.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:58.173+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:23:58.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:58.665+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:58.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:58.665+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:58.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:58 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:58.665+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:23:59.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:59.196+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:23:59.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:59.196+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:23:59.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:23:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:23:59.196+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:23:59.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:59.620+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:23:59.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:59.620+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:23:59.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:23:59 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:23:59.620+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:00.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:00.228+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:24:00.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:00.228+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:24:00.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:00.228+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:24:00.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:00.588+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:00.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:00.588+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:00.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:00.588+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:00.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:00 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:00.588+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:01.630 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:01.271+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:24:01.630 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:01.271+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:24:01.630 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:01.271+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:24:01.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:01.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:01.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:01.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:01.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:01.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:01.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:01 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:01.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:02.610 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:02.283+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.029657+0000 front 2024-03-26T03:23:12.029482+0000 (oldest deadline 2024-03-26T03:23:34.329710+0000) 2024-03-26T03:24:02.610 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:02.283+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:17.631312+0000 front 2024-03-26T03:23:17.631321+0000 (oldest deadline 2024-03-26T03:23:43.531293+0000) 2024-03-26T03:24:02.610 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:02.283+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:28.033389+0000 front 2024-03-26T03:23:28.033402+0000 (oldest deadline 2024-03-26T03:23:53.333347+0000) 2024-03-26T03:24:02.610 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6[49783]: 2024-03-26T03:24:02.283+0000 7f2f338c5700 -1 osd.6 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:36.134740+0000 front 2024-03-26T03:23:36.134770+0000 (oldest deadline 2024-03-26T03:24:02.034619+0000) 2024-03-26T03:24:02.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:02.609+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:02.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:02.609+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:02.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:02.609+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:02.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:02 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:02.609+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:03.563 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:03 smithi192 podman[158732]: 2024-03-26 03:24:03.272108619 +0000 UTC m=+5.678025068 container died d2cd545de5e366827ed43cdf9d9a9c3d59d3d069cf76b952642b52321f4ee066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:24:03.881 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:03 smithi192 podman[158732]: 2024-03-26 03:24:03.654494402 +0000 UTC m=+6.060410859 container cleanup d2cd545de5e366827ed43cdf9d9a9c3d59d3d069cf76b952642b52321f4ee066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T03:24:03.881 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:03 smithi192 bash[158732]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6 2024-03-26T03:24:03.881 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:03 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:03.562+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:03.881 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:03 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:03.562+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:03.881 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:03 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:03.562+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:03.881 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:03 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:03.562+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:04.199 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:03 smithi192 podman[158749]: 2024-03-26 03:24:03.880724674 +0000 UTC m=+0.610446338 container remove d2cd545de5e366827ed43cdf9d9a9c3d59d3d069cf76b952642b52321f4ee066 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T03:24:04.487 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:04 smithi192 podman[158839]: 2024-03-26 03:24:04.102502096 +0000 UTC m=+0.023068703 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:24:04.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:04 smithi192 podman[158839]: 2024-03-26 03:24:04.223171799 +0000 UTC m=+0.143738382 container create e6aac023777c2a22e898b1b2516d7a68bb33e6f766f9830c576e74b32c9bc61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:24:04.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:04 smithi192 podman[158839]: 2024-03-26 03:24:04.425588402 +0000 UTC m=+0.346154986 container init e6aac023777c2a22e898b1b2516d7a68bb33e6f766f9830c576e74b32c9bc61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:24:04.488 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:04 smithi192 podman[158839]: 2024-03-26 03:24:04.428573141 +0000 UTC m=+0.349139729 container start e6aac023777c2a22e898b1b2516d7a68bb33e6f766f9830c576e74b32c9bc61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:24:04.987 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:04 smithi192 podman[158839]: 2024-03-26 03:24:04.519126831 +0000 UTC m=+0.439693416 container attach e6aac023777c2a22e898b1b2516d7a68bb33e6f766f9830c576e74b32c9bc61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:24:04.987 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:04 smithi192 podman[158839]: 2024-03-26 03:24:04.721543675 +0000 UTC m=+0.642110272 container died e6aac023777c2a22e898b1b2516d7a68bb33e6f766f9830c576e74b32c9bc61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T03:24:04.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:04 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:04.592+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:04.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:04 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:04.592+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:04.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:04 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:04.592+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:04.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:04 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:04.592+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:05.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T03:24:05.273 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:05 smithi192 podman[158865]: 2024-03-26 03:24:05.221021575 +0000 UTC m=+0.494203767 container remove e6aac023777c2a22e898b1b2516d7a68bb33e6f766f9830c576e74b32c9bc61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T03:24:05.273 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:05 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T03:24:05.273 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:05 smithi192 systemd[1]: Stopped Ceph osd.6 for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:05.273 INFO:journalctl@ceph.osd.6.smithi192.stdout:Mar 26 03:24:05 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.6.service: Consumed 23min 29.528s CPU time. 2024-03-26T03:24:05.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:24:05.289 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T03:24:05.289 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T03:24:05.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.7 2024-03-26T03:24:05.602 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 systemd[1]: Stopping Ceph osd.7 for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:05.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.601+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:05.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.601+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:05.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.601+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:05.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.601+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:05.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.623+0000 7f87bccfe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:24:05.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.623+0000 7f87bccfe700 -1 osd.7 509 *** Got signal Terminated *** 2024-03-26T03:24:05.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:05 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:05.623+0000 7f87bccfe700 -1 osd.7 509 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:24:06.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:06 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:06.603+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:06.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:06 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:06.603+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:06.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:06 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:06.603+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:06.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:06 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:06.603+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:07.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:07 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:07.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:07.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:07 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:07.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:07.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:07 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:07.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:07.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:07 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:07.628+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:08.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:08.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:08.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:08.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:08.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:08.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:08.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:08.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:08.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:08 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:08.626+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.192:6802 osd.4 since back 2024-03-26T03:23:45.384984+0000 front 2024-03-26T03:23:45.385002+0000 (oldest deadline 2024-03-26T03:24:08.285056+0000) 2024-03-26T03:24:09.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:09.674+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6803 osd.0 since back 2024-03-26T03:23:12.581070+0000 front 2024-03-26T03:23:12.580926+0000 (oldest deadline 2024-03-26T03:23:35.481107+0000) 2024-03-26T03:24:09.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:09.674+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6807 osd.1 since back 2024-03-26T03:23:20.082847+0000 front 2024-03-26T03:23:20.082838+0000 (oldest deadline 2024-03-26T03:23:44.182784+0000) 2024-03-26T03:24:09.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:09.674+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6811 osd.2 since back 2024-03-26T03:23:30.083875+0000 front 2024-03-26T03:23:30.083838+0000 (oldest deadline 2024-03-26T03:23:54.783788+0000) 2024-03-26T03:24:09.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:09.674+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.66:6815 osd.3 since back 2024-03-26T03:23:34.784393+0000 front 2024-03-26T03:23:34.784465+0000 (oldest deadline 2024-03-26T03:24:00.084342+0000) 2024-03-26T03:24:09.988 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:09 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7[55693]: 2024-03-26T03:24:09.674+0000 7f87b92f2700 -1 osd.7 509 heartbeat_check: no reply from 172.21.15.192:6802 osd.4 since back 2024-03-26T03:23:45.384984+0000 front 2024-03-26T03:23:45.385002+0000 (oldest deadline 2024-03-26T03:24:08.285056+0000) 2024-03-26T03:24:11.450 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:11 smithi192 podman[158945]: 2024-03-26 03:24:11.192991188 +0000 UTC m=+5.683889148 container died eafdfc1ac5810a544ca36a44f251a15dc26171c82e622ae0e989b682523a044c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T03:24:11.725 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:11 smithi192 podman[158945]: 2024-03-26 03:24:11.531961123 +0000 UTC m=+6.022859088 container cleanup eafdfc1ac5810a544ca36a44f251a15dc26171c82e622ae0e989b682523a044c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-26T03:24:11.725 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:11 smithi192 bash[158945]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7 2024-03-26T03:24:11.987 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:11 smithi192 podman[158962]: 2024-03-26 03:24:11.724741656 +0000 UTC m=+0.528406114 container remove eafdfc1ac5810a544ca36a44f251a15dc26171c82e622ae0e989b682523a044c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T03:24:12.395 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:12 smithi192 podman[159055]: 2024-03-26 03:24:12.125645102 +0000 UTC m=+0.114252465 container create 72d4f4e3a581b1d7e575e0e4c6c78ed8c8d9dd0f0ab279213f1b01aac3c1b651 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T03:24:12.395 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:12 smithi192 podman[159055]: 2024-03-26 03:24:12.029839867 +0000 UTC m=+0.018447190 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:24:12.395 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:12 smithi192 podman[159055]: 2024-03-26 03:24:12.319870214 +0000 UTC m=+0.308477535 container init 72d4f4e3a581b1d7e575e0e4c6c78ed8c8d9dd0f0ab279213f1b01aac3c1b651 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T03:24:12.395 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:12 smithi192 podman[159055]: 2024-03-26 03:24:12.322880703 +0000 UTC m=+0.311488026 container start 72d4f4e3a581b1d7e575e0e4c6c78ed8c8d9dd0f0ab279213f1b01aac3c1b651 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T03:24:12.395 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:12 smithi192 podman[159055]: 2024-03-26 03:24:12.394823642 +0000 UTC m=+0.383430974 container attach 72d4f4e3a581b1d7e575e0e4c6c78ed8c8d9dd0f0ab279213f1b01aac3c1b651 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T03:24:12.737 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:12 smithi192 podman[159055]: 2024-03-26 03:24:12.612042252 +0000 UTC m=+0.600649575 container died 72d4f4e3a581b1d7e575e0e4c6c78ed8c8d9dd0f0ab279213f1b01aac3c1b651 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T03:24:13.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T03:24:13.146 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:13 smithi192 podman[159080]: 2024-03-26 03:24:13.098172989 +0000 UTC m=+0.480744666 container remove 72d4f4e3a581b1d7e575e0e4c6c78ed8c8d9dd0f0ab279213f1b01aac3c1b651 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T03:24:13.146 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:13 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-26T03:24:13.146 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:13 smithi192 systemd[1]: Stopped Ceph osd.7 for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:13.146 INFO:journalctl@ceph.osd.7.smithi192.stdout:Mar 26 03:24:13 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@osd.7.service: Consumed 19min 8.376s CPU time. 2024-03-26T03:24:13.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:24:13.161 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T03:24:13.161 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T03:24:13.161 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T03:24:13.473 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:13 smithi066 systemd[1]: Stopping Ceph rgw.foo.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:13.859 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a[64520]: 2024-03-26T03:24:13.472+0000 7fb0f2b81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:24:13.859 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:13 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a[64520]: 2024-03-26T03:24:13.472+0000 7fb101407ac0 -1 shutting down 2024-03-26T03:24:23.807 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:23 smithi066 bash[204652]: time="2024-03-26T03:24:23Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T03:24:23.808 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:23 smithi066 podman[204652]: 2024-03-26 03:24:23.490432568 +0000 UTC m=+10.125831882 container died cd8028a425d08a16e1c4f8de77bca651e82d24c1740de5dd35edf64a5114d471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T03:24:24.108 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 podman[204652]: 2024-03-26 03:24:24.038524497 +0000 UTC m=+10.673923804 container cleanup cd8028a425d08a16e1c4f8de77bca651e82d24c1740de5dd35edf64a5114d471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T03:24:24.109 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 bash[204652]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T03:24:24.468 DEBUG:teuthology.orchestra.run.smithi066:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T03:24:24.496 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 podman[204667]: 2024-03-26 03:24:24.239568101 +0000 UTC m=+0.731248266 container remove cd8028a425d08a16e1c4f8de77bca651e82d24c1740de5dd35edf64a5114d471 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:24:24.496 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T03:24:24.496 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-26T03:24:24.496 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 systemd[1]: Stopped Ceph rgw.foo.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:24.496 INFO:journalctl@ceph.rgw.foo.a.smithi066.stdout:Mar 26 03:24:24 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@rgw.foo.a.service: Consumed 3.851s CPU time. 2024-03-26T03:24:24.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:24:24.510 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T03:24:24.510 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T03:24:24.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T03:24:24.808 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 systemd[1]: Stopping Ceph prometheus.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:25.070 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.807Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T03:24:25.070 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T03:24:25.070 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T03:24:25.070 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T03:24:25.070 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.808Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.817Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.819Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.819Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a[69362]: ts=2024-03-26T03:24:24.819Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T03:24:25.071 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:24 smithi192 podman[159161]: 2024-03-26 03:24:24.828643203 +0000 UTC m=+0.122302903 container died 786874bf8abca698d9681bb9f305d45f8fa73f84f1f12704743c74265618151d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:25.459 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:25 smithi192 podman[159161]: 2024-03-26 03:24:25.208216022 +0000 UTC m=+0.501875730 container cleanup 786874bf8abca698d9681bb9f305d45f8fa73f84f1f12704743c74265618151d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:25.459 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Mar 26 03:24:25 smithi192 bash[159161]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T03:24:25.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T03:24:25.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:24:25.614 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T03:24:25.614 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T03:24:25.820 INFO:teuthology.orchestra.run.smithi066.stdout:Deleting cluster with fsid: eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T03:24:28.358 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:28 smithi066 systemd[1]: Stopping Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:28.825 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:28 smithi066 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a[80591]: ts=2024-03-26T03:24:28.476Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T03:24:28.825 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:28 smithi066 podman[205146]: 2024-03-26 03:24:28.528153656 +0000 UTC m=+0.146100902 container died 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:29.571 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 podman[205146]: 2024-03-26 03:24:29.152154158 +0000 UTC m=+0.770101446 container cleanup 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:29.571 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 bash[205146]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T03:24:29.571 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 podman[205160]: 2024-03-26 03:24:29.311661132 +0000 UTC m=+0.770400481 container remove 3ef1b23e1dded38560625849566756cb61f0b50e1c880163415dbf64b4148b9b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:29.571 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 podman[205160]: 2024-03-26 03:24:29.381658961 +0000 UTC m=+0.840398340 volume remove 964aad9b7fb2373b952381fac051e75842355c30c924f9a724d89b5df5da6ac6 2024-03-26T03:24:29.858 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T03:24:29.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 systemd[1]: Stopped Ceph alertmanager.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:29.859 INFO:journalctl@ceph.alertmanager.a.smithi066.stdout:Mar 26 03:24:29 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.801s CPU time. 2024-03-26T03:24:29.859 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:29 smithi066 systemd[1]: Stopping Ceph node-exporter.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:30.358 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 podman[205273]: 2024-03-26 03:24:30.050825069 +0000 UTC m=+0.118031922 container died 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:30.858 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 podman[205273]: 2024-03-26 03:24:30.473643105 +0000 UTC m=+0.540850006 container cleanup 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:30.859 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 bash[205273]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T03:24:30.859 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 podman[205287]: 2024-03-26 03:24:30.683281894 +0000 UTC m=+0.626435681 container remove 538de899f37d9df7582768a14545206dea20f4ae1b8590c1dec149476c663a46 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:24:30.859 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T03:24:31.358 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T03:24:31.358 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 systemd[1]: Stopped Ceph node-exporter.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:31.358 INFO:journalctl@ceph.node-exporter.a.smithi066.stdout:Mar 26 03:24:30 smithi066 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 4.953s CPU time. 2024-03-26T03:24:31.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T03:24:31.834 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T03:24:33.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:33 smithi192 systemd[1]: Stopping Ceph iscsi.iscsi.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:34.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:33 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a[60488]: debug Shutdown received 2024-03-26T03:24:44.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:43 smithi192 bash[159581]: time="2024-03-26T03:24:43Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T03:24:44.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:43 smithi192 podman[159581]: 2024-03-26 03:24:43.992581649 +0000 UTC m=+10.129361128 container died 9ee38dbf5ac9b1e0871807c056a8550317890b8e8d1186c00079e2cd1742b825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T03:24:44.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 podman[159581]: 2024-03-26 03:24:44.394927546 +0000 UTC m=+10.531707034 container cleanup 9ee38dbf5ac9b1e0871807c056a8550317890b8e8d1186c00079e2cd1742b825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T03:24:44.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 bash[159581]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T03:24:44.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 podman[159596]: 2024-03-26 03:24:44.620910915 +0000 UTC m=+0.624717931 container remove 9ee38dbf5ac9b1e0871807c056a8550317890b8e8d1186c00079e2cd1742b825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T03:24:44.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T03:24:44.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T03:24:44.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 systemd[1]: Stopped Ceph iscsi.iscsi.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:44.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Mar 26 03:24:44 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 3.514s CPU time. 2024-03-26T03:24:46.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 systemd[1]: Stopping Ceph grafana.a for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:46.839 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=server t=2024-03-26T03:24:46.530209241Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-26T03:24:46.839 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=tracing t=2024-03-26T03:24:46.530567553Z level=info msg="Closing tracing" 2024-03-26T03:24:46.839 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a[63950]: logger=ticker t=2024-03-26T03:24:46.530575995Z level=info msg=stopped last_tick=2024-03-26T03:24:40Z 2024-03-26T03:24:46.839 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 podman[159859]: 2024-03-26 03:24:46.540388147 +0000 UTC m=+0.103151754 container died d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 (image=quay.io/ceph/grafana:9.4.12, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:24:47.237 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 podman[159859]: 2024-03-26 03:24:46.921805728 +0000 UTC m=+0.484569380 container cleanup d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 (image=quay.io/ceph/grafana:9.4.12, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:24:47.237 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:46 smithi192 bash[159859]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a 2024-03-26T03:24:47.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:47 smithi192 podman[159873]: 2024-03-26 03:24:47.323125506 +0000 UTC m=+0.756459272 container remove d6cdbceae005d10933abf3d14f1d2b35d3af4c54226a0fdd97934c8e7c382d67 (image=quay.io/ceph/grafana:9.4.12, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:24:47.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:47 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T03:24:47.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:47 smithi192 systemd[1]: Stopped Ceph grafana.a for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:47.675 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Mar 26 03:24:47 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@grafana.a.service: Consumed 17.310s CPU time. 2024-03-26T03:24:47.937 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:47 smithi192 systemd[1]: Stopping Ceph node-exporter.b for eba7a548-eb1b-11ee-95d0-87774f69a715... 2024-03-26T03:24:48.237 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:47 smithi192 podman[159987]: 2024-03-26 03:24:47.936958703 +0000 UTC m=+0.115726652 container died b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:24:48.636 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 podman[159987]: 2024-03-26 03:24:48.259266301 +0000 UTC m=+0.438034287 container cleanup b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:24:48.636 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 bash[159987]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T03:24:48.636 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 podman[160001]: 2024-03-26 03:24:48.451840491 +0000 UTC m=+0.519448046 container remove b9ddcc4cac873722366e61dea262aab59c890fdab6730051ae6af741615fc0ec (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eba7a548-eb1b-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:24:48.636 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T03:24:48.987 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T03:24:48.987 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 systemd[1]: Stopped Ceph node-exporter.b for eba7a548-eb1b-11ee-95d0-87774f69a715. 2024-03-26T03:24:48.988 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Mar 26 03:24:48 smithi192 systemd[1]: ceph-eba7a548-eb1b-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 4.398s CPU time. 2024-03-26T03:24:49.320 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:24:49.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:24:49.390 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T03:24:49.392 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/remote/smithi066/crash 2024-03-26T03:24:49.393 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/crash -- . 2024-03-26T03:24:49.468 INFO:teuthology.orchestra.run.smithi066.stderr:tar: /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T03:24:49.468 INFO:teuthology.orchestra.run.smithi066.stderr:tar: Error is not recoverable: exiting now 2024-03-26T03:24:49.470 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/remote/smithi192/crash 2024-03-26T03:24:49.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/crash -- . 2024-03-26T03:24:49.541 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T03:24:49.541 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-03-26T03:24:49.542 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T03:24:49.542 DEBUG:teuthology.orchestra.run.smithi066:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T03:24:49.637 INFO:teuthology.orchestra.run.smithi066.stdout:2024-03-26T03:04:14.387440+0000 mon.a (mon.0) 1051 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T03:24:49.638 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T03:24:49.638 DEBUG:teuthology.orchestra.run.smithi066:> sudo egrep '\[SEC\]' /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T03:24:49.679 DEBUG:teuthology.orchestra.run.smithi066:> sudo egrep '\[ERR\]' /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T03:24:49.757 DEBUG:teuthology.orchestra.run.smithi066:> sudo egrep '\[WRN\]' /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T03:24:49.834 INFO:teuthology.orchestra.run.smithi066.stdout:2024-03-26T03:04:14.387440+0000 mon.a (mon.0) 1051 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T03:24:49.834 INFO:tasks.cephadm:Compressing logs... 2024-03-26T03:24:49.834 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T03:24:49.876 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T03:24:49.926 INFO:teuthology.orchestra.run.smithi066.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T03:24:49.926 INFO:teuthology.orchestra.run.smithi066.stderr:: No such file or directory 2024-03-26T03:24:49.933 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-26T03:24:49.933 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T03:24:49.935 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T03:24:49.935 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T03:24:49.935 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T03:24:49.935 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr: 57.6% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.1.log: /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.0.log: /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/objectstore_tool.2.log 2024-03-26T03:24:49.938 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T03:24:49.940 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T03:24:49.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T03:24:49.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T03:24:49.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T03:24:50.036 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.b.log: 67.2% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T03:24:50.057 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/objectstore_tool.2.log: /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.c.log: 92.1% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T03:24:50.058 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T03:24:50.112 INFO:teuthology.orchestra.run.smithi192.stderr: 81.5% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T03:24:50.179 INFO:teuthology.orchestra.run.smithi066.stderr: 93.9% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/objectstore_tool.2.log.gz 2024-03-26T03:24:50.240 INFO:teuthology.orchestra.run.smithi192.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T03:24:50.326 INFO:teuthology.orchestra.run.smithi192.stderr: 86.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T03:24:50.326 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T03:24:50.334 INFO:teuthology.orchestra.run.smithi066.stderr: 91.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T03:24:50.361 INFO:teuthology.orchestra.run.smithi066.stderr: 94.1% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T03:24:50.363 INFO:teuthology.orchestra.run.smithi066.stderr: 89.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T03:24:50.385 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T03:24:50.604 INFO:teuthology.orchestra.run.smithi066.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T03:24:55.257 INFO:teuthology.orchestra.run.smithi066.stderr: 89.3% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T03:24:56.589 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T03:24:58.298 INFO:teuthology.orchestra.run.smithi066.stderr: 90.6% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T03:25:12.645 INFO:teuthology.orchestra.run.smithi066.stderr: 90.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T03:26:46.986 INFO:teuthology.orchestra.run.smithi066.stderr: 92.2% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T03:33:45.357 INFO:teuthology.orchestra.run.smithi066.stderr: 93.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T03:34:26.277 INFO:teuthology.orchestra.run.smithi192.stderr: 93.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T03:36:25.726 INFO:teuthology.orchestra.run.smithi192.stderr: 93.4% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T03:36:35.118 INFO:teuthology.orchestra.run.smithi066.stderr: 92.7% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T03:36:46.484 INFO:teuthology.orchestra.run.smithi066.stderr: 93.0% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T03:36:46.486 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-26T03:36:46.486 INFO:teuthology.orchestra.run.smithi066.stderr:real 11m56.597s 2024-03-26T03:36:46.486 INFO:teuthology.orchestra.run.smithi066.stderr:user 7m18.540s 2024-03-26T03:36:46.486 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m18.691s 2024-03-26T03:37:40.888 INFO:teuthology.orchestra.run.smithi192.stderr: 93.6% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T03:38:12.830 INFO:teuthology.orchestra.run.smithi192.stderr: 92.9% -- replaced with /var/log/ceph/eba7a548-eb1b-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T03:38:12.832 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-26T03:38:12.832 INFO:teuthology.orchestra.run.smithi192.stderr:real 13m22.941s 2024-03-26T03:38:12.832 INFO:teuthology.orchestra.run.smithi192.stderr:user 5m44.919s 2024-03-26T03:38:12.832 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m17.194s 2024-03-26T03:38:12.833 INFO:tasks.cephadm:Archiving logs... 2024-03-26T03:38:12.833 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/remote/smithi066/log 2024-03-26T03:38:12.834 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T03:39:18.464 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/remote/smithi192/log 2024-03-26T03:39:18.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T03:40:10.891 INFO:tasks.cephadm:Removing cluster... 2024-03-26T03:40:10.892 DEBUG:teuthology.orchestra.run.smithi066:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 --force 2024-03-26T03:40:11.138 INFO:teuthology.orchestra.run.smithi066.stdout:Deleting cluster with fsid: eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T03:40:11.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eba7a548-eb1b-11ee-95d0-87774f69a715 --force 2024-03-26T03:40:12.293 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: eba7a548-eb1b-11ee-95d0-87774f69a715 2024-03-26T03:40:13.015 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T03:40:13.015 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T03:40:13.048 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T03:40:13.090 INFO:tasks.cephadm:Teardown complete 2024-03-26T03:40:13.091 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T03:40:13.101 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T03:40:13.101 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T03:40:13.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T03:40:13.218 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T03:40:13.227 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T03:40:13.227 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T03:40:13.230 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T03:40:13.244 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-26T03:40:13.268 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T03:40:13.268 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-26T03:40:13.268 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -160us[ -160us] +/- 92ms 2024-03-26T03:40:13.269 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -490us[ -491us] +/- 73ms 2024-03-26T03:40:13.269 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +676us[ +676us] +/- 76ms 2024-03-26T03:40:13.269 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:40:13.273 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-26T03:40:13.301 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T03:40:13.302 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-26T03:40:13.302 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -185us[ -184us] +/- 92ms 2024-03-26T03:40:13.302 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -505us[ -504us] +/- 73ms 2024-03-26T03:40:13.302 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +673us[ +674us] +/- 76ms 2024-03-26T03:40:13.302 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:40:13.303 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T03:40:13.311 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T03:40:13.312 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T03:40:13.340 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T03:40:13.469 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T03:40:13.564 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T03:40:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:40:13.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T03:40:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:40:13.631 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T03:40:13.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T03:40:13.691 INFO:teuthology.task.internal:Duration was 3280.534715 seconds 2024-03-26T03:40:13.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T03:40:13.725 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T03:40:13.726 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T03:40:13.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T03:40:13.796 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:40:13.811 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:40:14.302 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T03:40:14.302 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-03-26T03:40:14.302 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T03:40:14.328 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-03-26T03:40:14.328 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T03:40:14.358 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T03:40:14.358 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T03:40:14.370 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T03:40:14.427 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T03:40:14.427 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T03:40:14.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T03:40:16.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T03:40:16.018 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T03:40:16.018 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T03:40:16.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T03:40:16.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T03:40:16.184 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:40:16.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:40:16.224 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-03-26T03:40:16.227 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-03-26T03:40:16.284 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:40:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:40:16.323 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:40:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:40:16.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T03:40:16.350 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T03:40:16.352 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/remote/smithi066 2024-03-26T03:40:16.352 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T03:40:16.437 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398/remote/smithi192 2024-03-26T03:40:16.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T03:40:16.481 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T03:40:16.481 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T03:40:16.484 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T03:40:16.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T03:40:16.543 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T03:40:16.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T03:40:16.566 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T03:40:16.567 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T03:40:16.569 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T03:40:16.581 INFO:teuthology.orchestra.run.smithi066.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 03:40 /home/ubuntu/cephtest 2024-03-26T03:40:16.587 INFO:teuthology.orchestra.run.smithi192.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 03:40 /home/ubuntu/cephtest 2024-03-26T03:40:16.592 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T03:40:16.601 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T03:40:16.667 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T03:40:16.689 DEBUG:teuthology.nuke:shortname: smithi066 2024-03-26T03:40:16.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:40:16.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:33:57.769053', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+HD1F+CsAoAuBiU0YXX6ZO8LDL6Gw8Wd1zlXvGNg99R52pLLesdbE07P2/AiRrsQhexeAGcyfhTVX78iBrlkE='} 2024-03-26T03:40:16.714 DEBUG:teuthology.nuke:shortname: smithi192 2024-03-26T03:40:16.714 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:40:16.735 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622398', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:33:57.770832', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCFDGsZgpfauHHw95gyfgQdV5c8hqgFowRMuIWnKgKgQvujCn7mndJNuDK3/TV/nLRqwgRrrDuliz13Baa87nbI='} 2024-03-26T03:40:16.758 INFO:teuthology.orchestra.console.smithi066:Power off 2024-03-26T03:40:16.758 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T03:40:16.783 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off 2024-03-26T03:40:16.784 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:16.797 INFO:teuthology.orchestra.console.smithi192:Power off 2024-03-26T03:40:16.797 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T03:40:16.806 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-26T03:40:16.820 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-03-26T03:40:16.820 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:16.843 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-26T03:40:20.807 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:20.844 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:20.917 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-26T03:40:20.953 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-26T03:40:24.922 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:24.953 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:25.033 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off 2024-03-26T03:40:25.062 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-26T03:40:25.134 INFO:teuthology.orchestra.console.smithi066:Power off completed 2024-03-26T03:40:29.066 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:40:29.177 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-03-26T03:40:29.278 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-03-26T03:40:29.382 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3280.53471493721 failure_reason: '"2024-03-26T03:04:14.387440+0000 mon.a (mon.0) 1051 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_adking@teuthology success: false 2024-03-26T03:40:29.383 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T03:40:29.463 INFO:teuthology.run:FAIL